Humans don't handle all corner cases. People can be slow to react to completely novel or surprising situations. There will be corner cases where humans generally do better than a machine, but the simple rule to slow down and come to a halt if things look too weird or confusing will almost always be the right answer.
Ideally, driverless cars will one day be better drivers than humans and this will save tens of thousands of traffic deaths per year. Holding up progress because cars will be confused in extremely rare or improbable situations will cost more lives than it saves.
Not only are people slow to react to unusual situations, but this is taken advantage of by city designers to force people to slow down.
Random planters in the middle of the road? Streets that narrow and then widen? Drivers start slowly creeping along, which means they are less likely to injury pedestrians.
I think self-driving cars will only become better once they can do all the learning in real time and on-board. Otherwise, they will only be as good as the data they trained on - which is ultimately real meat driver data and a derivations of said data.