But then you get into a world where people can’t cross the street (moreso than what’s already available). I recall seeing a video on YouTube that explores your suggestion to its inevitable conclusion. Additionally, this could exacerbate suburban sprawl
I don't think so - you could still have pedestrian controlled crossings - press a button and the cars will stop. You just don't need any lights.
You might not even need buttons - driverless cars are already going to have to handle zebra crossings somehow.