> it's a shame that we've let it go so far,
It's hopeless to expect that things don't end up in this state. A decentralized system will naturally tend to a state of equilibrium balancing between desirability and pain, e.g. people will keep moving to a "nice" area until commutes or prices become unbearable.
I think the only way to end up with an utopia-like metropolis is to run it with a benevolent dictator government SimCity-like, which would probably involve restricted entry leading to very expensive real estate; therefore a lottery or similar admission system into low-cost housing would be needed to balance the needed support population. In other words probably unconstitutional in a dozen different ways and never going to happen.
That "state of equilibrium" is only unavoidable if there are infinite sources and sinks of people. That's a workable approximation if you're only studying one part of a much larger system, in isolation, but when considering the entire world, it falls down. If we have enough nice areas for everyone to live in them, that model stops being applicable.