It's some kind of perverse regulatory capture by landlords with the self-destructive assistance of homeowners. Actual urban real estate is preposterously expensive, but the only reason it stays that way is that you can't create more of it, i.e. put urban developments in high demand areas that are currently suburbs.
The landlords who own the urban real estate correctly deduce that allowing more to be built would lower rents. They then convince suburban homeowners to go in with them on preventing that from happening, even though it's not really in the homeowners' interest, because rezoning would reduce the value of housing (i.e. price per square foot) but increase the value of land because you could build more housing on it. The urban landlords are the only losers there, because they have a high ratio of housing to land, so they want the housing to be expensive rather than the land. The ordinary suburban homeowners, by contrast, have a high ratio of land to housing, so they benefit from making the land worth more, i.e. allowing more to be built on it, but are bamboozled into wanting the price of housing to be high and therefore oppose urbanization.
You also get a lot of rubbish arguments about "induced demand" which try to imply that building housing would raise prices, when what it actually does is raise prices in the area immediately surrounding the new development (because people like new developments) while making it more affordable in all of the places they're moving from. Which is then used as an excuse not to do it, even though it improves affordability on net while creating more of the areas people actually want.
Something similar happens with commercial space. The landlords want it to be scarce. The argument used in that case will be something like "businesses will buy up houses to build Starbucks" or "it will make traffic worse" as if they wouldn't happily give you a Starbucks and a dozen new housing units on that piece of land if you'd let them, and as if traffic gets worse instead of better when people are closer to things and therefore drive fewer miles. But again it's really the landlords with the limited land that is zoned for those things trying to keep anyone else from getting any, and the other arguments are just the kayfabe because "we want rents to be high" is unsympathetic.
I do understand why it is kept as low-density, that makes sense. But why not allow intermingled small shops, family-friendly restaurant/bars, schools, sport facilities… Surely they would make the area more desirable?
I read that the banks often set minimum rents on the commercial real estate built or renovated with their loans. The minimum is too high for most any business to be profitable, so you see a series of failed attempts until a big chain like 711 or Starbucks rents it. They still don't profit and just close the store but keep the rental because longer terms usually. Not sure why the banks have this incentive.
A pet shop in a strip mall closed a few years ago due to a rent increase. The spot has been empty since. I really think there needs to be some reverse market incentives. No tax breaks on empty space if you can't show you're making honest effort to rent it out, and progressively lowering the cost of rent to reflect the lower property value.
If it's actually a market, it should go up AND down. Otherwise it's just a scam.
A lot of this boils down to decades of compounding preferences. Americans were taught that their birthright is to live in detached homes in low-density areas. The density is seen as a negative, yes.
I don't get it either.