I feel like people don't realize that this is how aristocracy happens.

Like, yea, property taxes suck... but the point is you're wealthy even if you want to pretend you're not. Passing the tax burden onto others, just because you can, is exactly how we end up in a world where wealthy landowners live like literally nobility while the people renting their land pay the taxes for their infrastructure.