I broadly agree with you, but there is really a point here about land ownership.

Although developments of the land do improve the value, and thus land ownership has significant utility economically by incentivizing this, there isn't really an economic justification for the owner receiving value for the land itself- why should someone have exclusive rights to a piece of land they didn't create? They bought it, sure, but why did the previous owner have perpetual exclusive rights?

I'd advocate for a small property tax as a replacement for other taxes, because the component that does tax "land value" won't cause economic harm, but all of income tax causes deadweight loss. (Note, Land Value Tax is great in theory, but impossible to define practically- property tax good enough, much harder to game!)

Note that in practice, the biggest abuser of land hoarding is local governments with extremely restrictive zoning that stops productive development of the land- from an economic perspective they own the land, and have sold (or in reality, seized) some but not all of the rights from the 'landowner'. Although this can have advantages to help with coordination problems, in practice it's caused enormous economic damage to many cities by preventing development. At its heart, it's a problem with land hoarding.