But we should also not throw out the baby with the bathwater. All these attempts at blocking AI bots also block other kinds of crawlers as well as real users with niche browsers.

Meanwhile if you are concerned with the parasitic nature of AI companies then no technical measure will solve that. As you have already noted, they can just buy your data from someone else who you can't afford to block - Google, users with a browser extension that records everything, bots that are ahead of you in the game of cat and mouse, etc.