My point is that people choose to outsource the complexity of running a rate limiter and blocking bad actors to Cloudflare and others like them is not the issue you make it out to be.

Why is it good for me to do it myself but bad to pay Cloudflare $20 a month to do it for me. No one is forcing me to use their services. I still have the option to do it myself, or use someone else, or not use anything at all. Seems pretty free to me.

Many AI scraping bots are notoriously bad actors and are hammering sites. Please don’t pretend they are all or even mostly well behaved. We didn’t have this push with the search engine scraping bots as those were mostly well behaved.

You are setting up a straw man with a “hey why not let this hypothetical we’ll behaved bot in”. That isn’t the argument or reality. We didn’t have the need to block Google, Yahoo, or Bings bot because they respected robots.txt and had a reasonable frequency of visits.