What if content providers reduced the 30k word page for a recipe down to just the actual recipe, would this reduce the amount of data these bots are pulling down?
I don't see this slowing down. If websites don't adapt to the AI deep search reality, the bot will just go somewhere else. People don't want to read these massive long form pages geared at outdated Google SEO techniques.
You're painting this as a problem that is somehow related to overly long form text based web pages. It isn't. If you host a local cleaning company site, or a game walkthrough site, or a roleplaying forum, the bots will flood the gates all the same.
You are right that it doesn't look like it is slowing down, but the developing result of this will not be people posting a shorter recipe, it will be a further contraction of the public facing, open internet.
Kinda funny to see someone casually mention a roleplaying forum. That's what I run and it got 10x traffic overnight from AI bots.
Made it when I was a teenager and got stuck running it the rest of my life.
Of course, the bots go super deep into the site and bust your cache.
you'd be within your right to serve the bots fake gibberish data.
Maybe they'll crawl less when it starts damaging models.
That's only a stopgap measure, eventually they'll realize what's happening and use distributed IPs and fake user agents to look like normal users. The Tencent and Bytedance scrapers are already doing this.
Text content at the sub-page level is approximately 0% of web traffic. It's a non-issue.
Then remove the 30MB of advertisements, serve just the 3kB of text, and your server load will be completely fine.