But what do these bots gain from this?
It's all OpenAI / Anthropic / Singapore/China based crawlers. I guess they gain data that I provide for free. I get very little crawling by Googlebot / Bing etc which barely even index my site, I only have a single page "indexed" by Google.
Possibly content that appears only briefly? I'm not sure otherwise. It seems wasteful.
Have you ever use an AI chatbot? It's basically a democratized scraper.
Type something stupid, it "visits" 200 websites and regurgitates some random gibberish.
The most recent info of every single fucking thing on the web. Everything. Data locusts.
It's all OpenAI / Anthropic / Singapore/China based crawlers. I guess they gain data that I provide for free. I get very little crawling by Googlebot / Bing etc which barely even index my site, I only have a single page "indexed" by Google.
Possibly content that appears only briefly? I'm not sure otherwise. It seems wasteful.
Have you ever use an AI chatbot? It's basically a democratized scraper.
Type something stupid, it "visits" 200 websites and regurgitates some random gibberish.
The most recent info of every single fucking thing on the web. Everything. Data locusts.