Just FYI, the bigger companies all allow you to block crawlers via robots.txt:
# Block Anthropic (Claude)
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
# Block OpenAI (ChatGPT)
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
# Block Perplexity
User-agent: PerplexityBot
Disallow: /
# Block Google's AI Training
User-agent: Google-Extended
Disallow: /
User-agent: Google-Extended-Factual
Disallow: /
# Block Microsoft's Search & AI Crawler
User-agent: Bingbot
Disallow: /
This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot...
I presume it would also cut you off even more from referral traffic.
90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.
I once had the dumb idea to make websites in pdf but the idea is growing on me. Perhaps it should even be in animated gif with <img> <map>'s