Meta is an exception. They intentionally ignore robots.txt rules.
(No second chances, I blocked their whole IP range.)
Everyone ignores robots.txt because every robots.txt blocks everyone.
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
Think about incentives, what benefit does Anthropic obtain by going to hell?
Everyone ignores robots.txt because every robots.txt blocks everyone.
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
Think about incentives, what benefit does Anthropic obtain by going to hell?