15+ years of expectations around how a website can be run were upended by scrapers for LLM training.

They ignore robots.txt rules and seem to scrape as quickly and as broadly as possible. Unlike search engines, it doesn't matter to the LLM provider if the website is offline. Unlike search engines, they take steps to be anonymous and difficult to block, including using residential proxy services and other shared IP space.

I try and keep an open data repository online, but it was not designed for this. Cloudflare is hopefully a temporary fix while we redesign backend and frontend components.

The end result is going to be worse for users — slower querying, less flexibility as we primarily expose what we can precalculate, shifting CPU and bandwidth load to the user's device rather than our servers.