15+ years of expectations around how a website can be run were upended by scrapers for LLM training.

They ignore robots.txt rules and seem to scrape as quickly and as broadly as possible. Unlike search engines, it doesn't matter to the LLM provider if the website is offline. Unlike search engines, they take steps to be anonymous and difficult to block, including using residential proxy services and other shared IP space.

I try and keep an open data repository online, but it was not designed for this. Cloudflare is hopefully a temporary fix while we redesign backend and frontend components.

The end result is going to be worse for users — slower querying, less flexibility as we primarily expose what we can precalculate, shifting CPU and bandwidth load to the user's device rather than our servers.

You made me press command+w :-|

Luckily command+shift+t undoes.

For me its a bit nuanced, for a small website that may be constantly attacked, they may not have much of resources to work around except putting in cf safeguards in place, and I tend to look past that.

What I do not appreciate as well is companies with larger money pool also resorting to and not caring for, for example Dell has the stupidest security check that has disappointed me (a human) enough times one time that I chose to buy a competitors product.

Unfortunately scrappers are problematic enough that for more and more organizations alternative is to stop serving traffic.

For example OpenStreetMap is using nowadays this or equivalents to prevent downtime and/or runaway costs.

every fuckass website with 0.5 daily visitors has it now. russian minitrue had blocked cloudflare and the internet became completely unusable without a vpn. like you search for something and not a single link works.