A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.
Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.
BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.
The crawlers would have still just hammered their site though, right?
I think the idea is that they could store the data in BigQuery, and point users of the site there.
Sure, but now you're moving the site from "Free data presented in a pleasant way to view" to a "pay-as-you-go database". Your audience shifts dramatically, and you lose the ability to share the data you're trying to present.
True. But it sounds like they already lost that.
The crawlers would have still just hammered their site though, right?
Why is your comment exactly word for word of another comment just one level above in the comment chain?
Maybe because they restated what they said instead of addressing to the previous commenter’s point.
Because the crawlers would still have hammered their site, though.
(The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)
I think snowflake has similar, you can rent it out or make it free
Building on AWS is a financial time bomb.
Are you not able to put limits on how much the site can spend?
As of 2026, still not, and probably never.
Sure, but many a hobbyist has discovered the need for that the hard way.
Do you have any view on why the AI scrapers resulted in a heavier load than existing crawlers from eg search engines?
Where they more exhaustive or more frequent?
There's been an explosion of vibe-coded scrapers that behave poorly and ignore robots.txt. Presumably OP disallowed crawling of the search endpoint for the reason stated (that crawling it would result in endless permutations of search filters.)
not the OP but I'd say that google crawls you once and AIs scrape your page every time someone asks them a question that they think your page might be relevant to.
That’s a traffic design problem. You should be happy that your work is valuable and also protected it against excessive requests. Simple.