> Why is git.kernel.org “interesting” to crawlers
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
The crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
So? You can design a deterministic crawler for AI that
- Respects robots.txt
- Detects if something is likely a git forge and as a result
- Clone the git repository instead of trying to crawl the forge
None of this needs to be happening, and yet it does at an unprecedented scale.
But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
Which, as the post notes, it's incredibly stupid. So much for artificial "intelligence"
Makes me wonder how much garbage they actually collect across the web. That can't be good for the quality of the LLM.
A lot of work is apparently put into bypassing any kind of anti-scraping, no work is apparently put into figuring out if the site freely gives a way to get all that information in a less wasteful way.
Same here. Have a small gitea instance with cloned projects from github I‘m keeping in case the github version gets removed. Every few days there is an army of bots hammering my small vm with 40k req/min for 20 min straight. I had to install anubis to keep the instance online (while still slowed down).
I don't even have anything worthwile on my gitea instance, and the same thing happened to me.
The only thing that ended up helping was disabling access for anonymous users.
I run a non source code service and I also get these, also getting hit by literally millions of residential IPs being cycled after 1 - 2 hits. Ingesting thousands of copies of near identical pages. The bots seem to be programmed in an extremely simple manner, using non-obfuscated javascript to write the href attribute seems to stop them from seeing links, where previously they would endlessly crawl generated links. Unfortunately it also means noscript users get shafted.