Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
The crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
So? You can design a deterministic crawler for AI that
- Respects robots.txt
- Detects if something is likely a git forge and as a result
- Clone the git repository instead of trying to crawl the forge
None of this needs to be happening, and yet it does at an unprecedented scale.
But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
Which, as the post notes, it's incredibly stupid. So much for artificial "intelligence"
Makes me wonder how much garbage they actually collect across the web. That can't be good for the quality of the LLM.