Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening.
It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.
That'd fail for three reasons.
First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine.
Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS folks wouldn't have the resources to run that.
Third, assuming that scrapers are willing to play the JS game, suddenly you have a massively increased rate of full clones happening. Even if serving raw git is cheap for your backend, the bandwidth the elevated clone count drives is not.
1. Preventing access to open-source code by scrapers is not the goal behind this idea. The goal is to reduce the server load taken by the scrapers.
2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?
I don't think the show-commit or file-at-revision routes are what's causing the bot load.
Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.
I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.
This is a much more substantial critique :) I definitely don't think this is a trivial idea. I did a quick analysis and the naivest possible solution isn't very good: only looking at uncompressed objects requires scanning ~340 MB to deliver the <1 MB answer the question "show me the diffs by user Zhang Yi in file fs/ext4/inode.c on branch master in 2025". The bulk of that is scanning commit objects for commits by Zhang Yi in 2025 (171 MB), then fetching trees to filter those to just fs/ext4/inode.c (159 MB).
But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)