I tried codeberg, used it a year, then early this year in all their wisdom codeberg decided to show adverserial random text instead of my repo, reporteldly to mess up llm training to user agents they weren't sure were human.
Codeberg had one job, serve my repo, it didn't do that, when brought up, I was told it was a feature not a bug, they could maybe whitelist me but that wasn't my problem, it was that random people got totally blocked or from accessing the repo. I moved back to github.
Is your complaint about Anubis? I see the "checking if you are human" kind of text too when going to codeberg, but it is usually only a second or two. It is because I run a bit of obfuscation and resistance to browser finger printing.
But you find seeing the text for a couple of seconds too annoying to use codeberg? Maybe it is more than 2 seconds for you?
Everyone has their views on what is acceptable in the world and what they will put up with. Just, to my point of view, I think codeberg is trying to fight the good fight in keeping llms from crawling their website.
Codeberg has some extra "anti-scraping" measures than just Anubis which, judging from the community issues[0] sometimes ends up with false positives that cause people to get garbage (example issue[1], potential page with the garbage i found from Google[2]).
Interestingly, someone mentioned that you may get the garbage when searching Codeberg using Google's `site:codeberg.org`
[0] https://codeberg.org/Codeberg/Community/issues
[1] https://codeberg.org/Codeberg/Community/issues/2603
[2] https://codeberg.org/Codeberg-e.V./requests/!~codeberger~!.g...
It’s not Anubis as others have pointed out, they have more going on that blocks you sometimes.
With that said I never used code berg because of Anubis. Something about anime girls popping up turns off any desire to use the product.
I believe they're referencing iocaine[0], which I believe I've seen on codeberg before. I may be wrong though.
[0]: https://iocaine.madhouse-project.org/
[0] Gives me HTTP 418
A rare event you see a wild "HTTP I'm a little Teapot"
Interesting. Well, for information about the project: https://lib.rs/crates/iocaine
The site linked is excluded from the wayback machine.
> only a second or two
If only we had research on the effect of a second or two's effect on user experience.
I wouldn't have a problem with it if it was on expensive endpoints like search or deep history dives, where it matters for server load. But it's every single page, out of some strange sense of righteousness.
No, the issue is when i visited i got a page of senseless drivel (maybe https://lib.rs/crates/iocaine) with the message i was a bot. There was no way around it except "contact support" as the message said the fix, to solve it myself, was to not be a bot.
https://news.ycombinator.com/item?id=48845668 explains it better than i did.
I accessed through mobile, maybe got a bad ip, i don't don't know. But i never could access my repo, and incidently a day before when i tried to share some markdown file in the repo, they told me it wasn't there (i thought it was a them problem then, as i didn't see it).
So that experience and the way it was handled totally ruined my trust. If randomly people can't even access the repo, and that is "working as intended", what's the use to host it there.
They explicitly explained that the problem was not "they find seeing the text for a couple seconds too annoying to use codeberg". Codeberg offered to remove that for them so they'd never see it again. So idk why you're asking about that. Whatever the problem is, it must be something other than that, since that problem doesn't even exist.
The problem is blocking or speed-bumping users in general, everyone else besides themselves.
They didn't put things on a public hosting site for them to be hidden or obfuscated or even to have a tiny friction inserted between a user and that user discovering their stuff, or following a link to something.
Saying "2 seconds" as though that makes it insignificant is completely missing the point. 0.2 seconds, if it's 0.2 more than some other path is the same as a total block. Having a link to something do anything at all other than instantly provide that thing is outrageous and unacceptable, if you care about the experience you want to present to your users or audience.
(Still, I'd say codeberg was at least among the best options if you want someone else to host it for you. The bad job of providing a bad experience for legit users while trying to block AI scrapers can be true at the same time as no one else is doing any better without some other worse strings attached like github.)
Seeing the Anubis weeb interstitial is super annoying.
Agreed. I installed the NoPow add-on and that worked (in Firefox).
codeberg's one job gets interfered with by freeloaders hammering expensive views (like git blame) inconsiderately.
You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it:
* For every Git commit, fetch the version of every file in the repository at that commit.
* See git blame for every file at every commit.
* Attempt to download the archive of each repo at every commit.
* Run every possible pull request search filter combination.
* Run every possible issue search filter combination.
* Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before.
Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.
What do you use to monitor this? I don't really keep a close enough eye on my services to know what the traffic is doing (and haven't had any issues) but maybe I should start
I just tailed the web logs for a bit and saw that it was wild. For fun, I fed 10 minutes of logs into an AI and it picked up a lot of signal I didn't catch at first glance, like clients claiming to be MSIE 7 on Android 3 and such. I added some reject rules to the webserver in front of Forgejo but that only made a dent in the traffic, alas.
That's crazy. So 600,000 someones are absolutely starved for data.
Or one someone with too much money and too little sense misplaced a decimal point in their ScraPy setup?
I think it was the latter.
And the cruel irony is that these are FOSS Git repos I'm publicly sharing. I'd've been fine with them cloning the repo and analyzing away to their heart's content. That's not the way their scraper's wired, though.
This matches my experience of running a public mediawiki server. The bots (mostly Facebook/Meta) will for every single change fetch every single page again, over and over.
I can’t tell if it’s incompetence or malice.
Couldn't they just rate limit them? Are they literally using a new IP for every request?
There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
We used to call this snow shoeing.
In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.
Sounds like a feature. I didn't think Codeberg would have the balls to do something like it. I think I'm sold.
That's the reason I left as well. I complained, was told I need to drop the attitude, or leave.
so I left.
must be harder than I think running a src forge