Anubis is designed to stop a certain class of badly behaved bots. It intentionally doesn't run if a bot identifies itself with a UA, such as Googlebot, because then you can rate limit it or block by UA and with other tools.
Anubis is active when a user agent looks like a web browser (e.g. contains the "Mozilla" substring every major browser uses). The reverse proxy serves an interstitial page that does a proof-of-work check, validated server side, setting a cookie if it passes.
This means a legitimate user won't constantly get the proof of work check, because they already passed it. But AI bots rotating through tons of residential IPs to scrape your forum or git forge or whatever will be slowed down.
Overall, I like the idea. It's unobtrusive, privacy preserving, and seems to be working out well for a lot of sites.
It doesn't. It slows them down. To stop bots you need to employ the full suite of tools, fingerprinting, IP rep, behavioural analysis. Anubis will slow down your basic scrapers that try to crawl the entire web but it is useless against actual bots
Bots don't [currently] execute JavaScript or follow complicated redirects.
They don't now, but enough "high value to the bots" pages turning on JS or complicated redirects will simply result in the bot authors adding JS execution or redirect following so they can continue "botting" the sites they want to scrape.
It's a hole with no bottom. Each one-up on the anti-bot side will eventually be handled on the bot side.
Anubis is designed to stop a certain class of badly behaved bots. It intentionally doesn't run if a bot identifies itself with a UA, such as Googlebot, because then you can rate limit it or block by UA and with other tools.
Anubis is active when a user agent looks like a web browser (e.g. contains the "Mozilla" substring every major browser uses). The reverse proxy serves an interstitial page that does a proof-of-work check, validated server side, setting a cookie if it passes.
This means a legitimate user won't constantly get the proof of work check, because they already passed it. But AI bots rotating through tons of residential IPs to scrape your forum or git forge or whatever will be slowed down.
Overall, I like the idea. It's unobtrusive, privacy preserving, and seems to be working out well for a lot of sites.
It doesn't. It slows them down. To stop bots you need to employ the full suite of tools, fingerprinting, IP rep, behavioural analysis. Anubis will slow down your basic scrapers that try to crawl the entire web but it is useless against actual bots
The real answer is that it makes sites behave different requiring the bots to make slight adjustments.
And there are just not enough sites using Anubis for the people and companies running the bots to care to do that.
If you do care bypassing Anubis is trivial.
Bots don't execute JavaScript or follow complicated redirects.
Bots don't [currently] execute JavaScript or follow complicated redirects.
They don't now, but enough "high value to the bots" pages turning on JS or complicated redirects will simply result in the bot authors adding JS execution or redirect following so they can continue "botting" the sites they want to scrape.
It's a hole with no bottom. Each one-up on the anti-bot side will eventually be handled on the bot side.
That's not true . A lot of bots are just headless chrome instances .