Couldn't it be addressed in front of the application with a fail2ban rule, some kind of 429 Too Many Requests quota on a per session basis? Or are the crawlers anonymizing themselves / coming from different IP addresses?
Couldn't it be addressed in front of the application with a fail2ban rule, some kind of 429 Too Many Requests quota on a per session basis? Or are the crawlers anonymizing themselves / coming from different IP addresses?
Yeah, that's where IP intelligence comes in. They're using pretty big IP pools, so, either you're manually adding individual IPs to a list all day (and updating that list as ASNs get continuously shuffled around), or you've got a process in the background that essentially does whois lookups (and caches them, so you aren't also being abusive), parses the metadata returned, and decides whether that request is "okay" or not.
The classic 80/20 rule applies. You can catch about 80% of lazy crawler activity pretty easily with something like this, but the remaining 20% will require a lot more effort. You start encountering edge cases, like crawlers that use AWS for their crawling activity, but also one of your customers somewhere is syncing their WooCommerce orders to their in-house ERP system via a process that also runs on AWS.
I've had crawlers get stuck in a loop before on a search page where you basically could just keep adding things, even if there are no results. I filtered requests that are bots for sure (requests which are specified long past the point of any results). It was over a million unique IPs, most of which only doing 1 or 2 requests on their own (from many different ip blocks)
They are spreading themselves across lots of different IP blocks
Its called Anubis.
Isn't that the one that shows anime characters? Or is Anubis the "professional" version that doesn't show anime chars?
Yes that's Anubis. And yes you pay to not show anime cat girl.
That's genius.
Honestly the more Anubis' anime mascot annoys people the more I like it.
The point of this is to make things difficult for bots, not to annoy visitors of the site. I respect it is the dev's choice to do what they want with the software they create and make available for free. Anime is a polarizing format for reasons beyond the scope of this discussion. It definitely says a lot about the dev
Anime is only "polarizing" for an extreme subset of people. Most people won't care. No one should care, it's just a cute mascot image.
It says a lot more about the pearl clutching of the people complaining about it than it does the dev.
Anubis blocks all phones with odd processor counts, many Pixel phones for example.