The problem is the whack-a-mole game with hackers and script kiddies. It used to be the case that banning known colo ASNs was enough to get rid of nuisance by STROs, then there was a flood of hacked routers being used for DDoS that was really annoying to get rid of, and then came "residential IP" VPNs and commercial VPNs, both of which get routinely abused by AI scrapers and frankly, the AI scrapers are a worse enemy than the skiddies of 10 years ago. They ruin everything.
And you as a site operator can't really tell apart skiddies, griefers, AI scrapers and legitimate users apart any more.
In what I have seen personally, creating absurdly more load by hitting "expensive" pages that no normal user would ever click in that frequency. The AI scraper bots are really, really dumb - they just follow everything that looks like a link.
Another particularly annoying thing was when spam bots got brainy enough (if I were to guess with AI?) that managed to bypass our maths captcha. That one really still pisses me off because I don't like to torture users or having to use GDPR-violating services.
If all the traffic you see from a particular netblock is people posting hate speech, you're probably not losing much by dropping everything from that whole range.
> Almost as if you shouldn't be banning users because of their IP unless that IP specifically has openly attacked you.
There is no net benefit to allowing non-residential IP addresses by default, maybe add the Google search indexer to the exception list. And with residential IP addresses, unless you're international, it doesn't make sense to allow regions other than your target markets.
The only way to deal with the bot traffic plagueing the modern internet is to cut off as much traffic as you reasonably can.
I've never seen anyone using a VPN for anything other than disruptive behaviour. I had to block vast swathes of mobile broadband providers in a certain warlike Middle Eastern country because if I didn't I'd have anywhere from 100 to 1000 new users every single morning who'd all posted hate speech that won't post here for fear of triggering the right-wing apologists.
Now they just do that over VPNs, which makes keeping them out all the more difficult.
The problem is the whack-a-mole game with hackers and script kiddies. It used to be the case that banning known colo ASNs was enough to get rid of nuisance by STROs, then there was a flood of hacked routers being used for DDoS that was really annoying to get rid of, and then came "residential IP" VPNs and commercial VPNs, both of which get routinely abused by AI scrapers and frankly, the AI scrapers are a worse enemy than the skiddies of 10 years ago. They ruin everything.
And you as a site operator can't really tell apart skiddies, griefers, AI scrapers and legitimate users apart any more.
What are they doing exactly?
In what I have seen personally, creating absurdly more load by hitting "expensive" pages that no normal user would ever click in that frequency. The AI scraper bots are really, really dumb - they just follow everything that looks like a link.
Another particularly annoying thing was when spam bots got brainy enough (if I were to guess with AI?) that managed to bypass our maths captcha. That one really still pisses me off because I don't like to torture users or having to use GDPR-violating services.
Almost as if you shouldn't be banning users because of their IP unless that IP specifically has openly attacked you.
Or I guess you can just DENY ALL.
If all the traffic you see from a particular netblock is people posting hate speech, you're probably not losing much by dropping everything from that whole range.
> Almost as if you shouldn't be banning users because of their IP unless that IP specifically has openly attacked you.
There is no net benefit to allowing non-residential IP addresses by default, maybe add the Google search indexer to the exception list. And with residential IP addresses, unless you're international, it doesn't make sense to allow regions other than your target markets.
The only way to deal with the bot traffic plagueing the modern internet is to cut off as much traffic as you reasonably can.
They're using a VPN.
I've never seen anyone using a VPN for anything other than disruptive behaviour. I had to block vast swathes of mobile broadband providers in a certain warlike Middle Eastern country because if I didn't I'd have anywhere from 100 to 1000 new users every single morning who'd all posted hate speech that won't post here for fear of triggering the right-wing apologists.
Now they just do that over VPNs, which makes keeping them out all the more difficult.