I run some small websites, including a tiny forum that’s been a goldmine for scrapers. I had to significantly tweak some firewall rules and configuration after scrapers behind residential proxies suddenly accounted for over 99% of requests.
However, I also relaxed rules for automated traffic that was well-behaved, and I went out of my way to ensure that the Wayback Machine was able to hit everything. I should kick a small donation their way. They provide an incredibly valuable service and I love the benefit that I get from them just for personal side projects.
How do you separate the Wayback Machine from malicious bots that pretend to be the Wayback Machine? Are you whitelisting their IP blocks?
Because I get a ton of scraper requests that forge Googlebot, Bing, and Yandex user-agents that are totally not coming from their IP ranges. In fact, sometimes they all come from the same IP...
They have their own ASN, which I’ve explicitly allowed requests from.
https://www.peeringdb.com/asn/7941
TIL they have their own ASN. This is helpful, thanks.
I thought at least google (and possibly others) provided a way to verify the user agent?
Sorry, not following. I thought the user agent is a string that the caller can set to anything. There is no immediate, reliable way to tell whether the string is correct.
Yeah, a real browser would produce certain patterns and never certain others. So in some cases one could clearly say it's not a human using a browser. But a scraper could also make efforts to mimic human browsing. Mostly the frequency of requests can tell with high probability tell it's not human. But such algorithms might occasionally give false positives for real users, exactly like it has obviously happened for the archive.
What google service are you referring to? Not sure whether the archove uses any of Google's tracking. I have pretty strong blocking of trackers and ads. But the archive works for me.
Google's scraper bot at least used to be behind IPs that you could identify via reverse-then-forward DNS. Not sure if that is still up to date, though.
https://developers.google.com/search/blog/2006/09/how-to-ver...
Yep, verifying the IPs is still the way to go. You often see websites that do it wrong when you set your user agent to Google Bot and they give you a different version of the page without validating that.
Kudos on that. I don't know what site you're hosting, but I appreciate knowing that it is there
This is the way to go! Cut the cat and mouse, win win ish.