Couldn't they just rate limit them? Are they literally using a new IP for every request?
There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
We used to call this snow shoeing.
In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.
There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
We used to call this snow shoeing.
In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.