I really enjoyed the writing style in this article.
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.
It does. It's literally a felony but for some reason not a single person has pressed charges.
It's because the companies doing this bought the current government.
There is no evidence for who's doing this and nobody has made any effort to obtain any.
Sorry, no evidence for who's running... GPTBot and ClaudeBot?
turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set
Didn’t read the article, huh?
It goes beyond mirrors, it's just something criminals have been doing since forever, to abuse all websites.