It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.

It does. It's literally a felony but for some reason not a single person has pressed charges.

It's because the companies doing this bought the current government.

There is no evidence for who's doing this and nobody has made any effort to obtain any.

Sorry, no evidence for who's running... GPTBot and ClaudeBot?

turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set

Didn’t read the article, huh?