"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
Even the premise seems to not understand what it's trying to require. There are zero humans accessing a website by sending electrical signals to Comcast using a telegraph or pulsing a flashlight into a strand of fiber. Every web browser is a computer program running on a computer.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
(a) Prohibition.—A person may not—
(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or
(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.
(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—
(A) failing to identify itself including through a valid and accurate user-agent string; or
(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?
> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.
It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.
User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.
User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
I don't think anybody does a decent ua check, webdevs test only in their favorite browser, i.e. Chrome, so only Chrome is allowed. Is it intended? The very purpose of webcompat is to masquerade as Chrome.
How do you even implement an ua check? How many browsers do you know? How many browsers you don't know? How do you implement an ua check for a browser that will appear tomorrow?
I wrote my own http downloader. Do you know about its existence? How do you implement an ua check for it? Can I call it "wget" to make ua check easier for you? Can I call it "Mozilla/5.0" so that it's compatible with your ua check so that you don't need to change anything?
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
Which is precisely why trying to legislate them is folly. Obviously the reason Android calls itself "Mobile Safari" is so that servers send it the mobile version of the site instead of the desktop one etc.
Which makes "accurate" an unreasonable requirement, since there is already a preexisting non-waivable requirement for inaccurate. When someone is making their "turn a web page into a PDF so you can view it later on the plane" software or whatever, it has to use a user agent string that will cause the site to send the version you need turned into the PDF.
> That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
First, you're assuming that the person being prosecuted is the entities who are the nominal targets of the bill, when the real problem is when they're not. Because you're proposing a law that says anybody with an inaccurate user agent string is in violation, meanwhile everybody has an inaccurate user agent string.
Moreover, consider what you expect the outcome of legislating something like this is going to be. Sites could then try to either blacklist or whitelist user agents.
If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
If they do the second one, that doesn't work, because now every third party has to start pretending to be a common browser even when they're doing something totally innocent or they get blocked by default for being something else. Which makes the user agent string even less useful than it is now, and subjects innocent parties to arbitrary liability for doing something they have to do or their client is broken.
> As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
You're describing autocracy. The way you make it so that you always prosecute someone is to make it so that you can always prosecute anyone, which is very bad.
> you're proposing a law that says anybody with an inaccurate user agent string is in violation
No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.
> Sites could then try to either blacklist or whitelist user agents.
Yes, that is a desired outcome.
> If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
That would put them in violation, and make them prosecutable. Of course you need to start collecting data to prove that it's automatic, but this is already happening today for the purpose of temporarily blocking an IP. Having this law means that you also have some punitive measures at the end of the rainbow.
> If they do the second one ...
Anyone pretending to be a common browser when they're not is in violation. In addition, under this law there's no incentive for anyone to pretend to be googlebot since _that_ can get them blocked. You are intentionally and repeatedly trying to obfuscate these facts.
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.
It's a bit old but https://webaim.org/blog/user-agent-string-history/ is always a good parable.
You're slicing the meaning of UA strings too fine.
Bots can easily identify themselves through REP in RFC 9309. This isn't difficult or confusing.
This model is the exact wrong direction for a healthy society. It will just make it harder to use information that's already available to the public, on a public website, and thus just create moats around training data.
Instead, I'd like to see a comprehensive copyright modernization bill that addresses training data, fair use, and copyrights of generated assets, in a fair and balanced way that minimizes moats and tries to address a reasonable compromise of each stakeholder's conflicting wants (being clever to satisfy as much as possible), and then ban websites from discriminating against traffic based on if it comes from a bot or a human. You can set rate limits, even at the network level, if you want as long as they apply to everyone, or charge per request as long as the price is the same for everyone, and so on. But, if I'm allowed to do it manually I should be allowed to do an identical task automatically.
Lucky that it's only illegal if it could be reasonably predicted to cause harm then!
I care a lot less about this issue as a matter of intellectual property than one of capacity consumption and cost.
You have large VC companies evading rate limits and other restrictions and they're causing real economic harm through their actions.
I am dealing with the bots described in your first paragraph running a primary information source website. They are relentless and cost us thousands of dollars a month in wasted resources. They are almost certainly doing the bidding of much larger corporations as they are extremely sophisticated and seem to have infinite bots to send your way.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
I'm not sure all companies will have deniability. It's pretty easy to see companies which are trying to evade detection or misusing their existing spiders for AI training purposes.
Some examples I've seen have crawlers which mask themselves 99% of the time by faking Chrome, but which will leak their real identity in UA strings which share the overall fingerprint of the swarm.
(I'm looking at you, Replit.)
No, the definition of 'stealth bot' is a disaster. The proposed text criminalizes altering your user-agent string. It arguably criminalizes all existing user-agent strings, since their whole purpose is to lie about which browser you are.
Taking the first thing I could find, geekflare.com informs me that the current user-agent string for Google Chrome running on Windows 11 is
What's doing the work in the proposed text is the prohibition on using software "in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of" a service.> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
Note that someone who's scripting a headless browser doesn't need to use a deceptive user-agent string. They can use the user-agent string for their browser, which isn't deceptive except to the degree that all user-agent strings are 90+% lies.
The proposed text makes it illegal to (not criminalises) running a stealth bot that causes harm that was predictable. This seems reasonable to me. It's legal if either the bot identifies itself, it's harmless, or it should have been harmless (you can't put up a web server that crashes on every 404 and then blame me for crashing it).
A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
> A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
That doesn't mean anything. All of the code in a browser is "automation" -- normal users aren't hand-crafting the hundreds of HTTP requests the browser makes to render a single web page. Meanwhile there is a human somewhere telling the scrapers what to scrape.
> You are entitled to your own emails, posts, etc
Is that somewhere in the proposed text?
> but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
"Stating what they are" buys you nothing when you have no way of verifying it, and you're still not providing any way to distinguish this from investigative journalism or e.g. a new competing search engine.
Using a computer to do "by hand" is a vague concept, yes, but laws routinely embrace vague concepts. You used it yourself! "Hand-crafting", you said. It could be pinned down more closely to something like a single action that happens quite soon after you press a button or ... and further wrangling could ensue because it's never going to have all the vagueness removed. It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
> It could be pinned down more closely to something like a single action that happens quite soon after you press a button
So leaving your webmail open is banned because it will still be checking for new messages three months after you opened the tab without any further human interaction. Likewise RSS readers or really any client which is intended to fetch new posts/messages when they appear.
> It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
Which is kind of exactly the problem. A new law would presumably still be on the books in ten or twenty years. Do we expect user agents by then to do only the things they do now, or will they get more capable? Banning them from becoming more capable is the problem.
> All of the code in a browser is "automation"
I don't think you are interpreting what I said in good faith, but here goes:
If you use an API to control a browser into navigating to a certain webpage, as opposed to using human interface devices to assert said control.
And no, AI is not a human interface device.
I am curious about your view on accessibility tooling. Often screen readers or speech to text tools would effectively be external pieces of software instrumenting for instance browsers through their APIs. And surely using local AI going forward could enhance the tooling while also making the users less dependent on a small market charging exorbitant prices for poor user experience. What viable paths do you see for users perfecting user agents for themselves when the big browsers further lose their plot and focus on what's fun to maintain or profitable to ship instead of optimising for every user individually?
Is a human commanding the computer to do the action?
It's just an unfortunate reality that people with disabilities live with.
Some of them are already prevented from operating heavy machinery because doing so would put everyone at risk, so there is precedent here. I'm in favor of making concessions to the very unfortunate, though at some point you have to look at percentages, expected impact and decide what is reasonable and what isn't.
To your other question, in the last week there have been entire CPUs embedded within weights. Clearly it's possible to publish "open models" with an agenda so then what assurances you have that the enhancements provided by local ai would be more focused on the plot than what the big browsers do?
Specifically how would a person with disabilities be able to tell the difference between improved instrumentation and conditional access to information, as we have in other media? Consider this in the context of "compared to superintelligence, we all have an intelectual disability"
A browser is not heavy machinery. That is not precedent that applies to this situation.
I had a disabled relative (who passed this year), and am quite amazed and offended to hear the viewpoint that those in their situation must suffer an "unfortunate reality" for the sake of everyone else. That's taking the "non-disabled people must accommodate them" (which has problems and is worth a lengthy discussion elsewhere) pendulum and swinging it like a wrecking ball.
If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human? What about 5 seconds? Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction? Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling? Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
What changes if I write my own browser? What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences? What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read? What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want? What changes if it could be recalled but I never decide to do so? What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else? What changes if I download this program from someone else and use it other than how they intended?
Which of these situations is "crawling" and which aren't?
You may believe you have clear definitions, but you don't.
> If I have a browser extension that reloads a page every 5 minutes, so I can visually see if an update has happened, is that controlled by a human?
No, but nobody cares if you reload one page every 5 minutes
> What about 5 seconds?
Definitely no, people will start caring after you do this for an hour or so.
> Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction?
No, if you don't press a button to make it happen every time, it's automated. Also, nobody cares, you are not the intended target here.
> Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling?
No, but being entertained at others' expense suggests some emotional issues you may want to check on.
> Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
Yes. Scalpers are bad for everyone but themselves.
> What changes if I write my own browser?
Nothing.
> What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences?
It's still automated, thus a bot. There might be a fair use case if you could've refreshed the pages manually, but what you are describing is RSS.
> What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read?
I've addressed it here: https://news.ycombinator.com/item?id=50311855
You are unhappy with my response, though I have reason to believe you are biased given your recent loss. I'm sorry for that, though your argument there is the same but with less mass behind the wrecking ball.
> What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want?
Are you recalling it manually or automatically?
> What changes if it could be recalled but I never decide to do so?
Then no crawling happens so you don't have to worry about anything?
> What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else?
This would be automation, since a browser does not behave that way for a normal user.
> What changes if I download this program from someone else and use it other than how they intended?
This is not about programs, it's about websites being crawled against the wishes of their authors.
> You may believe you have clear definitions, but you don't.
Once again, I'm sorry for your loss and any grief my message may have caused you.
I'm taking you at your word, and believe you're stating these things in good faith. As I hope you believe me. Yet I hope what you want never comes to pass. If it does, that world would be confusing, and our society will have lost something valuable.
For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit? How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
> This is not about programs, it's about websites being crawled against the wishes of their authors.
This is very much about programs, because that is the only way anyone accesses a web site. Time-shifting is legal for TV, for example, and that's a good thing, even if broadcasters wish it was banned. We need parallels for the web that balance both users and site owners, and other than robots.txt as a frequently ignored handshake agreement, I don't know much that exists in this area.
Thank you for your condolences. I mentioned the relative passing primarily to explain my past tense and to be honest about not staying updated on that situation despite having been recently informed. They passed from age-related health problems after a good, productive life. I have their picture with a former US president on my wall, from the signing of a disability-related law. I believe my only bias in this area is that I have been directly informed through frank discussion, without the normal barriers you might imagine between non-relatives.
Is there already a limit? If I reload your web page 5000 times per microsecond, am I committing a crime right now under the current law?
> For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit?
You can't demand this degree of specificity from laws - that's not how good laws work. The limit should be reasonableness and intent.
Refreshing hacker news every 5 minutes isn't unreasonable. Refreshing some 5GB file on archive.org that never changes would be unreasonable.
Hell it's easy to come up with examples that would be outright fraud (deceit to acquire something of value) and deserve criminal charges. Refreshing a booking website in a manner to trick it into thinking you're about to reserve something to prevent other people from doing so - e.g. to keep people out of nearby campsites on a camping trip.
> How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
Up to a jury. But there's clearly a limit at which the answer is "no amount". It's well established that distributed denial of service attacks are illegal after all.
I don't disagree with your overall point. You cannot determine intent by looking at an HTTP request.
Criminalizing a fan who really, really wants Taylor Swift tickets so that you also prevent ticket scalpers, who would do the exact same observable actions, is not going anywhere good.
When limits must be decided like this, our definitions cannot be enforced by technology, and we can't afford judges and juries to handle the scale of the current situation. Look at the original post I replied to, which appears to believe we can easily define "automated" in a way that's useful to meet the wishes of site owners against the wishes of site visitors, regardless of visitor intent and regardless of accommodation for disabled visitors.
> You cannot determine intent by looking at an HTTP request.
I believe this is the crux of our disagreement. Clearly you can't, which is why you must assume good faith and service all innocent-looking requests.
What you can do however is temporarily log and search for patterns in multiple HTTP requests. If you can then prove that these requests came from a bot via the plethora of fingerprinting measures already in use, you have grounds to make a complaint to the FTC, which would have the ability to subpoena the identity of the offending entity and, depending on their history, issue a warning, a fine or file a lawsuit.
The current status quo is that attackers have an advantage in that the worst thing that you can do to them is to temporarily block them, and you don't really want to do this because you're never 100% sure whether any _individual_ request is automated.
While I can't say that this law is good for this purpose, I am pointing out that the solution looks something like it. The goal is not blocking the individual, but doing something about infra-level abusers, at a volume where 95% accuracy is sufficient proof of wrongdoing. All of your arguments are valid and would be tried in court in response to this passing, hopefully for the better but ultimately it depends on how good the justice system is. This last thing is something no amount of technology can fix but in that case illegal scraping is the least of our problems.
> our definitions cannot be enforced by technology
... that's why the proposed law doesn't try to?
It's a human level distinction that doesn't have technical distinction—the law handles these kinds of things just fine. The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The world you envision where we decide that the law can't assess intent is certainly an interesting one but I don't think it would be the most functional.
> the law handles these kinds of things just fine
No, people just like to claim that it does. Meanwhile legal departments are risk averse, so if nobody can articulate where the line is, you're banning all of the things you claim you're not banning. Even if a judge might ultimately come to a reasonable decision, do you want to be the one who pays the legal fees to "win" that case?
> The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The reason things like this aren't a practical problem is that normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with. On top of that, the law actually does have some pretty specific rules about the distinction between manslaughter and murder.
Whereas the trouble here is that we need to distinguish between things that we specifically don't want to deter and something else which is proposed to be prohibited. The type of law that handles that kind of distinction most frequently is the tax code. The consequence is that it has to be made very specific or the result is a train wreck.
Exactly.
> a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service
"stealth" - what does that mean? This is largely a meaningless term. "damage, impair, or burden" - again, too vague. It would be good to specify something specific, e.g. so many tens of thousand of damage (reciprocal to the assumed fine - a journalist or user is unlikely to be reselling data at this scale, an AI scraper may incur or inflict such costs). "that is reasonably likely to" - this phrase is doing some heavy, heavy lifting, but like others have mentioned, there aren't very many "reasonable" examples, and those have to be defined in a court of law.
Like you and others have said, the immediate reaction would be legal departments essentially banning any and all access, because there would be room for someone to file a suit, and even if you win, there's no guarantee you get your fees back...
As a human being,
"stealth" means that it's not trivial for other people to block you from crawling their shit and
"damage, impair or burden" is in the eye of the beholder, the above person trying to block you from crawling their shit
Yes there will be patent-trolls attempting to squeeze this. Which is why it would be in the crawling entities' best interest to provide electronic means to opt-out and abide by them.
Agreed. The law works pretty well for murder/manslaughter because it's a low frequency event that everyone viscerally understands and generally has evolutionarily-ingrained shared values about.
It works less well for other things, like, for example, white collar crime, even when it is low frequency, and even when it is built on financial concepts and norms that have existed at least for decades if not centuries, there's still plenty of gray area for reasonable doubt and spin.
It has pretty much not worked at all for computer and internet regulation except where it could be directly tied back to pre-internet crime definitions (eg. theft of funds). But data privacy, encryption, intellectual property, "hacking", etc, has all been a dumpster fire. And now we pour agentic gasoline on the whole thing? Our legal system is not equipped, least of all not with the current gerontocracy that is beyond clueless about any of this stuff.
> normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with
It's the same thing here, a normal person will never run afoul of this law. It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter. Sure translating to legalese is hard and I expect this kind of legislation will be mostly common law after a bunch of lawsuits but the giveaway that it will be fine is that assigning "bot/not bot" to a given situation is something we all agree on in the industry. There isn't really that much borderline here. You can't accidentally build a web scraper operating off a residential proxy network and it's pretty obvious when you decide to disguise your traffic because you don't have the consent of site you're scraping.
Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
> You can't accidentally build a web scraper operating off a residential proxy network
Unless I'm missing something, most of the time you ask an agent to do any type of comparison or analysis between mtiple options offered online, scrapping will be involved at some point.
Anything that doesn't have a well-known and very public, free API will be a prime target for scrapping.
> It's the same thing here, a normal person will never run afoul of this law.
Its terms are so broad and ambiguous that they're already doing it and the thing you're actually relying on is that they wouldn't be prosecuted, which is an entirely different thing.
> It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter.
"The courts will sort it out eventually" is the worst excuse for drafting it poorly to begin with, and doesn't do anything for you during the decades it takes for that to actually happen.
> Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
This seems like the thing where corporations co-opt the term "security" to mean "anyone doing anything we don't like" as if they had any right to that to begin with.
Journalists should be able to investigate misconduct. Customers and customer agents should have bulk access to pricing data. Real people shouldn't have to view the speech of other real people through the proprietary interface of a corporate intermediary that people only use because it has a network effect.
By "figure it out well enough" do you mean a laborious and expensive process fraught with miscarriages of justice despite good faith, best efforts? A process where being acquitted (or a plea deal to a lesser charge) might very well result in financial ruin?
Equating "acceptable browser vs abusive bot" with "manslaughter vs murder" is not useful.
And this is a society where many of the most powerful corporations got started by just blatantly ignoring the laws of their industry. See: Uber, OpenAI, etc.
A browser does not send 10k requests per minute to open a webpage. Even in the most and modern bloated webpages. And even if it did it wouldn't be to a single website.
A user operating a scraper or a crawler IS operating a scraper or a crawler.
The legal system isn't stupid - it doesn't think that a human using a web browser is a robot.
I mean it is stupid but not in that way.
Is an AI that is operating a browser on my behalf to answer a question I asked 5 seconds ago--is that a bot or a person?
Well, 10k individual anonymous crawlers accessing a site once a minute are as bad - traffic wise - as one accessing the site 10k a minute. Even worse bc you can't block them.
Having said that I agree crawling should be legal in general. If an agent is acting in my name, it is not much different than a browser.
Knowing the identity allows you to block them. You can also prosecute them for DDoS, which is already illegal.
What good is blocking the ones in your jurisdiction when the internet is global and then they just operate from somewhere else?
Because you can make it not without much difficulty. There's plenty of sites that operate entirely within the US and it's trivial to hard block everyone else. It's not perfect but assuming the law actually works (which isn't a given) inside US borders we now get the advantages of a high-trust system.
At absolute minimum it should mean that US ranges could see fewer captchas because bad actors can be whacked by the legal system instead of degrading user experience.
> There's plenty of sites that operate entirely within the US and it's trivial to hard block everyone else.
It's trivial to block IP addresses listed as being somewhere else. That doesn't block people in other jurisdictions from using a US IP address, either by renting or compromising one in the US or having administrative control over one somewhere else and then setting its metadata to whatever they want.
Well yes but once you're in the US someone owns the machine you're getting hit by and you can make them take it down. People won't just throw their hands up "well the operator isn't in the US nothing can be done", they'll just make DigitalOcean block them.
> Well yes but once you're in the US someone owns the machine you're getting hit by and you can make them take it down.
That's not how you get to high trust, since it requires the opposite -- you'd have to ID everyone and ban VPNs and restrict access to general purpose computers lest any fall under the control of a dastardly foreigner etc.
You can't build a high trust society inside of a police state because a high trust society is one where people do the right thing when no one is holding a gun to their head and a police state is holding a gun to your head.
> they'll just make DigitalOcean block them
Residential proxy. Most cloud providers are blocked or already on reputation watch lists.
A few thousand people downloading a “free vpn” app on their phone is all you need to be back up and running. They get a useful vpn in exchange for letting others borrow their IP.
Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.
> The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country
That costs the crawler money, not many do it and is something that can be fixed over time
banning country and AS numbers for cloud companies is pretty effective in my experience
> That costs the crawler money
They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put delays on even the users in your own region).
> banning country and AS numbers for cloud companies is pretty effective in my experience
It's the sort of thing that works when the first person initially rolls it out because the crawlers only bother to do something about it once a threshold percentage of people are doing it, at which point they use the workaround.
And then you have people insisting that it works even though it only used to work, and meanwhile you're still blocking actual humans.
Their own time to figure out how to get around the block is the biggest cost to them. (That cost is rapidly dwindling.)
Yep doing that (actually just countries) eliminated card testing on an e commerce site I maintain.
From hundreds a day, to zero.
The compromised smart tvs in your country were manufactured by someone that could be held liable for operating a "stealth" bot network, or for allowing someone to do so through negligence.
As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.
> As for journalists, nothing is stopping their legal department
and this is the point.
so people without a 'legal department' are shut off.
Sure but this probably wouldn't happen like that, what would happen is that when journalists make that petition it creates a real special case in the law available to everyone.
> A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
All of the above fall under "very little". We're talking in the ballpark of 5% extra sales max, at the expense of double the infrastructure costs.
They're also stealing your curated dataset: if you take the time to correctly label your products, add specifications and manuals because you want this to be a competitive advantage, your work was just re-appropriated for the benefit of humanity. But primarily for your larger competitors willing to pay for it.