...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans?
Obviously this is terrible, but I think there's a possibility it's the least terrible option? Another option is IP reputation, which I think is worse. Or scanning a code with a non-rooted phone, which I think is even worse than that!
> ...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans?
There isn't one, and pretending otherwise is nonsense because humans will always provide their credentials to something to act on their behalf.
In the limit you end up with Chinese phone farms.
Right. Botnet operators love cloudflare because they make so much money renting out compromised machines to pass their tests.
The only solution is regulation. If all content created by anyone has a copyright, how does an implicit opt-in (which is what happens if you don't create a robots.txt file for your website) for scraping make any sense? Moreover, even if you have a robots.txt, AI (or whatever) bots often don't respect it (or use workarounds - they outsource scraping of such "restricted" sites to unethical third-parties to get the data; Meta has even resorted to piracy, openly!). So clearly, the logic and the "honour system" has failed.
Cloudflare, Google Captcha, HCaptcha etc. are all shitty technical solutions because, as we are all discovering, it comes at the cost of our privacy (i.e. our personal data may monetise these services) and / or our computing resource and time. If current copyright laws aren't sufficient to prevent this, we have to acknowledge the system is broken. The answer could be enhancing it with some kind of Digital Millennium Copyright Act (DMCA) -like laws, but in favour of the creators against BigTech or rogue actors.
- Web-scraping and copyright law - https://www.neudata.co/blog/web-scraping-and-copyright-law
- Why DMCA Claims Against Web Scrapers Face Long Odds - https://capstonedc.com/insights/why-dmca-claims-against-web-...
Or the regulated agents standard that cloudflare is conveniently going to steward alongside Google...
Or you could let information be free, at least the stuff that’s on the public net.
As for issues like bots overloading websites or using too many resources scaling laws will take care of it quickly, it’s not like you can’t serve thousands of RPS from a Raspberry Pi these days.
I don't think regulation will stop web scraping, not least of which because it can be done from locations outside the jurisdiction of the regulations.
> we have to acknowledge the system is broken
The system is broken. It probably takes, what, 10 seconds or less to use a residential or foreign proxy, 6+ months to internationally track and prosecute a single offender? So like a million times more effort going the regulatory route.
Just as criminal laws don't end all crimes, copyright laws and anti-scraping regulation won't end all scraping. But it will greatly reduce it and limit it to rogue actors. Two examples I can cite here are the laws against email spams and laws against unsolicited marketing calls - they had a definite impact in reducing both (even in India, from where I am, where implementation of laws are often lax).
Exactly. Bot activity is a problem of volume, not all-or-nothing. Solving 95% of it would be a win.
> The only solution is regulation.
The thing why Cloudflare got invented isn't AI scrapers. These are just the latest development... the original reason why Cloudflare got created and why it experienced such a meteoric growth is DDoS and botnets.
Yes. We need regulation in the AI space. But it will be useless as long as bad actors aren't held accountable - and a lot of the bad actors aren't in our jurisdictions. You got hacked devices all over the world in giant botnets, controlled by Russia, Chinese, Iranian and North Korean actors. You got Chinese AI scraper bots as China is heavily investing into training their own models. You got Indian, Filipino and Myanmar-based scammers.
And frankly I have no idea how to get all of that under control. As much as I'd like to see sanctions against both domestic and foreign enablers of abuse (which includes residential ISPs) - it's going to be one giant ass whack-a-mole game.
And identifying a bot that is acting on my behalf. Claude go search this topic is basically the same as Googling something and clicking on the results. Human driven AI searching needs to be in a different box than AI scraping for training data.
Which sounds extremely difficult to differentiate
Hopefully it stays that way; "a bot acting on my behalf" is still a bot. At least it's often a well-behaved bot and uses a user-agent that can be detected and blocked.
Remote attestation should still be possible with a rooted phone if phone manufacturers weren't so shit. If the attestation happens at hardware level, it doesn't matter what programs or kernels you're running.
Or maybe we can actually start paying for all the things we use on the Web, making it prohibitively expensive to deploy fleets of bots.
Private invite only internets
LAN parties?!
You don't need a non-rooted phone to pass captcha checks, I have a rooted phone and can pass the captchas that ask you to scan a qr code. But I doubt phones without google services would manage.
How does scanning a QR code prove any kind of captcha?
https://support.google.com/recaptcha/answer/16609652 - it just launches the verification service.
They are not a problem unless you "believe" it is a problem. I estimate around 20-25K hits to my website from bots per day and I have all cloudflare protections disabled. Any decently optimized server should be able to easily handle that. (it's roughly 1 request every 3 seconds).
Yes and that is just the bot background radiation of the internet. I run a primary source of information site and these botnets are aggressive to a DDOS level. All to do some sort of scraping. Because they have sophisticated enough tactics to DDOS us if they wanted to. However I am not sure their objective as they have wasted enough of our resources to have scraped all our content 1000s of times over. That 25k traffic is a couple of minutes for us. And that adds up. 80-90pct of our traffic is this
Assuming that the bots aren't repackaging your content and preventing users from seeing your blog by serving that content to them first.
True. But it still wastes your server resources, right? And it's sad that you have to accept that as part of the "cost" of hosting a site ...
What resources are you concerned about? An n100 minipc should be capable of serving something like a blog at 20k+ requests/second (or saturating its network).
> keeping out bot
You can forget about it. It is not possible. Simple as that.
Let's say I'm selling concert tickets. How do I prevent bots from buying up all the tickets and scalping them?
- behavioural fingerprinting - ja4 - IP rep - queue mechanism - card country to IP country checks - app attestation - custom metrics based on knowledge of past scalpers
It's hard but it's not impossible. You can make it very inconvenient for scalpers. They need to poll at volume so their behaviour is very much detectable. A hard stance is required on IP rep, especially for more in demand concerts.
Do it like plane tickets do, tie a ticket to an identity + buyback up to a week or so before the concert in case someone wants to cancel (or authorize the transfer and capture only a week before). Ask for ID and ticket at the entrance.
I'd simply check filling speed, even with browser's autocomplete humans are slow due needing click submit.
Then when it's "processing", do them in bulk and prioritize slower users. There's huge opportunity do bot checks after checkout without affecting user experience.
Also on product launches you could add unique field which requires user to input, for example that way bots can't prepare for launches.
Yeah, this doesn't even begin to cut it
huh. no wonder my password manager's auto submit triggers bot detection (it's a fairly popular one).
Sell them via a Dutch auction. Eliminate the arbitrage opportunity for scalpers and make more money in the process.
That’s how you wind up with only kids of millionaires at your Taylor Swift concert.
So a Taylor Swift concert
Tie them to the buyer's identity, offer at-value buy-backs until X weeks before event, disallow resale.
web environment integrity
[dead]