Am I understanding this correct in that you can basically automate monetizing your web/api content to everyone or just agents ? Because I would be very much in support of charging agents per request, but I would want to still offer humans a free experience.
Depends on the website though. I want LLMs to scrap my B2B website, because then it's shown to the user and they will likely use my product afterwards
Their example of an /api/premium is quite nice! You could you like keep existing pages free, but provide specific output content for llm!
So if: cost monetized API < cost configuring scraper for your website OR feature provided by premium api > data got by scraping, then some people/business will likely pay
If not built-in, you can probably put it together through Cloudflare itself.
If a request goes to the protected path, if detected as bot: hard HTTP redirect to the path set in the monetization gateway, if human: allow and don't redirect.
Is there actually a reliable way to differentiate human from bot?
There are reliable ways of differentiating human from cheap, bulk scraping bots.
But if the bot is advanced / expensive enough, it gets a lot harder. Where this product's market sits is in giving a paid way to access content compared to having to spin up bots that run js, from real IP addresses, etc. all of which are more expensive
Agreed. To me this feels like the perfect solution for websites and ai crawlers. Instead of having crawlers paying proxy services and captcha solvers, they can pay the website itself. As a web scraper, I'd happily pay the website provider to get access if it meant easy access to the content. Heck, as a human, I'd pay to avoid the dumb captchas.
[dead]
Unless you have people's biometric data, you won't be able to separate agents from people. Except by payment.
Agents will be able to pay orders of magnitude more than humans, since they can just cache the documents at openai or anthropic, then use them over and over.
But then the cost to access a HTML page will also have to be thousands of dollars, since it can only be sold five times. (Once to Anthropic, once to OpenAI, once to Google, once to Meta and once to Apple).