I rather enjoy OpenAI's focus on cost-per-task.
I'm optimistic about them tackling frontier intelligence next. Astra 6.1 please?
I rather enjoy OpenAI's focus on cost-per-task.
I'm optimistic about them tackling frontier intelligence next. Astra 6.1 please?
They're doing its because they recently cut plan usage by about 50%, leaving Anthropic providing about 5x the value on their plans. [0]
The "cost per task" metric is just a weaselly way to sound like they didn't just double the cost for most users, and aren't having their proverbial lunch money stolen. They can no longer claim to provide more tokens, more capable models, or more usage in general. So they made up a new metric where "task" could be defined to mean whatever they need it to mean.
They had to become VERY specific in their wording to present any kind of value proposition that will entice people.
[0] https://newsletter.semianalysis.com/p/anthropic-subscription...
> leaving Anthropic providing about 5x the value on their plans
5x is calculated in terms of API costs, nobody pays those extreme Anthropic prices outside enterprise users.
> like they didn't just double the cost for most users
Most users aren't on their $200 plan. Most users (including on the $200 plan) continue being subsidized.
Enterprise users are the majority of the users in value. Subscribers are the loss leader hobbyists.
> 5x is calculated in terms of API costs, nobody pays those extreme Anthropic prices outside enterprise users.
Ignore value then. Anthropic still provides about 3x as many raw tokens. The 20x\200 plans provide about 10B for openai versus almost 30B for Anthropic.
Do you really think that Opus uses 3x as many tokens as Sol per task? I don't.
> Most users aren't on their $200 plan.
Fair enough. I should have said most heavy users.
> Most users (including on the $200 plan) continue being subsidized.
They continue to be fifty percent less subsidized, I guess.
Look, I don't fault them for trying to become profitable. I fault them for the weird weaselly business speak about cost per task. It's BS game playing and we all know it.
I'm honestly rooting for them, and hope it all works out. For now though I'm pretty salty that my bill jumped by 150% in one month. Gaslighting about made up metrics just makes me more angry.
How do you know? Have you done the A/B test yourself?
> Do you really think that Opus uses 3x as many tokens as Sol per task? I don't.
You don’t have to speculate. Pull up artificial analysis, pick either opus or sol and scroll down to total tokens and to total price. Opus (and sonnet) use 4x the tokens and 7x the dollars.
That difference alone removes the difference that semi analysis found.
> How do you know? Have you done the A/B test yourself?
Fair point. I just tried it, and I stand corrected. Claude uses almost exactly 3x as many tokens.
In the end Claude produced 2.25x more code, and used 3.01x more tokens (ignoring the cached system prompt, etc). It's answer was also much more substantial and feature rich - including many features I did not ask for.
I gave them both the prompt "Without searching the internet please write and test a python script that accepts a year (in any common format) as input and provides the date that easter fell or will fall on in that year. The output should be in discordian date format. You may not use any libraries."
The thing I found most interesting were the unit tests. Claude implemented the core code using both the Meeus–Jones–Butcher algorithm AND Gauss, but only used Gauss as a sanity check to ensure that it's primary code worked. Sol just validated that it matched some dates it could remember. That explains a good chunk of the almost doubled output, and it was a reasonable thing to do, though neither answer was "wrong."
Token usage looked like this (based on parsing it out of the jsonl, neither provider seems to make it easy to get real billing information per project/conversation.
People in HN live in a bubble. Plenty of real world use cannot use subscription plans (not even talking about enterprise agentic use) and is absolutely at the mercy of the cost per task metric.
We agonize over cost per task to the extreme for services/functionality built atop of APIs and can absolutely guarantee you that it is the real world metric that counts most right after “is the model good enough for the task at hand,” and furthermore that, yes, in fact, it is quite often that we see certain models/providers easily using 3x or more tokens on average for the same tasks (or even the exact same task) than a supposedly similarly matched model from another provider. Once you become intimately familiar with the thinking traces from specific models, you can preempt this to some extent with manipulations to the system prompt (essentially heading off entire thinking branches) but the nuking of reasoning traces in the name of preventing against distillation attacks has made this harder.
Cost per task, time per task, quality (good luck measuring objectively between models) are the metric that matter. I don't care how much a token costs, I'm not buying tokens, I'm buying solutions.
Exactly! I called that the Operations Equation and wrote a tiny bit more here about it: https://zero2data.substack.com/p/the-operations-equation
Once you are inserting models into non-creative (and software development is creative work) it is imperative to start creating your own small evals. Not just to check how different models perform in that equation, but to catch model drift over time.
the point is that "task" is a dubiously defined unit that the frontier labs adjust to whatever makes them look good.
Are OpenAI's models delivering 50% (or more) completed tasks than they were last month? Are they delivering 50% more than Anthropic per token?
Nope to both. Not even close.
The only way it works is if you define "task" in a way that OpenAI is good at it and Anthropic isn't.
First of all, only the $200 plan got a cut.
Second of all, barring more cuts, yes I do think the $20 and $100 plans do deliver more tasks because the models have gotten cheaper in the meanwhile (like this very announcement).
> Are they delivering 50% more than Anthropic per token?
So say the public benchmarks. Pull up artificial analysis, “Cost (USD) to run all evaluations in the Artificial Analysis Intelligence Index”:
Sol 6.1 (max): $1,082 Opus 5.5 (max): $8,708 Sonnet 5.5 (max): $7,260
>7x difference in cost for the same tasks.
> First of all, only the $200 plan got a cut.
The original plan doesn't exist anymore. To get the same usage you now need a $500/month plan. The new plan that costs $200 is different in every way that counts. It was actually a 150% price increase for people who want or need what the old plan used to provide.
Don't get me wrong, I genuinely LIKE OpenAI. It's probably worth it even at the new price. I just hate the rhetoric designed to Goodhart past the math.
> the models have gotten cheaper in the meanwhile (like this very announcement).
That is not good news for plan users. Plan users now get less of something that also has less worth to begin with. It would only be good news if they got more of it.
For example, if I buy the "unlimited car wash" package for $19.99/month and the car wash lowers the price per wash from $10 to $1 my subscription becomes less valuable, not more. The breakeven has moved, so I now need to wash my car more than 20 times before the package is justified.
If the car wash then increased the price to $50/month would it be wise for me to keep the package? Would lowering the price per wash make the package more valuable to me somehow? No, it would do the opposite.
I kind of expect them to go right to Astra 6.2, 6.1 was cancelled over the cybersecurity hubbub. I eagerly await either one.