I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration.
> In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a single working day, achieving comparable accuracy nearly 7× faster.
This is actually insane.
Hopefully the release ultrafast of Terra and Luna too.
I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency.
Dollar for tokens, Sol and Fable are the same price.
However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task.
We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day.
And, due to less guardrails, Sol also performed better, and we lost less tokens due to guardrails shutting down sessions (I feel like it’s illegal to take $50 of someone’s token money and then shut down a session with guardrails before they get an answer, and yet Anthropic do it to us constantly… either take our money and commit, or trigger the guardrails immediately)
We’ve literally saved tens of millions of dollars already (no exaggeration! already 8 digits) by switching to Luna for many workloads at my company.
The amount of workloads we can shift with an advisor model pattern continues to grow.
It’s seriously amazing.
Luna came out about a month ago, you're saying that the cost saving from switching to Luna has saved your company $20 000 000+ in 1 months spending on API usage?
OpenAI and Anthropic are believed to be making billions per month in revenue selling tokens to enterprises. It’s not that surprising for enterprise SaaS they would have individual customers making up ~1% of their total sales. Although, perhaps a bit more surprising those few customers are posting about it here.
Another interpretation would this is a counterfactual savings, like they previously paid $1M for y tokens, and now that tokens are cheaper they increased usage and paid $1M for 20*y tokens.
A second angle on the counterfactual savings would be Luna telling them not to pursue a potential session with the expected/extrapolated (from which sessions they did ignore Luna on, e.g. just to keep efficiency statistics current) sunk costs at time of getting shut down used to derive the quoted number.
A third angle is that it is not true and that's just some attempt at influencing an HN thread, which is happening quite regularly pro and against AI. We know that the pro-AI has a way bigger budget to burn with this kind of operations tho.
It makes sense at a company that sells AI as a service. Perhaps to make presentations...
The ones with terrible slides on their MTA ads?
Canva generates billions in annual revenue, it's not unbelievable
By "advisor model pattern" you mean https://claude.com/blog/the-advisor-strategy ?
How do you implement that outside of claude code?
the easier way to do this, with any harness, is to use an expensive main agent that is told to delegate all code reading, writing, exploration, research etc to weaker subagents to conserve tokens. It's an inversion of the pattern but the resulting split is the same
Yeah, that one is easy, but I was thinking of the other way around, where the weak model calls stronger one.
I append this to many of my opus claude code prompts
`You may use a Fable subagent to answer questions, solve problems, and provide an adversarial review of your ideas and code`
You can use a similar pattern in most any harness, and you can tell them to use other harnesses. In claude you can write `Use codex cli to have Sol56 Xhigh provide an adversarial review to your plan before presenting it to me` or `Use opencode cli with GLM 5.3 to verify all code review findings before presenting` or whatever you're doing, as long as those other tools are setup and ready to be called.
IMO: This isn't useful as a token saving pattern in my experience with agentic engineering, but it is useful as a quality-enhancer.
what has a single company accomplished with tens of millions of token spend?
A pelican on a bike accurate to a subatomic level
But the knees still bend the wrong way
higher valuation
What are you using it for?
[dead]
Evidently, Claude's tokenizer vocabulary size is ~15k[1]. On one hand, it's quite mind blowing. On the other hand, Anthropic models' token (in)efficiency makes a lot of sense in that light.
[1]: https://xcancel.com/magikarp_tokens/status/20878591737488549...
Not just that, they normalize everything into lowercase and use a special character to capitalize words (what about languages with non-trivial normalization/capitalization?) and mark beginning and end of each word, all of that diluting already small vocabulary. That smells like manual tuning of what should be done statistically, I wonder what technical merit they saw in that - I know they mentioned better generalization, but this is pretty counterintuitive.
Sadly it's not too counterintuitive; remember the old "how many r's are in the word strawberry"?
Also different tokens for the same named entity/concept if they almost entirely exclusively occur in non-overlapping contexts, and are themselves rare/uncommon in the first place, will result in behavior that's similar to the speech/phrasing/vocabulary registers humans exhibit, where the aspects of the named entity/concept get largely compartmentalized.
The most severe case along these lines were the old BERT models that ran over straight UTF-8 bytes (plus a handful special tokens).
But for the modern post-GPT2 LLMs such radical simplicity seems to mostly not be considered suitable. Note that CJK (the big one in particular, so Chinese semantic and Japanese Kanji) encodes each one into multiple UTF-8 bytes giving some automatic scaling for semantically dense languages; similar effects also apply to e.g. APL code.
Sol is way cheaper than Fable by the token.
Wait isn't Fable like 2x more expensive if we compare under 272k tokens
The comment you're replying to almost feels like it was written by a bot or am I crazy?
Peak under your skin a bit. Something weird is going on. I think we are bots/robots(sic)
Related, Under The Skin with Scarlett Johansson is an incredible movie.
[flagged]
Peek*
beep boop, everyone thinks I’m a robot. :|
What kind of bot would say `less guardrails` instead of `fewer guardrails`? I guess someone could instruct an LLM to deliberately make mistakes, but isn't that too paranoid?
We’re reaching transvestigation levels of people trying to spot AI text everywhere they look
Hmm, this sounds like something a bot would say to prevent being caught.
...Does a bot know the word "transvestigation"? Would a bot be allowed to say it by its corporate overlords?
It does now that is on hn. Thank you for your input.
15 years ago i was astounished by the intellectual deepness of this community. Now i'm aware this has always been a cult and their former cult leader Mr. Altman wants to destroy this capitalist society. He isn't even hiding motives. People just stopped listening carefully.
In some romantic languages less and fewer are the same word, so it is a common mistake for people that have them as their mother tongue.
In German the distinction is also different from what English speakers sometimes make.
“include common grammatical imperfections and awkwardness common in casual message board interactions.” I’m not saying that’s what’s happening here, but a high school student told me that’s basically what they do to make papers not sound like AI.
I agree. Weird to use <“> and <”> characters. Maybe written on phone, but certainly not on keyboard.
Other languages use different characters for quotes, if anything that's an indication that's not written by a LLM because it's not favoring the standard English character.
https://en.wikipedia.org/wiki/Quotation_mark#Specific_langua...
Written “on an iPhone” - yep, seems to automatically switch the quotes.
Common on non us keebs though.
You could say the token usage is "load-bearing".
Its token usage has earned its keep.
One thing I've deliberately deferred is drawing conclusions from all of this.
[flagged]
Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. It would be more meaningful to know how much time was required for a single complete answer to a difficult HLE question.
The animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel
> Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out.
I believe the assumption of the benchmark creators is that it's meant to measure the sequential speed that a single instance of the LLM and piece of hardware can get through all the tasks from start to finish, like running a race. As a rudimentary comparison, sort of like doing prime number calculations as a benchmark of the CPUs in one bare metal server. Of course you'd get a speedup in total number of primes searched if you ran the same software of GIMPS on 8 servers with the same hardware in parallel rather than 1 server.
Of course if you took all the individual questions in humanity's last exam and fed them in parallel into separate queries to Claude that land on separate hardware instances of the claude model you'd get a speed up. Because each question is independent and not related to knowledge/calculations that are performed in any other question it is indeed very open to speed up by breaking it into separately dispatched parallel tasks.
>Answering 2,500 independent questions is an embarrassingly parallel workload
So is calculating the total time required to answer all of the questions.
Amadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.
Yes. but for embarrassingly parallel tasks it barely matters.
The task the majority of people here care about (and which makes up a huge part of AI lab revenue these days) is collaborative coding with a single human which is not embarrassingly parallel.
I am aware. I was talking about HLE. Talking about how fast you can run through HLE in a serial manner is like talking about how how fast your non-simd single threaded cpu ray tracer is. Its simply not how any sensible person would tackle this problem.
[dead]
Maybe not embarrassingly parallel, but can be surprisingly parallel.
Such a deep and smart insight.
But why is that important if they're measuring time?
And just like that, GP disappears forever ...
750t/s tells the story. This isn't purely (or even mostly) driven by parallelization.
i assume they meant serially....
Presumably when they're talking about benchmarking, they're using a fixed amount of resources, like everyone else does when benchmarking?
I'm finding Luna suprisingly adequate for my work. I slept on it due to the benchmarks, but it's very fast and even on low reasoning I'm finding it more than adequate for "menial" work. (The speed is crucial for "interactive" work -- if a model is fast enough it goes from "async" to "real time", subjectively, which is a huge difference.)
In fact, I'd say it's overqualified for the kind of work I'm doing, because it spends >half the time verifying trivial changes (and the verification isn't as helpful as you'd expect, even with bigger models).
Maybe I can prompt it to be less aggressive about that (the new GPT models do it even without prompting).
Anyway, Ultrafast Luna would be amazing, though I strongly doubt they can offer Cerebras at anything approaching the current prices. Now we wait for Moore's Law? :)
Luna is incredible considering the cost. It’s very good for routine work.
Fast models is why I was hoping Taalas would get their butts into gear and eventually release a consumer priced card. I'd love to have a pcie card that screams along at 15k t/s even if on a heavily quantized 2026 level model forever.
Faster & cheaper tokens = more reasoning capability and more reasoning = better problem solving as far as I have seen.
Luna and Terra are great, but so are DeepSeek Flash, MiMo, Qwen 3.6, and even Gemma for some things.
MiMo Pro has had UltraSpeed for a while.
I'm considering a home project done in Rust. I've not even touched Rust before, I'm a Java dev. It'll be something reasonably complicated, using a bit of unsafe etc. Do you know how Luna does writing Rust code?
I feel odd to use these models, because it feels like a faster model doesn't feel that much faster if it spends reading files, making edits and running checks.
It feels too situational.
Which one of those takes time?
Luna has impressive results on DeepSWE.
I mostly split work between Luna and Sol. If something seems simple enough I always try it with Luna first.
Sol medium/high planner orchestrating -> Luna xhigh subagents doing implementation
...has been REALLY good for me. Even on xhigh, Luna is crazy cheap.
Subjectively I'd say it's way better than Sonnet at a fraction of the cost. Luna xhigh can do some decently challenging things on its own, but when orchestrated by a model that is actually good like Sol, I am finding it very very nice.
How are you doing orchestration - using sol for plan mode in codex? Or some other pattern/harness?
The cool kids have custom harnesses and workflows and stuff, yeah. I'm still using Superpowers in Codex. Planning in Sol, Luna subagents. https://github.com/obra/superpowers
I feel like I could be doing a lot better somehow. Regardless though Luna (xhigh specifically) is super good/cheap/fast for a lot of things
what about you
Feels like the 90's again where single threaded speed is improving fast. ASICs and wafer scale rather than node shrinks, but end result to me the consumer feels the same.
I wish, but I don’t think this qualifies as single thread performance. Cerebras is whole wafer computation though right? So you get the result sooner, but the same amount of computation has occurred, just with huge parallelism.
Yep, https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law
Output from Cerebras with GPT model is 750 tokens per second.
Don’t blink.
(Chatjimmy has 14,200 TPS.)
ChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.
At such output speed, I wouldn’t expect reasoning. (But I didn’t know it, thanks.)
700 TPS with reasoning is awesome and it speeds things up.
Cerebras as public traded company is worth keeping an eye what they produce.
> At such output speed, I wouldn’t expect reasoning.
As the sibling comment to yours mentioned, if they had a reasoning model “hardware-ified” onto a custom chip (as is their plan for IIRC this or next year, a new ASIC), it’d output fast decode speeds for the regular output as well as reasoning sections. Both would be ≈equally fast.
Yeah, I thought reasoning was literally just chain of thought in the output token stream, with the model itself adding delimiters to indicate what part of the output is internal reasoning, and what part is an answer to the user. Is that wrong?
No, as the sibling comment mentioned, your understanding was correct there.
What’s more, the only technical difference in speeds could be, and likely also is with the HC1 chip, between prefill (prompt processing) and decode (text generation) speeds. I don’t know whether it’s the case with Taalas’ chip, but in the “software-based” LLMs we typically see and use so far, those two stages hit different parts of a computer (processing/compute-bound vs. memory/bandwidth-bound).
You are right, reasoning is unrelated to tokens per second.
Anything will be fast if you etch it straight to silicon
The knowledge of ChatJimmy is terrible. Even Qwen on my iPhone is better.
ChatJimmy isn't a model, it's Llama 3.1 8B hardwired into silicon. The point isn't to be a good llm, but to showcase the speedup that's possible
well, yeah, it's based on a 2+ year old tiny model. It's very much an alpha proof of concept that they can perma-bake an LLM into silicon.
https://huggingface.co/meta-llama/Llama-3.1-8B
Haha, at first I thought you meant that the knowledge of the existence of an LLM that’s so fast is terrible because it’s ruined every other LLM for you!
Never heard of it before, that's fucking insane.
Apparently they baked the Llama 3.1 8B model weights [0] into silicon (the actual hardware is called Taalas HC1).
I guess for the trillion parameter models this would not scale due to cost? Imagine buying GPT 6 in the form of a PCI-E card, pulling these speeds, with up to 120 cct agent sessions. It'd be beyond wild.
[0] the weights are also using some cut down small format, but HC2 will have regular FP4 supposedly, and support for 20B params on one die
> Never heard of it before, that's fucking insane.
They've been acquired by AMD. Those saying the model sucks are completely missing the point: it was a proof-of-concept.
The question is: what happens to a model like Anthropic's Fable 5 that does, what, 70 tokens/s (and requires lots of output tokens) when the latest open-weights model is etched on silicon and does 14 000 tokens/s?
Shall the better model still have the upper hand or will the raw speed compensate?
Ralph loops automatically get real real interesting again. 200x the iteration speed. This is such a clear win I feel like there's hardly anything to talk about. Instead of one stubborn iterating idiot, you could have dozens of idiots competing in parallel, genetic algorithm style.
The other common orchestration pattern I see is "big model for planning, small parallel subagents implementing, big model reviewing" Today it's Sol dispatching a handful of Luna subagents. Tomorrow maybe it's Sol dispatching as many Fred subagents as it could possibly want.
But what patterns have we not even thought about yet in a world where subagents are 200x faster/cheaper?
What if instead of dispatching single Haiku/Luna/etc subagents, we dispatched "teams" of Fred agents? Maybe each team is 8 Freds. Five come up with competing ideas and the other three vote on a winner.
Or what if they were heterogenous teams? One Luna and a bunch of Freds.
What if instead of a two-tier orchestration system (Sol->Luna) it was three-tier or n-tier? (Sol->Luna->Fred->...Fred)
Those ideas overlap a bit, and crazy shit like Gas Town has already explored even wilder ideas I guess. But man, 14000 tk/sec opens up so much stuff.
At 14,000t/s that's effectively a motor cortex for an android, you no longer need to train the robot to walk, it has a general idea for how to walk (baked into the 1b model), and then just corrects based on sensor input, in real time.
I still personally think that a heavy lean into MoE will be better for that sort of thing. Our brains are subdivided into large parts but I'm sure (and I'm not a brain scientist) that those parts can be subdivided even further into systems that run at various frequencies and latencies depending on what they're used for.
I was thinking about it the other day actually. How our brains evolved structure. I imagine it was purely just down to evolution adding/clustering additional cells around the areas where additional cells were needed. And after long enough a natural brain architecture emerged.
Makes me wonder if we're on the right track with transformer architecture/attention but if it'd be more effective on a larger scale, like MoE with a billion "experts".
I am not a primary source, probably not even a secondary or tertiary source, so take this with all the grains of salt.
Oh for sure, but I think generally multiple experts are selected in an MoE pass for a token, so presumably it'd select programming related ones as well as general knowledge/language.
Only problem would be the routing layer works on the previous token as far as I understand so it might need more informational depth than just "a token" to select experts, I suppose in the same way attention works.
I wish I had the GPUs to run those sorts of experiments ha ha.
Yes, imho 14k TPS is just a beginning.
This Gas Town? https://github.com/gastownhall/gastown
Unfortunately AMD bought them, so I don't think we will get to see another release from them.
Aha, thanks, that’s fresh; press release is from Aug 6
https://ir.amd.com/news-events/press-releases/detail/1296/am...
Been waiting since Cerebras-GPT. [0]
[0] https://news.ycombinator.com/item?id=35490837
An irrational gripe of mine is how GPT uses 7× instead of 7x.
I recognize that the former is the multiplication symbol, but I don't think it should be used that way.
If anything, writing "7x" is actually just a poor emulation of writing "7×".
Saying something is "done at 7x speed" should be read as "done at seven times speed" not as "done at seven x speed". So using the 'times' (multiplication) symbol is the better form in my opinion; it just happens to be significantly easier to type "x" instead, which is how we got here.
But speech evolves. And we all pronounce "ten ex", not "ten times". Actual speech usage very clearly pronounce the letter 'x'.
So it's 10x. And no need for Unicode codepoints.
I fully agree that speech evolves, but maybe this is just in the US? I've never really heard anyone say it here in Australia, it's usually "times", although admittedly almost no one uses the phrase/format here.
Ah, yes. My gripe is certainly US American flavored.
A number plus an “x” is used beyond Trumpistan. I have never in my life heard anyone say “ten ex”.
Have you heard the expression "10x developer"?
Interesting I always thought that "x" (whichever) is used informally for multiplication and proper sign is dot: 3 · 4 = 12
It looks like there is a difference between English speaking languages and the rest in that regard.
Funnily enough, I didn't even have the speaking aspect in mind.
I don't like how the "times" symbol floats off the line -- it's a visual thing for me (again, irrational).
Seems like they will do Sol first while capacity constrained? I can't imagine the margins they'll be charging
I think this is what they need to actually have a margin :)
Isn't ultrafast just making hundreds of subagents?
No, that's Claude "ultracode"