Easily the most interesting part of this announcement is buried in the second to last paragraph:
"We're also launching GPT‑5.6 Sol on Cerebras at up to 750 tokens per second in July, bringing frontier intelligence to customers at unprecedented speed. Access will initially be limited to select customers as we expand capacity."
750 tokens/s on a frontier model is going to be extremely interesting. I doubt this new version is anything but a version bump in terms of capabilities but if we can start getting these answers back faster, they end up being more useful.
Just off the top of my head, I can think of the tedious task of finding certain functionality within a codebase. I usually can't beat an AI agent harness at this task today. If the AI model is 3x faster I have less of chance.
https://mikeveerman.github.io/tokenspeed/?rate=750&mode=thin...
This is what 750tps looks like, I guess.
You get used to it. I don't even see the code. All I see is blonde.. brunette.. redhead.
That’s an awful visualization. I can skim code quite quickly, but not when it shows up one character at a time in a small window, modem style.
At least that site should draw out a full page then start replacing that page with the next, starting from the top and working downwards, repeating each time it hits the bottom.
This is how tools like claude code and chat prompts output their tokens, so I'd say it's actually a pretty good visualisation.
Not for prefill. I suppose if you just want to imagine what generation speed looks like in the current generation of TUIs, it’s an okay visualization.
That's exactly what it looks like in the tools I use most (opencode and codex), so for that purpose it's a pretty good visualization.
> I can skim code quite quickly
are you by any chance hyperlexic? interested to hear more about this, like how fast is considered fast
Just to think what this will look like in a couple of years.
Hopefully like this (but smarter): https://chatjimmy.ai/
This is genuinely confusing to my senses. The future is going to be so strange/neat/me unemployed.
> strange/neat/me unemployed
I'm not sure if that's what you were going for, but I read it as if it were written by The Board in the game Control, and found myself with the appropriate level of existential dread.
We love/help/replace you
and I haven't played that game, so I read it in Ralph Wiggum's voice.. which also feels appropriate.
I'm in danger.
The future is totally illegible to me. I love these AI models, but I feel like I'm going to be jobless within 10 years.
Anomie is at an all time high right now.
10 years? An optimist, I see.
Yeah. It keeps catching me off guard that it answered me already.
Why is the insane speed of 13KTPS of this site is not more on the the top of the AI conversations?
Because there's been nothing to discuss since their announcement. Their API access immediately closed due to overwhelming demand and they didn't fab newer models than Llama3 yet.
Probably they will make bank selling to HFT for a while.
It's pretty well known by now.
I asked it for a block of C++ code and it hit 14,189 tok/s. I assume it cached someone else's session?
No - it's custom silicon https://news.ycombinator.com/item?id=48693490
Because I just tested it and it took 3-4 clarifications before it actually gave a correct response vs gemini/google search. It's not great, but good.
I'd rather wait 3x as long.
This caused me to have some sense what blistering fast AI actually is. What it means for the future is a question that remains.
Wow.. what?! How is this so fast?! Where can I read more?
Funnily enough, pasting your comment straight into Jimmy leads to a... Funnily suboptimal answer that does not answer the question.
As someone else already contributed, this is driven by a Canadian startup taalas that basically makes chips that are llms, so everything is very fast but also, baked into the chip. Once this kind of stuff is a commodity in like 10 years, our world will be very, very different.
Taalas HC1 AI uses Llama 3.1 8B, but takes up a massive 53B transistors and 815mm2 on TSMC N6 (nearly at the reticle limit of 858mm2). N2 is a little less than 3x as dense (110MTr/mm2 vs 313MTr/mm2).
This chip would still be 272mm2 on N2 which is an eye-watering $30k/wafer and bigger than a 9950x or Nvidia 5070.
This just isn't feasible. Some of the latest-gen LLMs seem to have 5-10T parameters or about 1000x more. I don't know that taping out just one chip makes economic sense let alone the 300-1000 chips required for a cutting-edge model. Things like continuing education so your model knows about the latest NPM packages or world news is super important, but seems like it would require new chips.
There are a TON of uses for an 8B parameter models on the edge, but this is WAY too big to put on the edge of anything. Something like a 10mm2 100m parameter voice model might be feasible on the edge, but only for expensive devices, but most of those are TSMC 28nm (up to 29MTr/mm2) or GF FDX22 (up to 40MTR/mm2) which would increase the AI chip to the point where it would absolutely dominate the BOM.
> Things like continuing education so your model knows about the latest NPM packages or world news is super important, but seems like it would require new chips.
They probably have a few ideas around that. Me, personally, I'd have one main expensive chip (replaced every 10 years, or whatever), with a secondary cheap chip in front of it that gets replaced every year or so.
The secondary chip could act the way RAG does, or perhaps both chips together can act as LoRA.
Either way, 99.999% of the knowledge is static, you just need to fine-tune the weights with that remaining 0.001% knowledge, which can be done using RAG or LoRA on a much smaller (thus cheaper) disposable chip.
The better solution would be making part of the chip cluster use something like FPGA which can be reprogrammed.
Text to speech or diagnostics equipment where the core model is relatively small and never changes seems like the ideal application. You might be able to fit something in the 25-30B range in 2nm to 14A, but it would need a way to update.
Large models are simply out of the question in my opinion. If you need 400+ different chip designs, it’ll be billions of dollars to tape out before you even make the first chip.
> The better solution would be making part of the chip cluster use something like FPGA which can be reprogrammed.
I'm not sure I follow (It's late, I am tired and I haven't had my dinner yet. That's my stupid trifecta!)
The original chip has the weights, so it's literally just a bunch of on-die (read-only) memory cells. The FPGA, while you could use it for the memory cells, would be way too expensive to use as pure memory. Typically one would hook up (read-only) storage to it, so you still need that read-only chip anyway.
The FPGA is just the compute bits, but this chip has on-die weights, not just compute.
I was proposing that the they have the base weights on a primary (permanent) chip, and have a secondary (replaceable) smaller chip with weights for a specific use-case, or for fine-tuning with new knowledge/updates to the model.
The matrices can be multiplied LoRA style, applying the matrix in the secondary chip to the primary chip, resulting in up-to-date weights through which the prompt is pushed.
Yeah, they're clearly just starting out and just shipped their very first proof of concept. But to me, their plans seem generally reasonable https://taalas.com/the-path-to-ubiquitous-ai/, and like I wrote, if this kind of thing succeeds and could become some kind of cheaply producible commodity component, I think there's huge value in that. Alas, maybe not as a frontier model replacement, but say 10 years from now you can drop a cheap raspberry pi like device in your Lan and have a fast local engine for things like text sentiment analysis, text summarisation, voice recognition, basic vision and things like that, that would be pretty exciting to me (but maybe as you outlined, impossible in practice)
There is a reasonable kernel of an idea here, but only if you dial expectations WAY back. The 10 years speculation is just wrong though. Even in 10 years, their 8B param model isn't going to be in consumer devices.
6nm is just 7nm++ and the process will be a decade old in a few months. In the decade since, we've only had a slightly less than 3x increase in transistor density and that's including EUV, BSPD, and GAAFET (which means progress is likely going to slow down even more).
Even if we hit another 3x increase, their 815mm2 design will still be a bit over 90mm2. For comparison, the entire M5 Pro/Max CPU die is just 61.7nm.
If our current progress somehow holds (not likely), even 20 years from now the 8B model would be 30mm2. You need 30 years of dead consistent progress to get it down to an includable 10mm2.
As you can see, this doesn't make sense to invest in. As to the stuff like voice recognition or basic vision, these can often fit within 100m parameter models which would be around 10mm2 on their current 6nm design. That's doable today in custom edge computing devices.
The other possible use is cheap fallback models for AI companies. Moving to N2 and shrinking chips to 600mm2 to improve yields a bit would give about 50B parameters with 3 chips plus another FPGA-ish programmable chip for continuing training and interconnects for everything. You'd need hundreds of thousands of chips produced for that exact AI model just to get costs below $100,000 per board.
That seems like a lot of money for the AI model you are essentially giving away, but maybe it still beats the power and price of GPU server racks.
the flash models have fallen in size at least between deep seek models. Is there a limit to the shrinking capacity of the models?
That’s why this stuff should be a government mega project ultimately.
It is not market viable but it is sure as heck revolutionary. Like an atomic bomb but including more… peaceful uses.
That’s exactly where government should take rein like with ISS etc. However the models are too rapidly advancing for now for it to make sense
The government isn't going to be making chip fabs go any faster which is the biggest limitation here.
The second big issue is that it takes months to fab chips meaning your hardware AI is months to maybe a year or more behind the times when it lands.
I do think it makes sense for something like a medical scanner where the model simply doesn't need constant updates, but that doesn't need government involvement to ship.
https://taalas.com/
Taalas https://taalas.com/the-path-to-ubiquitous-ai/
Previous HN discussion: https://news.ycombinator.com/item?id=47103661
Damn that is crazy.
This is the reaction every time it's posted, and deservedly so.
Not opening here... HN killed?
What
How?
Which model is behind it?
It’s pure silicon. Llama3.
hugged to death?
[dead]
I started with a 2400baud modem, I've seen how this goes
Sometimes I visualize a setup like this [0], based on 2D art by Simon Stålenhag. Someone has their home robot sitting on a desk connected to their old PC with thick cabling, dumping endless lines of each subsystem's <think> logs to diagnosis why it did something weird earlier in the day. Systems pushing 750+ tokens per second per subsystem might even be considered on the slow side for realtime tasks by then.
[0] https://www.therookies.co/entries/39513
Probably will not be looking at text like this in a few years.
Probably not. Everyone will still need a lot of reasoning tokens and tool calls. Running the tests for every round is tiring but must be done.
Imagine a Beowulf cluster of these…
That's a name I haven't heard in a while.
First post?
The user has many comments and updoots if you look at their profile.
We’re being silly and spamming Slashdot spam comments :p (“imagine a Beowulf cluster” and “first post?”)
Me too!
I always think of Furbies because of that geocities (memories!) site.
probably something like this https://sb0xw.csb.app/
For comparison, openrouter says opus 4.8 is ~55 tokens/s and fast mode is ~102.
750 tokens/s for their largest model is going to be nuts
What about 15k tokens per second? [0] I remember looking at this earlier in the year and it being so fast that it feels fake. And, yes, this model is old - but still awesome for what it is.
[0] https://chatjimmy.ai/
It’s not just old, it’s also tiny and quantized. It’s llama 3.1 8b at 3/6-bit quant. This is the type of thing you can run on almost any device…
I get that, but not at 15k tokens/s.
But it’s irrelevant. 750 tokens/s on a full frontier model is useful. 15000 poor quality tokens is much less useful no matter how much scaffolding you put around it.
You are missing the point. This is a technology demonstration on prototype hardware, and no one intends it to be seriously useful.
Their architecture has fundamental speed and efficiency advantages over GPUs or Cerebras. They expect to scale up to real LLMs by splitting a model layer-wise across several chips, which they can do without incurring any throughput penalty.
> They expect to scale up to real LLMs by splitting a model layer-wise across several chips, which they can do without incurring any throughput penalty.
I’ll patiently wait to see this in reality. Their demonstration hardware is a 250W chip that is enormous in die area for the model size. They’re making a lot of claims, but until they can deliver then it’s nearly vaporware in my view.
I’d be happy to be proven wrong, but I think they’re going to quickly run into hardware realities quite soon if they think they can just chain a bunch of chips together to achieve the same performance on larger sizes.
Why can't they do it? Jim Keller's company is also taking a different approach [0].
The simple fact that we think what we have now is scalable is basically what you are saying can't be done: " just chain a bunch of chips together to achieve the same performance on larger sizes". How do you think current architectures work? And what is being used today is all proprietary to one company!
[0] https://tenstorrent.com/solutions/llm-inference
Actually it's the opposite. Per mm of silicon it's massively less efficient and making enough chips and powering them is a major bottleneck right now. Worse, scaling to larger models requires more of our absolute best quality silicon manufacturing, where e.g. an H200 mostly just needs more memory.
I’ve been using 1,000 t/s on a near frontier model for a month now. It’s very useful for agentic coding.
It does require new approaches for me personally since I get a lot less time to think or read its output.
I think you missed the point and don't understand / aren't considerate of SLM utility.
But I’m not missing the point. If you can run one frontier model at 750t/s, then you can probably run many many instances of an SLM in parallel at a rate that exceeds 15k/s. That’s kinda the point of the flash or ultrafast variants. And they’re on something much more modern than llama3.1.
Yes, you are missing the point. 1) It's a demo. [0] 2) It hasn't been updated for 4+ months.
You don't need LLMs for everything. That is 100% the point. You can burn down the world with all of your frontier LLMs that are being used for simple queries OR we can do something faster and more efficient like this. Just because you can run a SotA model at "fast" speeds, again, severely misses the point.
And no, you can't run anything from Anthropic or OAI on-prem, so until you can there's really no comparison. If people want to continue down the path of gate-kept models with no other options then we'll all follow you off the cliff.
[0] https://taalas.com/products/
Why are you representing this as such a binary here? For SLM we don’t need the Taalas stuff at all. Just run it locally on your own device if it’s truly a small model. And there’s plenty of larger models that can be run on-premise just fine.
I think it’s impressive that a frontier model can achieve 750t/s. That’s all. You can get similar insane token speeds from other open weight models too.
The irony here is, according to you, my take is the binary one. When your response is: well, we can all just run it on our devices - we don't need any other options!
You seem to be cool with a very small and gated ecosystem with whatever tech billionaires want you to have access to.
I grew up in the era where compute was diverse and open. You may think this is OK, but it's not. The more options we have and the more diversified they are the better tech will move back towards.
I'm not the one with the myopic view here. Enjoy your "on-device" models over in your utopia of a walled garden.
I think you’ve got things quite backwards if you think that the desire to run models on device or use any of the variety of open weight models (big or small) on premise is somehow bowing down to tech billionaires. Quite the opposite really.
Once again, my statement is that the Taalas product is not a fair comparison because it runs an old outdated model. If you want to run a similar model at similar speeds (albeit not serially, but in parallel) you don’t need their product.
> Once again, my statement is that the Taalas product is not a fair comparison because it runs an old outdated model.
Either you didn't look at the page I linked or you're having comprehension problems.
> If you want to run a similar model at similar speeds (albeit not serially, but in parallel) you don’t need their product.
Except, you can't. There's no commodity hardware out there today that can run even an "old outdated model" at this speed and power utilization. Again, maybe read first and try to understand my original point?
> "...my statement is that the Taalas product is not a fair comparison..."
You actually hadn't stated this. You said it wasn't needed. Which is it?
> If you want to run a similar model at similar speeds...
You can't. Find me a single system that can run this, again, "old outdated model" at even similar speed. You're hung up on the model. The point is that if we all just stay in this wonderful world of inefficient large models we will all end up at the mercy of OAI, Anthropic, Google, etc. When other companies, like Taalas are putting research dollars in to making AI scalable, affordable and efficient. Do you really think commodity hardware is going to be attainable anytime in the near future on this trajectory? Do you need a laptop to cost $10k USD before it clicks? That is exactly how you end up kissing Altman's ass in this situation.
[dead]
I just tried it, and the answer is non-sense.
I asked it something simple, list some good indie puzzle games, and half the answers are games that don't exist. Imo quality > speed.
They baked the LLM into a CPU
at 15K tokens/s... do you need code anymore
Yeah, that's the point, right? With tool calling the LLM becomes code. So instead of asking it to write an accounting software, you can hire the LLM to be your accountant.
But you'd still need code if you need something done in a consistent way.
Not necessarily. Consider a human assistant who performs repetitive tasks at an acceptable cost and accuracy while dealing with edge cases often autonomously.
If we want reliability - we come up with processes to make it reliable and not rely on individuals getting it right. Code is a way to create a reliable process in the digital world.
For some things that's acceptable or even good. If I want to add up a list of a million numbers human assistants aren't bringing any advantages though.
Maybe acceptable in some cases but the original example in this thread was about accounting and they use software to do the counting not humans.
And even id humans/llms do it there would still be a need for systems of record with things like audit log etc.
Using gpt-5.4-mini in off-peak hours already feels like super-speed to me. That's probably no more than 100-150 tk/s. I can't imagine 750!
I've always eyed Cerebras but never had a use for it that would justify paying for the API directly. Although now that I think about it, trying out the API would probably cost less than a subscription for a month...
Try gpt-5.3-codex-spark - it's 1000 TPS and from my experience more capable than 5.4 mini.
If you have a subscription it's a different pool of usage.
Used it, very fast but tiny context window and doesn't have good reasoning. (good for quick simple code changes)
MIMO 2.5 Pro ultraspeed has a 1M window. 1,000 tok/sec is great for planning since you can have a rapid conversation with a lot of turns.
Agreed, 1000tok/s just fills up the context window (which is big by 2004 standards) super fast. But seems like 5.3-spark was just a taste of what’s to come.
2004 standards? O.o
In 2004, I took a class where we trained "language models" that were bigram word models, on an archive of a couple years of the Wall Street Journal.
I remember someone who literally announced they were dropping the class to the whole room at the end of a lecture, saying "This isn't AI!!!"
1904
Back when we were kids, we would get 0 tokens/sec _if we were lucky_
The ChatGPT subscription gives you access to the -spark model(s) in Codex which are blazing fast (but pretty dumb) which I think runs on Cerebras hardware too.
is this specifically in codex? have been trying to use the models for months on opencode then pi but it says chatgpt subscriptions don't have access to it - i was under the assumption that OpenAI doesn't lock down their models based on harness a la Claude Code
What plan are you on? It is only available to Pro users.
I have a pretty good use case for gpt-oss. The amount of time savings has actually been wild. Definitely worth a try. Just to be clear, it gets like 2000tok/s
But it seems that there is some queuing/load balancing on their side, I mean when opus is actually outputting this 55t/s it feles fast, but apart from it's internal reasoning I think there's sometimes just waiting.
Oh wait yeah good point. At 750 tokens a second and the same amount of human patients they can set it to think for the same amount of time but four or five times the amount of thinking tokens, which may improve the quality of the eventual output.
the more advanced models also utilize a lot more tokens, and a lot of these extra tokens may go towards safeguards at a higher rate than prior models as well.
not to say a speed boost isnt there but if they didnt increase tokens / s at all youd likely see things slow down a lot with the new model compared to current
I think regular users will still have the old speed, so should be easy to tell whether it is more thinkier than 5.5.
> I can think of the tedious task of finding certain functionality within a codebase. I usually can't beat an AI agent harness at this task today.
Yup, I remember "racing" the AIs to figure things out in codebases just a year ago. Today, I have no chance. Whether it is due to degraded reasoning capabilities on my part or better models, I don't know.
At least in my case, much of the code in the codebase I'm working on is AI generated so even if I have an accurate mental model of how everything works, I have no idea where any of it is located or named.
To be fair, whenever I join a pre-existing code-base [1], it's the same. I have no idea and have to map it out ;)
[1] Not AI codebases (and of course, AI code bases I guess)
I can't be the only one whose memory is so bad that I am like this in my own code base.
I seem to remember - but cannot find, even with an AI boost - someone's "law of computing" or somesuch describing the amount of time that has to pass before code you wrote is indistinguishable to you from code written by someone else. At any rate the interval is not so long.
In your own codebase you can at least run a "you simulator" to arrive at the answer fairly reliably. "Where would I have put this bit? Oh yes, of course ..."
You are not. :) my memory is disturbingly fried.
AI is always going to be able to write a grep statement faster and more accurately than a human
When AI is ready, it won’t need to grep at all. That is, it will train on the data in-situ instead.
Now start thinking, if possible.
https://www.youtube.com/watch?v=43QHhEfzz-Q
I'm skeptical of how fast "up to" 750t/s really means. Maybe if they make it extremely expensive so it frees up enough capacity?
GPT‑5.3‑Codex‑Spark currently runs on Cerebras chips and it's giving me around 150t/s. Still relatively very fast, but nowhere near the 1,000t/s they claimed at launch. (Also it's not a very good model.)
That said, I'm super bought in to faster models being better for most use cases than smarter models.
If it's 150 t/s, that's barely faster than Nvidia GPUs who are batching a lot more and are a lot more cost effective. Add in the Groq piece and Nvidia claims it can do 400 tokens/s.
Soon the bottleneck will be how fast your laptop can grep for a string.
I saw videos of coding with Mimo-V2.5-Pro UltraSpeed, which is advertised at 1,000 tokens/s, which is very impressive.:
https://www.bilibili.com/video/BV1fME16uEW7
If the time-to-first-token latency also greatly improved, this could be very useful for end-to-end in controls, like autonomous driving for example.
It’s awesome, particularly since it’s at DeepSeek tier prices (3X of DS-V4-Pro). At 1,000 tok/sec though you can really rip through tokens. (About $9 an hour if you manage to run the output nonstop.)
It tends to cost more than DS since it doesn’t seem to have as many input cache hits.
Yep this is a glimpse into the future of 500+ t/s, which is in my opinion the next big thing that validates Jevon's paradox (the models are already smart enough)
Faster tokens = more reasoning loops, so it can actually make the models smarter as well.
Yeah! So at a much smaller scale, being able to boost Step 3.7 Flash up to 40tk/s on my Spark-alike with proper triple head MTP was the thing that made it superior to Qwen 3.6 27B in wall clock time despite Step reasoning more
A lot of the open Chinese models get their results through huge reasoning loops. Being able to boost decode perf is what will make them worth it, and I’m sure OpenAI and Anthropic could do similar (if they aren’t already)
“Smart enough” really depends on how many other people have encountered a problem close enough to yours and solved it somewhere on the open internet, IMO.
Most of the frontier models can, when prompted and tooled correctly, do a lot of “reasoning” tasks that amount to resolving how the user has explained a particular widely known paradigm.
The more difficult and obscure the issues you provide them with, the faster you notice them reward hacking by altering the criteria until they are no longer attempting to solve the problem. Using “advisor” style loops helps hold this off at the cost of tokens, but there is still a fairly short limit at which they will essentially give up if they can’t find all of the necessary information - sometimes the issue is actually worse if they find a small amount of information instead of nothing - they’ll extrapolate from that tiny piece of data and generate plausible-sounding hallucinations almost every time.
And god forbid your problem involves doing something a different way than the majority of people do it. Unless you can write a full spec on it, the models will repeatedly spiral back into adjusting everything about your problem until it matches one of the most popular approaches in their training data.
> how many other people have encountered a problem close enough to yours and solved it somewhere on the open internet
I'm 100% sure that all our web, cc, codex or whatsoever sessions are used in the training, RL or either both.
This makes the size of the universe models know about at least one order of magnitude bigger than the open internet.
I think you have misused the term "order of magnitude" or just don't grasp the scale of the internet.
I get how this is a trueism now but I never really understood why it would be useful to scrape cc/codex sessions for training. The relative amount of human input for that is so low (isn't that why they are so loved and used?), how could it actually be useful to them? Wouldn't you wanna focus on people not using it?
It's more useful as a set of feedback on the model results. You can do sentiment analysis on the user responses to see if they found the model results useful/frustrating/etc and use that to guide future training
Because you provide them with the "problem" and the "solution" and once you have both you can scale your RL pipeline.
I think this is a rosy estimate. The vast majority of what people do with these models is just the same old shit, I would be surprised if 1% of it were genuinely novel stuff worth folding back into the training data.
Even if "is just the same old shit" they have much more data and of a much higher quality to scale the RL pipeline.
This may have been the case one year ago, but with contemporary models such as Opus, I run into this less often.
[dead]
I think the glimpse that is there will be exclusive access. So much for the open in openAI. If this technology really transforms society in the ways expected with inequality an unavoidable consequence equal access should be required like internet access was (isp can’t give preference to specific user traffic)
[dead]
At a certain rate we will be able to move towards continuous / real-time inference systems. The discrete, turn based solutions are quite confining with how they must be trained. Continuous and real-time would fundamentally alter the domain.
From an information theory perspective we are still in dial-up territory with regard to the actual information rate. 750 tokens per second would be a really bad dialup connection. Imagine 10 millions tokens per second.
We still have the problem that auto regressive decoders are memory bound.
The new Blackwell hardware combined with TensorRT-LLM and speculative decoding consistently can hit 1,000 TPS/user barrier, comparing to closer to ~250 TPS/user (out of 10k+/TPS on the server)
Is there something I missed, this looks more like 14.4 to 56 on a 64kbps backing channel modem story. I have no doubt that there are still massive gains to be found, but they seem to be using existing constraints more efficiently, not that fios is coming.
I don’t have the budget to work on the foundational model scale, but with a draft model 10x–20x faster than target and an 60-80 acceptance rate I can see how they could promise 750/TPS (with a lot of other hard work) but I would appreciate where I should look to figure out what I am missing.
agree, from my POV the constraints are still there but we've optimized now. still haven't solved the core problems.
1000TPS - what model size?
Maverick 400B is what Nvidia used for their claim of 1k+ TPS on Blackwell GPUs.
Is there anyone exploring or writing about this in public? I've felt for a while that the turn-based model was not quite right, but also felt too stupid and ill-informed to have much of an opinion about what else it could be.
Thinking Machines, the started founded by former OpenAI CTO Mira Murati. The interaction models demo’s in their videos imo breaks the awkward turn-based barrier. Returning responses quickly reaches a threshold where it starts to feel like a natural conversation. Their approach to solving this problem is rather clever.
I have an active 'sleep' mode, where when the user is AFK the LLM goes into a loop with a sleep 10 between turns, and determines (via tool use) if something should be done. That's still a 'turn' in a way, but it's all the LLM just sort of sitting around like a human would, pondering what to do next.
But I could imagine after each space(eg, word) having a 27b model on a nice rig, with thinking off, doing a quick look at the sentence and determine if it should interrupt and start a real turn with thinking on. Which kind of is non-turn based in a way. If you're typing fast, it might hit that run every 3 or 4 words, but that's sort of how a human might be when a person is talking to them. That is, waiting for enough info to interrupt, if needed.
There might be a way to process chunks of a sentence using commas as break points, eg for comma delimitated phrases in sentences, so the whole sentence doesn't need to be re-processed each "should I break in" assessment at word break.
Could be fascinating. Could actually do some of this right now.
I don't think this is what the parent poster was thinking, but the idea even at this level seems fun.
Yeah, I've played with some similar stuff on my 9070xt. But ultimately all the ceremony on top is cloaking that it's still just two or more models taking turns prompting each other to give the illusion of continuous thought. It's still one thought at a time, with every thought starting from scratch with a big chunk of prior context.
The idea of true continuous thought and memory-generation is very interesting, though I can't even begin to conceive of how it would work.
Or if it's even correct? Maybe our brains are secretly actually turn based too?
I think they're definitely attention based. They're just immensely faster than LLMs, because a lot of processing is in silicon in a sense. Think of a ball flying towards you, you don't have to think, the data is handed to your conscious mind, speed, direction, which literally knows how to snag the ball out of the air.
But we have multiple things vying for attention, and some are immediate. Being on the phone talking to someone with great attention, and then touching a burning surface -- you immediately pull your hand back (lizard brain) before even being aware you're doing it. The same with peripheral vision and something surprising coming at you from the side. It snags your attention.
So maybe we are turn-ish based, but just multiple parallel processes each with their own turn? Neurons have their own 'trigger', and I think the brain has layers of triggers, each aggregating and filtering up to the top which then triggers.
I think doing this all with an LLM is silly, some of it should be innate, such as peripheral vision. Data handed to the main thread when triggers occur. I wouldn't want an LLM to handle "walking" fully either.
Some octupus have a sub-brain in each tentacle, each thinking and feeling, there are serious questions as to what its mind is like. I feel initial LLM powered androids may have to be like this a bit.
I agree with you however I think even then you're still giving our brains too much credit. The speed definitely comes from that processing being "in silicon".
Your ball throwing example however will be handled by really small and really fast "fine tuned agents" dedicated to catching that ball. Eyes to motor neuron system. There are the illusion of free will experiments that demonstrate your brain only rationalises and explains whatever activity took place after the fact (It's explanation may even be entirely wrong).
That would be interesting.
Do you feel most of the speed upgrade will come from the software or hardware side?
And more importantly those 10 million tokens/s should cost fractions of a penny. Tokens need to be dirt cheap so I hope they build out massive solar+battery powered data centers asap.
No anything but wasteful, weak, expensive, environmentally harmful solar. Nuclear is the only path forward for superior energy production, at least until we figure out fusion.
How is solar any of those things?
Your comment made me think of another real time. Real time, dynamic code/apis.
Imagine a world where there is no code, just things mildly handshaking and then creating data APIs on the fly. Where communication is fuzzy and locked in on an individual basis. No years of RFCs, no RFCs at all, just... data.
Just data, man.
An API arbitration aberratically assigned at authorized access, abridged and annotated, analytically assuring absolute assurance.
Why remove the code and binary artifacts, though? Don't you want to verify that the business logic is accurate and the processing is deterministic?
In some circumstances there is no substitute for something that you know will produce the same answer for a given input, consistently. And that's before even considering the watts per response.
The AI is the business logic, and the processing, and all of it. The context window is effectively infinite, with layered context window depth and speed.
Think of short and long term memory, or think of RAM vs SWAP. Dip into swap to pull needed data into RAM context. SWAP can be anything storage related, including a symbolic database or a best-encoded set of priorities.
If a person knows 100 knots, but hasn't tied one in 23 years, they might have to think a bit before they get full use of their long term memory... and tie that knot. I don't see an issue with layered speed context, that is, GPU ram, slower RAM, DB storage, all in the same format.
Imagine a world where a 'factory' is just high-tech 3d printing, with a dozen different methods (eg, plastic, laser+metal, etc), and getting specs for everything possible is, well, an immense amount of work. Imagine having a billion item catalog of things to print, and, imagine new requests for new things to print.
And the request doesn't come from an expert, but from some dude who sketched something on the back of a cardboard box.
The LLM can pull from long term storage for how those things were done before, how similar things were done before, and just get to work.
Regardless, the connection was what I was talking about before. Data transfer. Do you need http? json once established? What? Imagine instead that's all in the wind?
And it's so fast, so capable, that dynamic is easy.
Feels like the universe did that and life spat out. Theres going to be a structure
It's very easy to see how world changing this technology will be. In a few years these AIs are going to be negotiating how they communicate with each other. Humans won't necessarily be included in that negotiation unless we have some kind of specific reason to. So many communication layers are going to be opaque to humans. We just have to trust our AIs are communicating efficiently and safely.
It will be fun running into this scenario where it's run without democratic control, be proprietary and for profit.
I'm pretty sure the LLM will get fed up and start writing an RPC
Also > An API arbitration aberratically assigned at authorized access, abridged and annotated, analytically assuring absolute assurance
Cool that you wrote all the words starting with "a" but I don't understand what you mean
What this made me think of is life before computers, where people mildly handshake, create agreements on the fly. "Where communication is fuzzy and locked in on an individual basis."
TBH, to me, this imagined future looks a lot like it'd have all the problems we already have.
I made this https://github.com/alehlopeh/hallu
Neat. Not precisely what I was thinking, but 100% definitely very cool and the same mental scope. It's like we wear different shoes, but go to the same cobbler.
I can imagine shoe-horning* this so the agent saves prior builds of every successfully delivered or deployed item. In my example, perhaps if someone orders new design $x, it's shipped, and review is 4+ stars, it gets added as 'successful builds'.
* have to keep with the shoe theme, even though shoe-horning is not really necessary
Wow. Sci-fi stuff!
I’ve thought about this before. No flaky config files, no updating endpoints, no status monitors. Just fuzzy everything that works almost all of the time.
Ahh yes slop at the speed of light, how useful!
AI is improving and seems to be reaching the point of not being slop (I am talking about flagship models).
If you’re still calling it slop at this point you have an axe to grind.
Do you use LLMs for anything but code?
I do, a lot of things, it’s extremely useful.
bean in mind that "GPT‑5.6 Sol on Cerebras at up to 750 tokens per second" not necessarily means the same model (in terms of inference result). It can mean anything like a very quantized model, a different level of model activation per inference etc.
Of course we can trust that wouldn't name the same thing with different levels of intelligence, right? Right?
yeah but it’s trivial to just try it out and compare.
I still use GPT-5.3-codex-spark which also runs on the Cerebras chips. Spark can run at >1000 tok/s but it's highly limited in it's context window size so it's not suitable many workflows.
Granted this will be a bit slower (relatively speaking) but it will still be awesome.
Same - I had some "AI-assisted coding interviews" where I had to bring my own AI tools, and found the speed of codex-spark to be important for making progress quickly (and not sitting there waiting on Opus to think for 10 minutes).
> second to last
There's a word for this that you should never pass up an opportunity to use: penultimate. (You should also never pass up the opportunity to use "defenestrate," but it sadly does not apply here.)
A friend of mine had his visa accepted because of this. He was explaining what he plans to do in US and he threw in “penultimate” into a sentence somewhere.
The council stopped him, said that if he knows such words he definitely won’t overstay his visit to work as a dishwasher, and accepted his B1/B2. Seriously.
Not sure if it would be the same if he used “defenestrate” when talking about his plans.
This is a strange one. We know the hardware capabilities of Cerebras force them to do aggressive REAP pruning to serve Kimi K2.6. Meaning that about 750B parameters is the upper limit of what they can serve economically. Not sure if this means Sol is smaller than anyone thinks or that they're just going to charge so much that a very inefficient serving regime is feasible.
This is something Xioami already did with MiMo-2.5-Pro a month ago, and at a higher speed (1,000 t/s).
750 tps at GPT-5.5-Pro prices would be ruinous!
Last I heard, Cerebras chips were entire wafers and would be extremely expensive. How could OpenAI possibly have enough of these to serve a popular model at scale?
Cerebras is Milli Vanilli. They spend 10 years burning cash on a failed idea (which is frankly insane, since they should have figured out the limitations of heir stack in like... a weekend) and struck accidental gold with their 'Giant ass wafer'.
The company is valued like they broke open the grail, when in reality it's more like they bought a Cybertruck, got it stuck in the mud, and realized "You know what this thing does better than all other cars... shovel mud"
I'm shorting Cerebras with margin to virtually zero.
This would be amazing for some of our "real-time" workflows, that need to fallback to AI for one reason or another. What used to happen is a rules based system did the majority of work, and occasional corner case would fall back to humans. Then we moved AI in, still not real time, but much faster. Cerebras could make that even faster.
It all depends on the context window size. A small context size with fast performance won't be very useful today, as most workloads (like requests behind codex) usually have very long context.
OpenAI also announced two days ago that they're starting to make Cerebras style chips themselves [0], will be interesting to see how fast SotA model inference will be by the end of the year.
[0]: https://openai.com/index/openai-broadcom-jalapeno-inference-...
I don't understand how you refer to this as "Cerebras-style". Cerebras is wafer-scale and unique. Jalapeno is an inference-optimized conventional chip.
Cerebras is different than what jalapeno is.
Jalepeno is for mass scale inference.
Cerebras is extremely expensive and difficult to scale, hence the limited release.
Even if their chip is a difference maker, end of the year is wayy too optimistic. It’ll at minimum be a multi-year effort to bring it to production at scale.
I don't see any indications that OpenAI is doing wafer-scale work.
I tend to doubt they would. Cerebras notably doesn't have a kv, is wildly high bandwidth, but within/across the chip, not able to dump/restore kv super well. I doubt openai is going to build something that is as expensive to run. Also, wafer-scale is absurdly hard & weird to pull off, so I doubt that would be their first foray.
At thousands of tokens per second, LLMs (harnesses) can start to do a broader tree search of possibilities even in inefficient token space. This unlocks capabilities outside programming.
The speed sounds great，faster models make that gap much more visible..
3x faster burn than 3x expensive token, generate more tokens, more fees
this means they also earn at a faster rate in some setups :)
Does the Cerebras variant offer input caching and corresponding discounts? Last I checked Cerebras would not cache or would cache but not give discounts for the cached input, making it impractical for agentic use and multiturn conversations.
"we can start getting these answers back faster, they end up being more useful."
Dude, 10x token speed is going to be absolutely nuts. Half the "parallel subagent workflow" business seems to be driven simply as a means to avoid tapping your thumbs waiting for the infernal robot to finish something. If things come back speedy quick all the time, it should keep up with the "speed of the human" and let me stay focused on one thread instead of half a dozen. Plus the cost of screwing up gets significantly lower because you just re-fire with an adjusted prompt and iterate.
Someday these things will be 100x as fast as they are today and that is when things will get insane.
it also makes the parent brain-dead because all those subtokens are missing from the context thus unable to steer the hyper dimensional context driven generation, and the subagent is dumb as a post so synthesizes something very weedsy while you're specifically attempting to understand the forest
You have an agent spawn the agents for you! You can ask Claude to do it for you, he is happy to use sonnet when you ask for grok and opus high when you ask for deepseek.
> I usually can't beat an AI agent harness at this task today. If the AI model is 3x faster I have less of chance.
Yes: we have these new tools that are extremely good at helping us search through our codebases. Not just to find where/how functionalities are implemented: IMO bug searching is even way more powerful.
But: why would you want to compete with AI to do that? I cannot compete with grep/ripgrep... And I'm cool with that.
This lets you focus more on the more interesting parts, where AI/LLMs suck fat balls.
From what I know about batch processing/ concurrency in inference this is a pipe dream... Or its going to cost an arm and a leg. I think they're lying or its going to be a much smaller model and not "frontier"
You have speculative decoding that easily increases speed 2-4 times with no loss of quality, and of course MoA architectures that speed up inference 10 times or more, although with some quality loss.
Better hardware, and other techniques on top of that and you speed up even further.