People underestimate the importance of speed on quality of thought, because people underestimate just how much quality is a result of simple iteration.
When an LLM thinks, it typically just makes one pass. It outputs tokens from top to bottom, beginning to end, and then it's done. But when people think, especially strong thinkers, we typically iterate and revise our thoughts on the fly. We do numerous passes. We stop and restart, we reconsider, we review, we reevaluate. Sometimes we do this so quickly and automatically that we don't even realize we're doing it. I think a lot of what separates a highly intelligent or effective person from others has less to do with the quality of their first pass and more to do with just how many additional passes they're able to do in the same amount of time, and of course what kind of criteria they're habituated to consider during their review passes.
Introspecting about this is difficult, but experimenting with LLMs is easy. First, simply ask an LLM to do something complex. For example, to come up with a new business idea, or to plan the next month of your life, etc. After it finishes, tell it:
"Review what you just wrote, according to some appropriate list of evaluation criteria that you come up with first. And then, based on the results, iterate and generate a better response if warranted."
It's insane how much better the next answer will usually to be. Often it'll catch and erase tons of hallucinations, logical errors, and inefficiencies. And you can simply copy-paste this again and again until you begin to hit diminishing returns. Or, in a harness like Claude Code, for example, I might shortcut this whole process by saying, "Use sub-agents to iteratively review and iterate on your work until convergence."
The reason why most people don't prompt LLMs to do this (besides simply not thinking of it) is that it takes time.
But what if it didn't?
What if the LLM's response came back in milliseconds rather than minutes? Then there would be almost no reason NOT to do this. In fact, one could almost imagine it baked into the assistant/harness -- a massive step change in practical quality, enabled by nothing more than speed.
Going back between two different company's AI tools when facing a tricky architecture question often surfaces holes in an approach I'd been building.
Similarly, if I ever get a bit too vibey and don't carefully review code changes myself, the blast radius is generally significantly resolved by a carefully tuned "did you consider x, y, and z" skill after a first draft partnered with a "deploy an adversarial review agent for the worktree".
When I was doing some important work, I had 2 worktrees, each running a frontier model from a different company, and I passed their responses between them in markdown files and had them critique them. I thought they'd prefer their own solutions, but they frequently preferred the other model's solutions.
I went back and forth like this until we converged on a solution "everyone" was satisfied with.
---
Context: I'm a solo dev and working outside my area of expertise, so I'm leaning heavily on LLMs. It's not great (definitely too vibey for my taste, and I keep running into issues) but the alternative is spending the next few years studying several specializations instead of shipping. So this is the "least bad" thing I could come up with.
I'm definitely increasingly making time for "learning sprints" to catch up on specific knowledge gaps. For example today I spent 2 hours debugging something because I was missing a fact that would have taken me 1 minute to learn...
I went from "fully hand crafted" to "fully vibed" (when Fable came out), back to "fully hand crafted" (once I realized I no longer understand the code!), and now I'm at "making very careful use of AI, asking for the smallest possible changes, and double checking everything"...
> I thought they'd prefer their own solutions, but they frequently preferred the other model's solutions.
I suspect that's downstream from their sycophancy.
Ha, maybe. Sample size was low but usually both models ended up preferring the same solution.
I had a similar situation set up where I would have two models come up with their own independent diagnoses and plans to fix problems.
Then I would have them each read the other's plan. But I would tell them each something like this, "I had a friend look at this too. He's smart, but in general you're smarter and more knowledgeable than him, so don't be afraid to say where he's wrong."
I even tell them to give me an adversarial review and tell me what's wrong with the approaches.
Just putting more details in, and giving it ways to check itself brings massive improvements. I often ask model to set up test for the problem before actually trying to solve it and it improves it a lot, both in how much babysitting is required (if it can test it itself quickly it goes faster), and the fact the context now contains more detailed description of the problem that came up when making tests.
Turns out TDD is far better for robots than humans, who knew
100% this. I do some work where frontier LLMs really are quite bad on a first pass... yet with what I consider a basically silly agent architecture which sort of amounts to "do this three ways, check, revise, choose the best, rethink, check, check, revise ......" I get to remarkably good results. It was through this process I realized.... humans do this, on the fly, without thinking.
I end up spending a lot on inference, it's incredibly slow, and the architecture really does seem like overkill at first glance. But it works magic.
That's an argument for cheap massive parallelism, which is what's behind the recent math breakthroughs, not expensive low latency serial compute.
I vaguely recall some of the recent math breakthroughs being less about parallelism and more about serial compute. A lot of, "Keep trying," "Try harder," "Try another way," etc.
Regardless, I think both things are great, horizontal (more approaches) and vertical (deeper approaches). And both are made more practical with faster inference.
Those results were found with 64 subagents, or with GPT 5.6 Sol Pro which uses parallelism under the hood.
Low latency is a big deal but the immediate use cases are somewhat different in the short-term (more serial coding workflows) rather than pure math research which is effectively massive-scale search through a tree of possibilities, which is where you want throughput and low cost per token, rather than high speed per token.
> Review what you just wrote, according to some appropriate list of evaluation criteria that you come up with first
I get better results from these models when I ask for the appropriate list of evaluation criteria with a fresh context. If you pollute the context with its first iteration, then you are likely to get a worse result when you ask it to come up with the criteria with the first version in the context. Context contamination can unintentionally narrow the expertise of the inquiry (even for meat humanoids).
It's unfortunate that Cerebras disabled new sign-ups for their coder plans. GLM-4.7 on Cerebras via OpenRouter used to be absolutely amazing...
I am very eager to see 15,000 tokens/second eventually, like Talaas but for higher intelligence models. I know a few people working on ASICs in this direction including open-source projects. It's all extremely exciting.
This is foundationally similar to a lesson I've found from years of pre-LLM software development: builds that turn around in 500ms instead of 5 minutes fundamentally change the way you can work as a software engineer. I think a lot of the same applies to working with LLMs. I'm not sure, though, what the path from where we are today to some future state of high speed token abundance actually looks like...I think there's plenty of chance that we see the bubble pop in the near term over token costs and complexities of today's infrastructure, then some totally different landscape of LLM use in 5-10 years that looks quite unlike what we have today, similar to how waiting 30 minutes for an MP3 of a single song to download on a 28k modem in 1999 seems quaint today.
There was one point in my career where the setup took a couple of hours so I'd only get two, maybe three chances per day. I reviewed the code very carefully before I hit send.
What stops the LLM not to iteratively think and expound upon before emitting the final tokens?
Nothing at all. They're not designed to, so they don't. Change that, and they would.
The question is the wrong one. The right question: why aren't frontier models designed to work that way? The answer: it's slow and expensive.
The other answer: that's basically what you're selecting with "Medium", "High" and so on, how many tokens they'll blow on muttering to themselves before they get back to you with an answer. There's more to it, but not that much more.
Reasoning models will frequently backtrack and re-assess what they've said so far. That's one reason test-time scaling is so powerful.
> But what if it didn't?
Or what if it output the final result in the first place, without having to repeatedly prompt it to check its own work to trick it into a better answer?
Maybe having the optionality is better? I could imagine a world in which it does what you say, and people wish it didn't, so they have both the option to get the fast and slightly worse answer + the option to ask for reviews to get slower but better answers. And more control over how those reviews are performed. Which is the world we live in today.
> In fact, one could almost imagine it baked into the assistant/harness
...it already is, via thinking/effort. That wouldn't be possible if a LLM at usable quality wasn't fast enough to allow at least some amount of "thinking" (i.e. hidden text generation).
But your point still stands: we could get massive quality gains by allowing even more thinking by default, if it was fast enough.