I agree with the disconnect between the professionals, but i have yet to see anything that makes me believe that there is a class of 5000 internal people who are that much further than the top users are.
The internal models are more eager, and there is the scale difference of being able to spin up 10.000 vs 10, but to get something valuable out requires skill.
Agents that produce good results 90% of the time vs 93% of the time are not enough of a difference. Multiplied too many times, that error rate is too high for scale to become "magic".
Finding proofs is one of the exceptions.
Just look at the codex/claude harness. They regularly lag behind 'external' users.
I feel like you're taking it somewhat personally as a statement on ability or intelligence levels. It's really just about the scale difference and the access to that infrastructure. That is what enables that class to look at and think about problems differently.
Having 10,000 (or more) agents working for months on a problem is categorically different from the type of usage most people are familiar with. The complexity explodes to a point that requires the "archaeology" he mentions for people to even try to understand what's going on.
This is the key insight. Those capabilities are currently limited to frontier labs and literally cost them millions of dollars to achieve. But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point where we can design systems that are able to go and do things that we don't fully understand, and they will handily outpace our forensic bandwidth.
> But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point
It seems we’re perpetually in this state.
As far as I can tell, we seem to keep moving past states and redefining them while we do.
Chess, ok, Go, ok, doing new math, ok, breaking out of sandboxes and hacking public-facing infrastructure, ok...
Still waiting for the initial results from the first state to "land"...
I hear you, but also, "a broken clock is right twice a day." And really, there are people still alive who have more or less witnessed the entire rise of the machines.
He’s just trying to pitch himself as part of the priest class.
My understanding is he is not talking about "top users" as being further ahead in skill, he is talking about access.
The kind of FOMO he is triggering is the same kind that wanting to be in an exclusive country club gives a certain kind of businessman. One simply can't reproduce that network outside of the exclusive access, no matter how skilled they are.
Lauren Tan from SpaceXAI/Cursor/Grok has been on a few podcasts recently. I was surprised how much info she shared, and she sounded quite honest about what works and what doesn’t, in the context of building grokbot. I wouldn’t be able to do all the verifications she does, with my token budget. But it doesn’t seem like rocket science either (pun not intended).
I agree, and to keep sane I enjoy reading Boris Cherny discuss how he made his own discoveries while designing and building Claude Code cli.
To me he seems half led by measurements (like output quality) and half by his own learning-as-he-goes (like his inventing "plan mode" after an evening of vibe coding and realizing it kept implementing too early in the process).
90% vs 93% is a big deal IMO, especially when you're trying to feed outputs into inputs.
I think the point being made is that when you're feeding outputs into inputs, both 93% and 90% rapidly collapse to 0% without some way to recover.
You might be thinking along the lines of the O-ring theory. Indeed long supply chains accumulate errors exponentially.
However the origins of theory explored the outcome of disastrous and irreparable outcome of space shuttle exploding, lives lost, and billion of dollars destroyed.
When LLM goes off the rails the cost is much lower, and retry is on the table. In fact our core competency shifted from writing code to creating environment where agents can retry cheaply and course correct based on feedback.
A modest increase in the number of turns it takes to reach decorehenve could make a difference between Decoherence and reaching the a grounded checkpoint.
If failure is essentially guaranteed somewhere in that chain, success becomes impossible. You don't need to lose lives to have simply have a fruitless endeavor. And when we're talking about law, we're talking about more than a fruitless endeavor. Law also isn't software that can have tests crafted and fed back into the loop.
Which is precisely what I'm responding about above. 93% actually collapses much slower than 90% since the rate of reduction is based on that of all of the previous rounds, not just the current chance. E.g., this is the difference between 14 rounds before an expected failure instead of 10. An extra 4 rounds deep at the same error rate is a big step forward despite a small change in the base percentages. Add just few more percentage points again and you've more than doubled the depth.
I.e. it's precisely when you look at these chained that the difference is actually very large. It's really only the 1st turn or two of prompts where the difference seems negligable.
How are you running that math and what are you consider to be "expected failure"? If even 30% of your runs go into the garbage, that sounds horrifically unreliable, and both 7 and 10% failure rates hit those margins in less than 5 iterations.
Not to mention that the deeper your iterations, the more money burned on garbage before you realize that data in turned to garbage out turned to garbage in somewhere in the chain.
I'm taking 0.90^n and 0.93^n and seeing 0.93 does more than the initial 3% better over time, which preciesly saves you from things like spending tokens on garbage by this larger factor. You're free to pick whatever target reliabilities from that. E.g. you can maintain above 80% accuracy over 3 turns rather than 2 or 50% over 6 turns instead of 3 or you can pick another percentage point and see how the depth to expected error/expected error at depth numbers are much better than 3% and how another 3% actually doubles accuracy because this is all non-linear.
There's also the matter of "90% compared to what". If it was 90% per attempt compared to a calculator then I'd just use a calculator. If it was 90% per attempt for something I have a 10% chance of getting right per attempt then it's not really so bad. OTOH if it's 90% per attempt at keeping my car in the lane then it's obviously quite bad. I.e. there isn't one "reliable percentage" to target which is why I'm comparing the curve instead of just the most stringent things I can think of in isolation.
Completely agree that error tolerance depends on what it is that we're doing. Some things demand no errors at all, some things have a cost to attempt, some things you can't pre-evaluate for correctness if you don't already have the prerequisite knowledge and you only get one chance. Not everything is software where we can conjure 50 more end-to-end tests to ward off demons. When it's not a harmless throwaway weekend project, 7% is already a huge error rate before you ever think of feeding it back into itself.
First off, the 90% to 93% success rates are not step-by-step failure rates between successive steps of pursuing a solution. They are overall end-to-end success rates on extremely difficult problem sets that most humans would score close to zero on, and most humans with sufficient training and experience would score significantly less than 90%. And they are problem sets that grow successively harder as AIs get successively more capable. A few years ago AIs were being tested on high-school level math problems. Now they are being tested on problem sets that challenge people with PhDs. Or problem sets that humans are (so far) incapable of solving.
Compare that to humans. How often do you think humans get it right the first time? Almost never, for sufficiently complex problems. And how often do humans get the overall result wrong, without realizing it? Also, almost always for sufficiently complex problems, or sufficiently ordinary people. And what percent of the general public would be able to get even one question right in a problem set based on Math Olympiad problems? What percentage of mathematicians are capable of solving even one Millenium Problem?
Furthermore, you have to take into account that AIs do not particularly use output to input architectures. I frequently see self-corrections to plans when I'm developing using AI assistants, and plan changes are sometimes major. ("Oh I see. I need to revise step 4 of the plan." -- Claude paraphrased). Clearly current-generation AIs use more than input to output feedback; they also use supervisor/client strategies of AI and collaborative networks as well. And often provide self-correcting infrastructure, varying from "Claude: Oh. That generated a compiler error. Let's fix that.", to "Claude: One of the test cases is failing. Let's see what's going on.". And that self correcting infrastructure allows AIs to try again and again until they succeed. (Not unlike what humans do when solving difficult problems).
As a member of the 0.2% (but not the 0.00006%), I have to imagine that frontier models, when applied to research topics, are using most of those 10,000 instances to investigate alternate hypotheses in parallel. Which is also something that humans do across entire disciplines. Based on my personal experience with difficult technical and commercial and scientific problems, I sincerely believe that most breakthroughs are not made because whoever made them is more brilliant than the other hundred or thousand or dozen qualified people who are working on the same problem, but are made because they happened to be lucky enough to start with a somewhat random hypothesis that luckily turned out to be the right one. In case you were concerned that this is a sour grapes point of view, yes, our team was the one that came in first.
So even 90% may be a very big deal, for problems that human get right about 0.1% of the time. Or even for problems that AIs get right about 0.1% (but have enough instances available to pursue 10,000 hypotheses in parallel).
p.s. Ironically, it really looks like humans score significantly less than 90% on the "is 90% good" problem.
> The internal models are more eager, and there is the scale difference of being able to spin up 10.000 vs 10, but to get something valuable out requires skill.
Yeah, but the skill is knowing when it's talking nonsense.
Same deal with the recent stuff about vibe coded Photoshop clones. Yeah, they work, but you'd have to do Adobe-level market research to understand what bar the product needs to pass, and the vibe-coders didn't have that, so they saw an image editor and mistook it for "basically fine".
> Agents that produce good results 90% of the time vs 93% of the time are not enough of a difference. Multiplied too many times, that error rate is too high for scale to become "magic".
Sure. However, one of the ways we see things change with, e.g. METR time horizons is that agents fail (IIRC) 10% of the tasks of some (human) duration all the time, pass 10% of that (human) duration all the time, and the rest have a random success rate you can fit to a sigmoid. As the agents get better, they still pass 10% all the time, still fail 10% all the time, but it's a harder set of questions.
2.5 years ago, I was thinking "I know what's a fun idea for April fools day! A programming language written entirely in runes!", the LLM produced non-compiling code for the compiler. 1.5 years ago, I thought "I'll give that another go", it managed it trivially and wasn't even interesting enough to bother with as an April fools day joke, not even with the video of a Viking explaining it to you in Icelandic.
We should probably assume the companies are 1-2 models ahead of the public, at all times. The HuggingFace incident happened about 2 months before the public got models of comparable competence.
> vibe coded Photoshop clones. Yeah, they work [...]
I think what people (sometimes) miss is that building something is one part of the work, but keeping it running, developing and refining it is a whole different story. And I think all those vibe-coded software will not stand the test of time.
Photoshop is 36 years old now and it wouldn't be the most-used, most-established solution for professionals if Adobe didn't do great work over all that time.
Not really rooting for Adobe here, dislike their subscription models just as anybody else, but I think you'll have to recognize that they kept it running and never stopped.
Absolutely.
I'm… well, "enthusiastic" would be the wrong word, but perhaps "impressed"?… with LLMs-as-coders, but they're objectively poor at business decisions, and Adobe keeping their business running all these years is a thing that I don't think current LLMs could have done.
LLMs don't need to focus on the business. They can make the ideal software for you. Who cares if it's not the perfect software for me? I can have my LLM do that. Especially if there's an open source base for me to start.
A separate question. Consider I'm responding here to (amongst other things):
> And I think all those vibe-coded software will not stand the test of time.
When I use LLMs to write software for me, it is rare that I think "this is great, I should share it". Even when I do, I am aware of how fast these things are improving, so while I put great effort into, for example, having Claude make me a nice isochrone map generator, I fully expect that a year from now any similar project (even if different enough the LLM will not simply clone my git repo) will be a one-shot prompt.
What I did will only stand the test of time if all the AI labs shut down and nobody takes over from them. Same with vibe-Photoshop-clones: a year from now, anyone who wants this kind of thing will get it with one prompt, not thousands, and this specific instance will likely be forgotten.
No doubt the unreleased models are better at the things that LLMs are good at (coding, hacking, math) than what's been released, but there is still zero sign of their intelligence becoming broader and more general, and knowing how these models work and are trained there is no reason to expect them to become much more general unless changes are made.
The marketing message being pushed by Anthropic (& also OpenAI) is that these narrow math/hacking/coding skills are representative of the model's intelligence, when that seems to be the exact opposite of the truth - these narrow human/super-human competencies are entirely atypical of the models more general capabilities.
> but there is still zero sign of their intelligence becoming broader and more general
Ahem, getting better at these things is the sign:
- you, two and a half years ago, https://news.ycombinator.com/item?id=39720601Yes, I failed to anticipate environment/playground-based RL-training and how much difference that would make, although I think the broader point about limitations of lack of continual learning still stands, and I can't see these simulated environments ever become complex and/or varied enough to be any substitute for the need to learn on the job.
However, gotcha acknowledged, but my point was about the jaggedness of AI, and how huge the divide in capability is between the domains where RLVR works, primarily coding, hacking and math, and everything else. The success and scale of RLVR in areas such as coding only make this gap more noticeable.
This is what the frontier can do with a computer now.
https://stillwet.art/
Where are the verifiable rewards? It's simply not true that RLVR amenable domains are the only things LLMs are improving massively on. Even Vending Bench is improving.
Intelligence is a jagged beast. That's just the nature of things. We evolved in certain environments and situations and sometimes there's a mismatch and it causes all sorts of wonky things. We just call them funny names like optical illusions and cognitive biases. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters (Watson Test).
The public has yet to produce a result of comparable impact to HuggingFace, to my knowledge. It's not really about the model, it's more about orchestration infrastructure. Seems like we've hit the point where you don't necessarily need a smarter bot, you just need an arbitrary number of them working together.