Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning.
We are probably going to need a lot more GPUs.
GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.
Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.
The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL.
While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.
I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.
It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).
I mean, theoretically you can solve every finitary problem with a brute force solution...
Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.
Moreover we've known for quite a while now that glial cells also participate in cognition and moderate learning (e.g.: [1]). When you take those connections into account the numbers get really staggering. 85 billion glial cells with trillions of protein channels facilitating communication between the glial syncytium [2].
The real question is not how many "weights" the human brain has (neurons+synapses may or may not translate into "weights", and brain might be also inefficient for what it is), but rather how much evolutionary and social "compute" was necessary to pack everything into that capacity.
For intelligence, I expect the next breakthrough to be colocation of memory and compute in the same chip. And we'll need much more of this memory, probably a few petabytes.
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.
I'm surprised it is that low. Are not all top AI labs "cheating" and workaround LLMs's low sample efficiency by hiring people to generate more data points - similar problems with answers, so they can train models on those and improve scores? A good benchmark for general intelligence probably should be a complete black box, no sample data given/leaked at all.
Very interesting. My prediction is that Mythos would outperform Sol.
Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.
At some point his claim should be fully falsified no?
Are you joking? They spend billions of dollars training LLMs to get a 7.8% on arc agi 3 whereas DINO models are near sota in image classification, provide meaningful embeddings to the point where image segmentation is just PCA. The spend on DINO cannot be more than five million (correct me if I'm wrong)
His main anti-LLM predictions have been consistently either wrong or misleading.
There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.
His main LLM predictions have almost nothing to do with Arc AGI...
What exactly was he dead wrong about that is proven by any of this?
GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.
He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.
He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.
Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
That does not at all look cherry picked or taken out of context...
LeCun's ideas cannot be reduced to a 6 second clip...
You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...
You can watch the whole Lex Friedman interview, it's on youtube. It's not out of context at all. He goes on about how LLMs will never be able to do things that they do trivially. And he has just doubled down for years.
Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.
I think you misunderstand what he thinks LLMs can and can't do. He says repeatedly that LLMs may be great for generating text and code, but that a fundamental model of artificial intelligence should be able to perceive and use information outside of text. Also, that a supervised learning approach is not exactly representative for how we learn. That it's a small piece but we largely learn with unsupervised learning. His main criticisms of LLMs are that they are supervised, probabilistic, and learn largely from text instead of observations. His claims about performance come downstream and I'd still argue that he's been (somewhat) right about those as well.
That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.
While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.
To me LLMs have gotten better since 2024, but their fundamental flaws still seem there.
They hallucinate when it comes to really challenging tasks such as math proofs. They still do not reuse code well and will rewrite functions instead of perusing the standard library.
But this is good news. LLMs are awesome and they are only the first step towards AI being applied everywhere. They are a Model T
> What exactly was he dead wrong about that is proven by any of this?
He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.
That's true, but he's still correct, it's just that the context is now so large that only people using agent loops see "context rot"
His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).
With JEPA models so far, it seems that PCA on latent vectors suffices.
tldr: embeddings have a lot more room for improvement
From CoreWeave, at current prices (~$2.46/hr spot to ~$6.16/hr on demand) would correspond to $22M–$55M.
The dataset is really where the cost is though - they used LVD-1689M - 1.6B images of curated web data from roughly 17B instagram images. This probably cost a huge amount of hours in human annotation, compute for algorithmic filtering, etc and not to mention probably a 20-50 person team working on this model.
You might want to change assumptions about how expensive these models are.
Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.
These aren’t raw base models they are the result of a ton of RLHF and various adjustments.
Bitter lesson wildly overstated in this context.
More RLHF is in fact scaling.
Yes, but not in the “dump another chunk of all written language in the bucket and stir”-sense which is what bitter lesson became synonymous with.
That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.
GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.
Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.
rlhf = reinforcement learning from human feedback
(had to look it up)
The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL.
While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.
I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.
It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
There goes my plan to buy a PC for the next decade
The whole of knowledge work is being automated. We've barely begun to see the GPU build out. This is just the start.
I'd imagine they're going to 10x this, maybe 100x this.
Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).
I mean, theoretically you can solve every finitary problem with a brute force solution...
Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
Or a lot better efficiency.
> Dramatic difference
Isn't this just the difference between getting 0 right and getting 1 right?
And a lot more electricity to power them.
And dozens of data centers in every state so tokens are dirt cheap.
This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.
I thought models werent allowed tools on arc-agi?
> We are probably going to need a lot more GPUs.
Or a breakthrough in algorithms etc.
The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.
The human brain has 80 billion neurons and a 100 trillion synapses. I think you're underselling the processing power of that warm chunk of meat.
The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.
Moreover we've known for quite a while now that glial cells also participate in cognition and moderate learning (e.g.: [1]). When you take those connections into account the numbers get really staggering. 85 billion glial cells with trillions of protein channels facilitating communication between the glial syncytium [2].
[1] https://www.sciencedirect.com/science/article/pii/S193459091... [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/
The real question is not how many "weights" the human brain has (neurons+synapses may or may not translate into "weights", and brain might be also inefficient for what it is), but rather how much evolutionary and social "compute" was necessary to pack everything into that capacity.
20 watts for inference AND training!
For intelligence, I expect the next breakthrough to be colocation of memory and compute in the same chip. And we'll need much more of this memory, probably a few petabytes.
This is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"?
Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.
I'm surprised it is that low. Are not all top AI labs "cheating" and workaround LLMs's low sample efficiency by hiring people to generate more data points - similar problems with answers, so they can train models on those and improve scores? A good benchmark for general intelligence probably should be a complete black box, no sample data given/leaked at all.
it seems the older models were capped at 10kusd for the runs though?
Very interesting. My prediction is that Mythos would outperform Sol.
Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.
At some point his claim should be fully falsified no?
Mythos probably wouldn't, otherwise they'd have included it in their release. Next version of Mythos probably will though.
And yeah.. Reality has not been kind to LeCun.
Are you joking? They spend billions of dollars training LLMs to get a 7.8% on arc agi 3 whereas DINO models are near sota in image classification, provide meaningful embeddings to the point where image segmentation is just PCA. The spend on DINO cannot be more than five million (correct me if I'm wrong)
JEPA is just getting started
His main anti-LLM predictions have been consistently either wrong or misleading.
There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.
His main LLM predictions have almost nothing to do with Arc AGI...
What exactly was he dead wrong about that is proven by any of this?
GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.
He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.
He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.
Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
>He's merely said they don't think
He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
> He said even 'GPT 5000' couldnt do things that they could do a month later, let alone by 5000.
What things specifically and when?
https://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu
You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
That does not at all look cherry picked or taken out of context...
LeCun's ideas cannot be reduced to a 6 second clip...
You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...
You can watch the whole Lex Friedman interview, it's on youtube. It's not out of context at all. He goes on about how LLMs will never be able to do things that they do trivially. And he has just doubled down for years.
Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.
I think you misunderstand what he thinks LLMs can and can't do. He says repeatedly that LLMs may be great for generating text and code, but that a fundamental model of artificial intelligence should be able to perceive and use information outside of text. Also, that a supervised learning approach is not exactly representative for how we learn. That it's a small piece but we largely learn with unsupervised learning. His main criticisms of LLMs are that they are supervised, probabilistic, and learn largely from text instead of observations. His claims about performance come downstream and I'd still argue that he's been (somewhat) right about those as well.
That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.
While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.
All of us were shocked with RL on LLMs.
To me LLMs have gotten better since 2024, but their fundamental flaws still seem there.
They hallucinate when it comes to really challenging tasks such as math proofs. They still do not reuse code well and will rewrite functions instead of perusing the standard library.
But this is good news. LLMs are awesome and they are only the first step towards AI being applied everywhere. They are a Model T
It's like scaling a swiss cheese and the holes grow bigger with it. You can't get rid of the holes without making a different cheese.
> What exactly was he dead wrong about that is proven by any of this?
He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.
That's true, but he's still correct, it's just that the context is now so large that only people using agent loops see "context rot"
His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).
With JEPA models so far, it seems that PCA on latent vectors suffices.
tldr: embeddings have a lot more room for improvement
DinoV3 paper: https://arxiv.org/pdf/2508.10104#page=36
"we use a rough estimate of a total 9M GPU hours"
From CoreWeave, at current prices (~$2.46/hr spot to ~$6.16/hr on demand) would correspond to $22M–$55M.
The dataset is really where the cost is though - they used LVD-1689M - 1.6B images of curated web data from roughly 17B instagram images. This probably cost a huge amount of hours in human annotation, compute for algorithmic filtering, etc and not to mention probably a 20-50 person team working on this model.
You might want to change assumptions about how expensive these models are.
DINO is a transformer model?
JEPA can use a transformer, and DINO does so yes
ASI is going to be here by the time Lecun gets started.
Falsifying Yann Lecun isn't exactly a priority for anyone seriously working in this space.
Mythos doesn't appear to be on the verified leaderboard for ARC-AGI 3
Notice how neither him, nor Ilya, nor Mira shipped anything relevant recently
It's telling
“Bro” spent most of his career in the wilderness because everybody thought ML/NN/etc were a dead end.
I’d not wager against him having at one one more break though architecture before he retires.
[dead]