AI training costs are increasing around 3x annually across each of the last 8 years to achieve its performance improvements. Last year, spending across all labs was $150bn. Keeping the 3x trend means that, to keep pace with current advances, costs should rise to $450bn in 2025, $900bn in 2026, $2.7tn in 2027, $8.1tn in 2028, $25tn in 2028, and $75tn in 2029 and $225tn in 2030. For reference, the GDP of the world is around $125tn.
I think the labs will be crushed by the exponent on their costs faster white-collar work will be crushed by the 5% improvement exponent.
Be careful you're not confusing the costs of training an LLM and the spending from each firm. Much of that spending is on expanding access to older LLMs, building new infrastructure, and other costs.
That’s a fair criticism of my method, however model training costs are a significant cost centre for the labs. Modelling from there instead of from total expenditure only adds 2-3 years before model training costs are larger than the entire global economy.
"Model collapse" is a popular idea among the people who know nothing about AI, but it doesn't seem to be happening in real world. Dataset quality estimation shows no data quality drop over time, despite the estimates of "AI contamination" trickling up over time. Some data quality estimates show weak inverse effects (dataset quality is rising over time a little?), which is a mindfuck.
The performance of frontier AI systems also keeps improving, which is entirely expected. So does price-performance. One of the most "automation-relevant" performance metrics is "ability to complete long tasks", and that shows vaguely exponential growth.
Given the number of academic papers about it, model collapse is a popular idea among the people who know a lot about AI as well.
Model collapse is something demonstrated when models are recursively trained largely or entirely on their own output. Given most training data is still generated or edited by humans or synthetic, I'm not entirely certain why one would expect to see evidence of model collapse happening right now, but to dismiss it as something that can't happen in the real world seems a bit premature.
We've found in what conditions does model collapse happen slower or fails to happen altogether. Basically all of them are met in real world datasets. I do not expect that to change.
In 2025 you can add quality to jpegs. Your phone does it and you don't even notice. So the rhetorical metaphor employed holds up, in that AI is rapidly changing the fundamentals of how technology functions beyond our capacity to anticipate or keep up with it.
This is an especially bad example, a nice shiny grille is going to be strongly reflecting stuff that isn't already part of the image (and likely isn't covered well by adjacent pixels due the angle doubling of reflection).
Sure, you can view an LLM as a lossy compression of its dataset. But people who make the comparison are either trying to imply a fundamental deficiency, a performance ceiling, or trying to link it to information theory. And frankly, I don't see a lot of those "hardcore information theory in application to modern ML" discussions around.
The "fundamental deficiency/performance ceiling" argument I don't buy at all.
We already know that LLMs use high level abstractions to process data - very much unlike traditional compression algorithms. And we already know how to use tricks like RL to teach a model tricks that its dataset doesn't - which is where an awful lot of recent performance improvements is coming from.
And if you get that "sometimes" down to "rarely" and then "very rarely" you can replace a lot of expensive and inflexible humans with cheap and infinitely flexible computers.
That's pretty much what we're experiencing currently. Two years ago code generation by LLMs was usually horrible. Now it's generally pretty good.
I think humans who think they can't be replaced by a next token predictor think too highly of themselves.
LLMs show it plain and clear: there's no magic in human intelligence. Abstract thinking is nothing but fancy computation. It can be implemented in math and executed on a GPU.
"What's actually happening" is all your life you've been told that human intelligence is magical and special and unique. And now it turns out that it isn't. Cue the coping.
We've already found that LLMs implement the very same type of abstract thinking as humans do. Even with mechanistic interpretability being in the gutters, you can probe LLMs and find some of the concepts they think in.
But, of course, denying that is much less uncomfortable than the alternative. Another one falls victim to AI effect.
> "What's actually happening" is all your life you've been told that human intelligence is magical and special and unique. And now it turns out that it isn't. Cue the coping.
People have been arguing this is not the case for at least hundreds of years.
I as a human being can of course not be replaced by a next token predictor.
But I as a chess player can easily be replaced by a chess engine and I as a programmer might soon be replaceable by a next token predictor.
The only reason programmers think they can't be replaced by a next token predictor is that programmers don't work that way. But chess players don't work like a chess engine either.
Hallucination has significantly decreased in the last two years.
I'm not saying that LLMs will positively replace all programmers next year, I'm saying that there is a lot of uncertainty and that I don't want that uncertainty in my career.
AI training costs are increasing around 3x annually across each of the last 8 years to achieve its performance improvements. Last year, spending across all labs was $150bn. Keeping the 3x trend means that, to keep pace with current advances, costs should rise to $450bn in 2025, $900bn in 2026, $2.7tn in 2027, $8.1tn in 2028, $25tn in 2028, and $75tn in 2029 and $225tn in 2030. For reference, the GDP of the world is around $125tn.
I think the labs will be crushed by the exponent on their costs faster white-collar work will be crushed by the 5% improvement exponent.
Be careful you're not confusing the costs of training an LLM and the spending from each firm. Much of that spending is on expanding access to older LLMs, building new infrastructure, and other costs.
That’s a fair criticism of my method, however model training costs are a significant cost centre for the labs. Modelling from there instead of from total expenditure only adds 2-3 years before model training costs are larger than the entire global economy.
Your math is a bit less than it should be because you doubled instead of trebled 2026
The current trained models are already pretty good enough for many things.
Is that so? Ok let the consumers decide - increase the price and let's see how many users are willing to pay the price.
They are mediocre plagiarism machines at best.
Are LLMs stackable? If they keep misunderstanding each other, it'll look more like successive applications of JPEG compression.
By all accounts, yes.
"Model collapse" is a popular idea among the people who know nothing about AI, but it doesn't seem to be happening in real world. Dataset quality estimation shows no data quality drop over time, despite the estimates of "AI contamination" trickling up over time. Some data quality estimates show weak inverse effects (dataset quality is rising over time a little?), which is a mindfuck.
The performance of frontier AI systems also keeps improving, which is entirely expected. So does price-performance. One of the most "automation-relevant" performance metrics is "ability to complete long tasks", and that shows vaguely exponential growth.
Given the number of academic papers about it, model collapse is a popular idea among the people who know a lot about AI as well.
Model collapse is something demonstrated when models are recursively trained largely or entirely on their own output. Given most training data is still generated or edited by humans or synthetic, I'm not entirely certain why one would expect to see evidence of model collapse happening right now, but to dismiss it as something that can't happen in the real world seems a bit premature.
We've found in what conditions does model collapse happen slower or fails to happen altogether. Basically all of them are met in real world datasets. I do not expect that to change.
The jpeg compression argument is still valid.
It's lossy compression at the core.
In 2025 you can add quality to jpegs. Your phone does it and you don't even notice. So the rhetorical metaphor employed holds up, in that AI is rapidly changing the fundamentals of how technology functions beyond our capacity to anticipate or keep up with it.
> add quality to jpegs
Define "quality", you can make an image subjectively more visually pleasing but you can't recover data that wasn't there in the first place
You can if you know what to fill from other sources.
Like, the grill of a car. If we know the make and year, we can add detail with each zoom by filling in from external sources
This is an especially bad example, a nice shiny grille is going to be strongly reflecting stuff that isn't already part of the image (and likely isn't covered well by adjacent pixels due the angle doubling of reflection).
Is this like how crypto changed finance and currency
I don't think it is.
Sure, you can view an LLM as a lossy compression of its dataset. But people who make the comparison are either trying to imply a fundamental deficiency, a performance ceiling, or trying to link it to information theory. And frankly, I don't see a lot of those "hardcore information theory in application to modern ML" discussions around.
The "fundamental deficiency/performance ceiling" argument I don't buy at all.
We already know that LLMs use high level abstractions to process data - very much unlike traditional compression algorithms. And we already know how to use tricks like RL to teach a model tricks that its dataset doesn't - which is where an awful lot of recent performance improvements is coming from.
Sure, you can upscale a badly compressed jpeg using ai into something better looking.
Often the results will be great.
Sometimes the hallucinated details will not match the expectations.
I think this applies fundamentally to all of the LLM applications.
And if you get that "sometimes" down to "rarely" and then "very rarely" you can replace a lot of expensive and inflexible humans with cheap and infinitely flexible computers.
That's pretty much what we're experiencing currently. Two years ago code generation by LLMs was usually horrible. Now it's generally pretty good.
I think you are selling yourself short if you believe you can be replaced by a next token predictor :)
I think humans who think they can't be replaced by a next token predictor think too highly of themselves.
LLMs show it plain and clear: there's no magic in human intelligence. Abstract thinking is nothing but fancy computation. It can be implemented in math and executed on a GPU.
LLMs have no ability to reason whatsoever.
They do have the ability to fool people and exacerbate or cause mental problems.
LLMs are actually pretty good at reasoning. They don't need to be perfect, humans aren't either.
what's actually happening is all your life you've been told by experience if something can talk to you is that it must be somewhat intelligent.
Now you get can't around that this might not be the case.
You're like that beetle going extinct mating with beer bottles.
https://www.npr.org/sections/krulwich/2013/06/19/193493225/t...
"What's actually happening" is all your life you've been told that human intelligence is magical and special and unique. And now it turns out that it isn't. Cue the coping.
We've already found that LLMs implement the very same type of abstract thinking as humans do. Even with mechanistic interpretability being in the gutters, you can probe LLMs and find some of the concepts they think in.
But, of course, denying that is much less uncomfortable than the alternative. Another one falls victim to AI effect.
> "What's actually happening" is all your life you've been told that human intelligence is magical and special and unique. And now it turns out that it isn't. Cue the coping.
People have been arguing this is not the case for at least hundreds of years.
Considering we don't understand consciousness at ALL or how humans think, you might want to backtrack your claims a bit.
Any abstraction you're noticing in an LLM is likely just a plagiarized one
Why isn't it then
I as a human being can of course not be replaced by a next token predictor.
But I as a chess player can easily be replaced by a chess engine and I as a programmer might soon be replaceable by a next token predictor.
The only reason programmers think they can't be replaced by a next token predictor is that programmers don't work that way. But chess players don't work like a chess engine either.
this boring reductionist take on how LLMs work is so outdated that I'm getting second hand embarassment.
Sorry, I meant a very fancy next token predictor :)
Lots of technology is cool if you get to just say “if we get rid of the limitations” while offering no practical way to do so.
It’s still horrible btw.
Hallucination has significantly decreased in the last two years.
I'm not saying that LLMs will positively replace all programmers next year, I'm saying that there is a lot of uncertainty and that I don't want that uncertainty in my career.
Pretty crazy, and all you have to do is assume exponential performance growth for as long as it takes.