I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money
I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
I'll take the opposite here. If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
I think it's worth acknowledging that the power of LLMs at this point is not really so much in the smarts, but in the coordination and the surrounding harness tech. "Written english" turning into sequences of commands[0]. The whole agentic "stuff" in general. Tools + coordination is the superpower. The reasoning... it doesn't have to be _that_ good for the rest of the stuff to work. On good codebases and infra, at least.
And I say this as someone who really would rather most of this stuff disappear!
[0]: programming is obviously text to commands, but there's a loooooooot of futziness that LLM reasoning has let us remove in some flows
> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
I think it’s more that pushing frontier is extremely costly and there is no free lunches in same way as 2024.
Maybe, though that's another kind of progress in itself. Very impressive progress!
Anything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.
I have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments.
I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.
I really liked codex in the last few weeks, especially its ability to clean up after Claude's (prose) messes and do reviews.
But in the last few days something seems to have happened that made Codex's models massively stupider (for what I am doing).
Really weirdly, it suddenly refused to even run tests it previously wrote itself (and previously ran), because of some false positive about cybersecurity.
That by itself is not evidence of stupidity. Trying to make a 200+ file PR full of research notes is, and the PR didn't even solve the problem I asked it to.
astra is more parlor tricks than real gains tbh
i swear they trained in on threejs in particular so those idiots on twitter could spam their garbage demos
I'm sure that's part of it, but I run it side by side in my review bot, and Astra medium effort consistently catches more issues than Sol 5.6, using fewer tokens.
For coding it's a little harder to tell, but at least the prose feels a little better.
They've not plataued but they're certainly not as impressive as the hype would have them to be.
The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
You don't need human input. Any coherent input will do the trick.
You can, for example, hook it up to a logging system and have it fix errors as they occur on your platform.
Have you tried this? How did it go?
I’d be curious about:
- your setup. How it all works - The types of errors it fixed and how quickly - Any regressions or issues it caused - The cost
Thanks!
Impressed with the model performance or the chatbot/agent performance?
Really? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value.
For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine.
Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious
No idea what you are missing and yes, Opus is quite solid, but Fable is clearly way better for me.
I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value for me.
If Fable doesn't add additional value in your workplace, it means you aren't being ambitious enough in how you integrate agents into your workstream.
Yes, it's probably comparable to 4.8 if you are just using it to write code and put up a couple pull requests. That's not where things are now.
You shouldn't be down voted, AI native companies have already moved up to the next level beyond writing individual PRs.
And where are things now?
By ambitious if you mean we are not like all the linkedin influencers with their “i one shotted an app this morning…” then no, we are not. Nobody is. I have been in software engineering for 18 years and 6 different companies including FANG and 99% of the people, on 99% of the days arnt writing new apps from scratch. Thats simply not how anything works.
And what even are these ambitious companies and people one shotting and building with Fable? AI has been around for almost 3 years now. Tell me one app or software you use which has gotten significantly better and has amazing new useful features landing on a weekly basis? If anything, every single software product I use has gotten worse.
This is just "you're holding it wrong" with a little smooch of condescension. If only we plebeians could comprehend what magnificent works those who have ambitiously integrated agents into the workstream have wrought!
Sometimes you actually are holding it wrong. It's pretty reasonable to think that Fable isn't worth the massive increase in cost, but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
> Sometimes you actually are holding it wrong.
I couldn't imagine being so presumptuous as to know that my workflow fits all sizes, and all others are just holding it wrong – or worse, they're not doing real work. It would take a bigger ego on my part, or maybe less social awareness, to presume this.
> but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I don't even use claude, I give exactly zero shits about fable or opus or bingus bongus.
I mean it's very easy to comprehend and you don't need me for it.
Just download claude code or codex and ask it to give suggestions about where to integrate agents into your workstream.
To be perfectly clear: my comment has nothing to do with your workflow, but rather with the way you've condescendingly implied the person's work is trifling and inferior because they don't use tools the same way you do.
The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
> Quality data can be created synthetically at an exponential rate as models improve
No it can't? Every time the labs try this we see model collapse, e.g. shoving goblins into every conversation.
And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
> Every time the labs try this we see model collapse
The latest studies demonstrate model collapse is not a given and synthetic data can be used just fine. The latest models are proof of that, they're all trained on large swathes of synthetic data. It can't be used as the -only- data source of course, but that's not how it is being used. This is an obvious conclusion, too, because there's no difference between synthetic data and the data people can create, the difference is whether that data is revealing new information about the thing the model is trying to learn. If the synthetic data is just teaching the model the same thing over and over again it results in overfitting, so it needs to be done intelligently.
For example, if I have an example of a puzzle, I can generalize that example and create thousands of synthetic data examples, with different rotations/perspectives, rather than having to find the data naturally. It's not that the models are just generating data out of thin air, they're generating the synthetic data on top of real world data. The smarter the models get, the better they are at generating quality synthetic variations and finding valid synthetic variations.
> And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
It is accelerating how quickly researchers and engineers can do their jobs.
https://news.mit.edu/2026/ai-helps-design-new-materials-that...
This is only the beginning, too... Look ahead a year or two.
> The latest studies demonstrate model collapse is not a given
Which studies? [edit: I'll assume you mean these two given by @dorolow: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515]
> It can't be used as the -only- data source of course, but that's not how it is being used
Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
> because there's no difference between synthetic data and the data people can create
I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
> It is accelerating how quickly researchers and engineers can do their jobs. > https://news.mit.edu/2026/ai-helps-design-new-materials-that...
That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.
> This is only the beginning, too... Look ahead a year or two.
Lol that excuse is getting really old
> Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
It doesn't need to. We're not even close to exhausting the useful synthetic data within the human data we have, let alone all of the new data that is being created.
> I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
It's not. It's just bytes of information. A machine and a human can write the same bytes (and often do). Like I already said, model collapse happens when you are overfitting on data without useful, fresh training signals. That's the key difference between the data. The data itself isn't in some way "special", some unique configuration of bytes that imbues special powers, it's that the useful information in it has already been exhausted by the model. You can get the same phenomena by having a poor distribution of human training samples as well. I think you're confusing LLM generated data with synthetic data. Synthetic data doesn't need to be created by an LLM, although an LLM can assist in the creation.
Wiki:
> In early model collapse, the model begins losing information about the tails of the distribution – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data.[11] In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its variance.[10][12][13]
As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear.[25] Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse")
> That's pretty clearly a hype article
It was just the first article I saw on a quick google search, there are thousands of these stories. It's easy to dismiss anything that doesn't align with your worldview as hype, but you're the one lacking evidence now.
> I'm asking for empirical measurements of timelines, not hypotheticals.
Go and find it then? You haven't bothered looking.
> Lol that excuse is getting really old
You're doing the same thing people have been doing for years, comparing this very second in time and failing to extrapolate. HackerNews was full of developers who said that AI would never be useful for programming, it can't do x, y, z. Now these same people don't write code by hand anymore and haven't looked at their codebases in months.
You had people in mathematics saying the same thing, now you have Terrence Tao posting articles about how AI is stealing their job.
You had artists, designers and photographers saying the same thing, now they can't tell the difference between something human created or AI created.
[dead]
We use large amounts of synthetic data for training at work and have not observed any sort of model collapse when done properly.
Edit: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515
There's a lot of deluland posts about.
You're not well informed. Helps to keep an open mind if you want to keep up to date.
Can you provide sources for these claims?
What claim do you have a problem with?
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
https://openai.com/index/jalapeno-first-results/
> We used AI to design the chip, and designed the chip so AI could program it AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
https://www.anl.gov/article/scientists-deploy-ai-agents-to-a...
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
It's not even synthetic data as such - often it is environments. So the models create their own data solving tasks in generated environments. I am making one such environment for computer use agents, 600 tasks, each of them a mini app.