Can you give concrete examples of "something impossible happens based on known physics"? I have followed the AI debate for a long time but I can't think of what those might be.
Can you give concrete examples of "something impossible happens based on known physics"? I have followed the AI debate for a long time but I can't think of what those might be.
Optimal learning is an interesting problem in computer science because it is fundamentally bound by geometric space complexity rather than computational complexity. You can bend the curve but the approximations degrade rapidly and still have a prohibitively expensive exponential space complexity. We have literature for this; a lot of the algorithmic information theory work in AI was about characterizing these limits.
The annoying property of prohibitively exponential (ignoring geometric) space complexity is that it places a severe bound on computational complexity per unit time. The exponentially increasing space implies an increase in latency for each sequentially dependent operation, bounded at the limit by the speed of light. Even if you can afford the insane space requirements, your computation can’t afford the aggregate latency for anything useful even for the most trivial problems. With highly parallel architectures this can be turned into a latency-hiding problem to some extent but this also has limits.
This was thoroughly studied by the US defense community decades ago.
The tl;dr is that efficient learning scales extremely poorly, more poorly than I think people intuit. All of the super-intelligence hard-takeoff scenarios? Not going to happen, you can’t make the physics work without positing magic that circumvents the reality of latencies when your state space is unfathomably large even with unimaginably efficient computers.
I harbor a suspicion that the cost of this scaling problem, and the limitations of wetware, has bounded intelligence in biological systems. We can probably do better in silicon than wetware in some important ways but there is not enough intrinsic parallelism in the computation to adequately hide the latency.
Personally, I find these “fundamental limits of computation” things to be extremely fascinating.
So I studied Machine Learning too. One of the main things I learned is that for any problem there is an ideally sized model that when trained will produce the lowest error rate. Now, when you do multi-class learning (training a model for multiple problems), that ideally sized model is larger but there is still an optimum sized model. Seems to me that for GAI, there will also be an ideally sized model. I wouldn't be surprised if the complexity of that model was very similar to the size of the human brain. If that is the case, then some sort of super-intelligence isn't possible in any meaningful way. This would seem to track with what we are seeing in the today's LLMs. When they build bigger models, they often don't perform as well as the previous one which perhaps was at some maximum/ideal complexity. I suspect, we will continue to run into this barrier over and over again.
> for any problem there is an ideally sized model that when trained will produce the lowest error rate.
You studied ML before discovery of "double descent"?
https://youtu.be/z64a7USuGX0
I did, however I have also observed ideal sized models since then in algorithms designed with knowledge of it.
Any reference material (papers/textbooks) on that topic? It does sound fun.
Not the person you are responding to, but much of the conclusions drawn by Bostrom (and most of EY’s ideas are credited to Bostrom) depend on infinities. The orthogonality thesis being series from AIXI, for example.
EY’s assertions regarding a fast “FOOM” have been empirically discredited by the very fact that ChatGPT was created in 2022, it is now 2025, and we still exist. But goal posts are moved. Even ignoring that error, the logic is based on, essentially, “AI is a magic box that can solve any problem by thought alone.” If you can define a problem, the AI can solve it. This is part of the analysis done by AI x-risk people of the MIRI tradition. Which ignores entirely that there are very many problems (including AI recursive improvement itself) which are computationally infeasible to solve in this way, no matter how “smart” you are.
As far as I understand ChatGPT is not capable of self-improvement, so EY's argument is not applicable to it. (At least based on this https://intelligence.org/files/IEM.pdf from 2013.)
The FOOM argument starts with some kind of goal-directed agent (that escapes and then it) starts building a more capable version of itself (and then goal drift might set in might not)
If you tell ChatGPT to build ChatGPT++ and leave currently there's no time horizon within it would accomplish either that or escape, or anything, because now it gives you tokens rendered on some website.
The argument is not that AI is a magic box.
- The argument is that if there's a process that improves AI. [1]
- And if during that process AI becomes so capable that it can materially contribute to the process, and eventually continue (un)supervised. [2]
- Then eventually it'll escape and do whatever it wants, and then eventually the smallest misalignment means we become expendable resources.
I think the argument might be valid logically, but the constant factors are very important to the actual meaning and obviously we don't know them. (But the upper and lower estimates are far. Hence the whole debate.)
[1] Look around, we have a process that's like that. However gamed and flawed we have METR scores and ARC-AGI benchmarks, and thousands of really determined and skillful people working on it, hundreds of billions of capital deployed to keep this process going.
[2] We are not there yet, but decades after peak oil arguments we are very good at drawing various hockey stick curves.
(1) You'd be surprised just how much of Claude, ChatGPT, etc. is essentially vibe coded. They're dog-fooding agentic coding in the big labs and have been for some time.
(2) It is quite trivial to Ralph Wiggam improvements to agentic tools. Fetch the source code to Claude Code (it's minimized, but that never stopped Claude) or Codex into a directory, then run it in a loop with the prompt "You are an AI tool running from the code in the current directory. Every time you finish, you are relaunched, acquiring any code updates that you wrote in the last session. Do whatever changes are necessary for you to grow smarter and more capable."
Will that work? Hell no, of course it won't. But here's the thing: Yudkowsky et al predicted that it would. Their whole doomer if-you-build-it-everybody-dies argument is predicated on this: that take-off speeds would be lightning fast, as a consequence of exponentials with a radically compressed doubling time. It's why EY had a total public meltdown in 2022 after visiting some of the AI labs half a year before the release of ChatGPT. He didn't even think we would survive past the end of the year.
Neither EY nor Bostrom, nor anyone in their circle are engineers. They don't build things. They don't understand the immense difficulty of getting something to work right the first time, nor how incredibly difficult it is to keep entropy at bay in dynamical systems. When they set out to model intelligence explosions, they assumed smooth exponentials and no noise floor. They argued that the very first agent capable of editing its own source code as good as the worst AI researchers, would quickly bootstrap itself into superintelligence. The debate was whether it would take hours or days. This is all in the LessWrong archives. You can go find the old debates, if you're interested.
To my knowledge, they have never updated their beliefs or arguments since 2022. We are now 3 years past the bar they set for the end of the world, and things seem to be going ok. I mean, there's lots of problems with job layoffs, AI used to manipulate elections, and slop everywhere you look. But Skynet didn't engineer a bioweapon or gray goo to wipe out humanity - which is literally what they argued would be happening two years ago!