The more frontier models progress, the less concerned I am about this. Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power. Ego evolved alongside intelligence in nature because of natural selection, but it doesn't appear to be an inevitable emergent property of intelligence.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
> Of course, there's still the "paperclip optimizer" problem.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
That’s because these aren’t intelligent like you, I, or even a dog. They are mathematical models formulating likely responses with no agency or drive. They appear to be intelligent, but they are the epitome of “fake it till you make it”.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.