> Of course, there's still the "paperclip optimizer" problem.

The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?

The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.

Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.