The parallel is not to nuclear energy -- which has killed hilariously few people all things considered -- but to nuclear weapons and the nuclear arms race, which are an actual existential risk and arguably it was just dumb luck that a nuclear war never happened.
The parallel maybe if it was actually AGI and not just LLMs. We're very far from that but these safety discussions pretends it's all the same.
There's a massive difference between "LLMs doing cybersecurity testing should have extensive sandboxing and monitoring" and "AI is broadly threatening humanity like nuclear weapons". Just because it's useful to have people think about the possibilities doesn't mean it's an emergency requiring grand intervention.
In the "our biggest danger is a daily danger" camp, I prefer analogies like:
* Asbestos: AI models making business-decisions which cause a ton of unrecognized damage, which cannot be healed, and replacing them with reliable processes involves expensive re-architecting.
* Radium Toothpaste: AI assistants that end up dealing psychological damage, especially if they are pitched as your friend or confidante.
* Chemical spills and plant accidents: Oops, our server-banks were manufacturing our latest desperate attempt to Create Digital God and in the process we accidentally destroyed/crippled/compromises some people's sites and databases because we weren't really paying attention to what was going on.
I see parallels between this race and Mutually Assured Destruction (MAD).
I think we are in a MAD crisis, as the motivation to continue unsatisfactory control over development of AI is from unlimited capital inflow.
This is because we are looking at the trajectory not just their current capabilities. LLMs went from barely being unable to focus on a task to purposefully escaping sandboxes within like 2, maybe 3 years?
Any strong claim requires evidence and hand wavvy stuff about a technology just infinitely getting better isn't good enough on its own.
Human history is full of examples of this stuff plateauing and hitting hard scaling walls. Not only technology wise but economics and useful applications.
So much of this tension with AI is because everyone's concept of AI comes from movies and books, and nerds drinking Kool aid (see also: web 2.0).