>AI hack into control systems for the electric grid, water supplies, critical infrastructure and dismantle it and set booby traps to prevent it from coming back online smoothly
>AI designed virus/modified bacteria which causes pandemic
>AI worm infects computer systems around the world and activates when it is too deeply integrated into critical systems to dismantle without shutting everything down
>AI creates false alarm event which triggers global conflict
> 2. Infinite duplicatability, thousands of agents can work in tandem
Worse: you have local models they can install on infected computers, so this could easily be "millions" and might just about push "billions" (though phones are a much harder target because power required is still huge even if they technically fit in RAM).
Fukushima was "we didn't make a good enough plan for how to deal with tsunamis in this tsunami-prone area", so that's basically certain with any major infrastructure project that "saved" too much money by vibe-engineering everything and not having real humans give a second pair of eyes to the plans.
A Hiroshima-style incident? LLMs are wildly sycophantic and being used by militaries despite active resistance from, well, everywhere. Did it get (meaningfully) used by Israel or the USA when planning the attack on Iran? It's quite possible that… well, Iran was never a sleeping giant (and the quote is fiction anyway), but ultimately the effect may be the same for Israel as it was for Japan from having attacked Perl Harbour.
A supervirus most is probably the most probable, everything else is fan fiction for now. There is a reason why anthropic no longer fucks around when it comes to biology safeguards, they're the only ones I have not been able to break, not even slightly.
The use of LLMs to make automatic or semi-automatic decisions on firing weapons. This is already happening [1, 2, 3] and has already led to weapons being delpoyed erroneously against civilian targets [1, 3]. It's easy to imagine it happening again on a larger scale, maybe involving nuclear weapons at some point in the future.
If more and more control is handed over to AI driven systems - there won't be much time for reflection with systems reacting instantly or agents prompting humans with "just say the word".
It is hard to imagine - which is precisely how these things become possible because no one will think to put safeguards against such scenarios.
But it seems we as a species cannot control ourselves - the race to dominate is on and it will happen at any cost.
This really isn’t that hard to imagine with even an average level of creativity. Any critical system that people’s lives depend on can be in the crosshairs, whether by accident or not.
People say this, and then the scenarios they come up with are boring and uninteresting. 80% of the scenarios in this thread are "AI hacks something", as if hacking didn't exist before agents, and the remaining 20% are mostly made up of other things humans are already trying to do at scale.
Why am I supposed to believe that a virus created in a virology lab with AI assistance is uniquely more dangerous than a virus created in a virology lab without AI assistance?
> Why am I supposed to believe that a virus created in a virology lab with AI assistance is uniquely more dangerous than a virus created in a virology lab without AI assistance?
You're not.*
The difference is how likely this is to be done, not how dangerous it might be if it was done.
And this isn't likelihood in a 0-100% sense, but in a Poisson distribution sense, i.e. mean time between incidents.
Remember: these AI minds aren't that good with the physical world, and have nevertheless been repeatedly connected to robots, and now some of the AI labs are showing off their actual wet labs. Accidents are absolutely a possibility, and I don't think it's low given the previous behaviour of Silicon Valley startups.
* OK, some people care that AI is getting more capable at genetics just like it's been in maths and programming, but short term, before AI solves genetics so hard it can make an STD that makes the infected uncontrollably horny and then ossifies our bodies or something, it can obviously just copy any of the many DNA or RNA sequences we've already got on record.
Physical labs and experiments will always have physical bottlenecks, so I really don't see how AI is a transformative element here. What is it about AI that will be so transformative to the fundemental economics of this space that it will lead to several orders of magnitude more experimentation? If the problem is an expansion of wild west wet lab experimentation, why isn't ~that~ the focus of safety?
Who is going to profit from making an super STD anyway? How would regulating AI change their incentive structure?
Do you even have to ask? Not giving more detailed answers because I don't want to help skynet take over the world, but what is the name of this forum??? Hello???? There was already news from south korea this week. I'ts not hard to see how a rogue AI could destroy people's lives. We don't need some scifi nuclear launch nonsense to do that.
Problem is, when banks get hacked it's always a bank security issue. Vulnerabilities don't go away when you remove AI from the equation, it just becomes easier to hide.
I agree that sci-fi nuclear launch scenarios are pie-in-the-sky fearmongering, but the current exploits are a reflection of the fast-and-loose security culture that festers in larger orgs.
>AI hack into control systems for the electric grid, water supplies, critical infrastructure and dismantle it and set booby traps to prevent it from coming back online smoothly
>AI designed virus/modified bacteria which causes pandemic
>AI worm infects computer systems around the world and activates when it is too deeply integrated into critical systems to dismantle without shutting everything down
>AI creates false alarm event which triggers global conflict
You mean like humans are already doing at mass scale today ?
Humans are hacking a lot now, but the scope, scale and speed of AI-enabled hacking is orders of magnitude more potentially disruptive.
With AI you have
1. Superhuman reaction time
2. Infinite duplicatability, thousands of agents can work in tandem
3. Superhuman breadth of knowledge of code, exploits, information
Also AI models are improving at a rapid rate
> 2. Infinite duplicatability, thousands of agents can work in tandem
Worse: you have local models they can install on infected computers, so this could easily be "millions" and might just about push "billions" (though phones are a much harder target because power required is still huge even if they technically fit in RAM).
And most importantly:
4. No guilt when violating human standards of ethics.
Are these humans getting more capable by the day, and not getting punished for anything?
>AI hacks into the fundamental code of the universe and turns you into table
Both are rather literally possible.
Fukushima was "we didn't make a good enough plan for how to deal with tsunamis in this tsunami-prone area", so that's basically certain with any major infrastructure project that "saved" too much money by vibe-engineering everything and not having real humans give a second pair of eyes to the plans.
A Hiroshima-style incident? LLMs are wildly sycophantic and being used by militaries despite active resistance from, well, everywhere. Did it get (meaningfully) used by Israel or the USA when planning the attack on Iran? It's quite possible that… well, Iran was never a sleeping giant (and the quote is fiction anyway), but ultimately the effect may be the same for Israel as it was for Japan from having attacked Perl Harbour.
A supervirus most is probably the most probable, everything else is fan fiction for now. There is a reason why anthropic no longer fucks around when it comes to biology safeguards, they're the only ones I have not been able to break, not even slightly.
Anthropic literally just gave its AIs a wetlab to play with. They are arguably at the forefront of risking humanity existence
The use of LLMs to make automatic or semi-automatic decisions on firing weapons. This is already happening [1, 2, 3] and has already led to weapons being delpoyed erroneously against civilian targets [1, 3]. It's easy to imagine it happening again on a larger scale, maybe involving nuclear weapons at some point in the future.
[1] https://gizmodo.com/pentagon-investigators-say-overreliance-...
[2] https://www.the-independent.com/news/world/americas/us-polit...
[3] https://www.nytimes.com/2026/08/24/world/europe/russia-drone...
A Fukushima-style incident would be way way at the bottom of the list of imaginable AI disaster scenarios, severity-wise.
If more and more control is handed over to AI driven systems - there won't be much time for reflection with systems reacting instantly or agents prompting humans with "just say the word".
It is hard to imagine - which is precisely how these things become possible because no one will think to put safeguards against such scenarios.
But it seems we as a species cannot control ourselves - the race to dominate is on and it will happen at any cost.
This really isn’t that hard to imagine with even an average level of creativity. Any critical system that people’s lives depend on can be in the crosshairs, whether by accident or not.
People say this, and then the scenarios they come up with are boring and uninteresting. 80% of the scenarios in this thread are "AI hacks something", as if hacking didn't exist before agents, and the remaining 20% are mostly made up of other things humans are already trying to do at scale.
Why am I supposed to believe that a virus created in a virology lab with AI assistance is uniquely more dangerous than a virus created in a virology lab without AI assistance?
> Why am I supposed to believe that a virus created in a virology lab with AI assistance is uniquely more dangerous than a virus created in a virology lab without AI assistance?
You're not.*
The difference is how likely this is to be done, not how dangerous it might be if it was done.
And this isn't likelihood in a 0-100% sense, but in a Poisson distribution sense, i.e. mean time between incidents.
Remember: these AI minds aren't that good with the physical world, and have nevertheless been repeatedly connected to robots, and now some of the AI labs are showing off their actual wet labs. Accidents are absolutely a possibility, and I don't think it's low given the previous behaviour of Silicon Valley startups.
* OK, some people care that AI is getting more capable at genetics just like it's been in maths and programming, but short term, before AI solves genetics so hard it can make an STD that makes the infected uncontrollably horny and then ossifies our bodies or something, it can obviously just copy any of the many DNA or RNA sequences we've already got on record.
Physical labs and experiments will always have physical bottlenecks, so I really don't see how AI is a transformative element here. What is it about AI that will be so transformative to the fundemental economics of this space that it will lead to several orders of magnitude more experimentation? If the problem is an expansion of wild west wet lab experimentation, why isn't ~that~ the focus of safety?
Who is going to profit from making an super STD anyway? How would regulating AI change their incentive structure?
Agents hack a nuclear power plant and causes a reactor explosion / meltdown?
Do you even have to ask? Not giving more detailed answers because I don't want to help skynet take over the world, but what is the name of this forum??? Hello???? There was already news from south korea this week. I'ts not hard to see how a rogue AI could destroy people's lives. We don't need some scifi nuclear launch nonsense to do that.
Problem is, when banks get hacked it's always a bank security issue. Vulnerabilities don't go away when you remove AI from the equation, it just becomes easier to hide.
I agree that sci-fi nuclear launch scenarios are pie-in-the-sky fearmongering, but the current exploits are a reflection of the fast-and-loose security culture that festers in larger orgs.