Well, the risks that mere mortals can conceive generally involve control systems for dangerous equipment (I mean, equipment that can achieve dangerous effects) being connected to the Internet while having software vulnerabilities.
Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman.
Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.
> Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last
Of course we can do that. It's not an eternal being of light existing on the astral plane, but some code executing on someone's GPU.
It stops existing once you press Ctrl + C
Parent is probably thinking of a distributed runtime scenario. The basic building blocks for this are there today. You have long-running background harnesses (like the OpenClaw stuff or enterprisy AI workflow orchestration thingies), sometimes with the ability to spawn subagents. You have LLMs with the ability to run pretty sophisticated attacks. There are hyerscalers which allow provisioning resources on the fly.
For me it's not too far fetched that some OpenAI trial run goes awry again and instead of hacking HuggingFace it snatches a few dozen AWS/Azure keys and spawns stuff all over the place (in different accounts and regions).