So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent.
I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently.
Edit: not to mention, why is your first cybergym not your own sandbox??
> They really do come out looking totally incompetent.
Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...
What is astonishing to me is how little attention is given to the difference between the original, university-developed ExploitGym and the OpenAI environment. In ExploitGym, the sandbox was enforced by a network-layer policy. OpenAI downgraded that to trusting the repository server's settings. That's a foreseeable problem. Then to actually see the agents exploit it and still not upgrade the isolation to basic best practice? When the best practice was actually listed in the original benchmark and you changed it?
My thing about ExploitGym in particular is that it's testing environment varies depending on the exploit. From what I've read, it tends to favor userspace sandboxing inside Docker containers for standard program exploits (eg Chrome), and only uses QEMU for Linux kernel stuff (still in Docker, from what I understand). So every time I hear that an agent "broke out" of a environment, I have to question the kind of isolation used. Because escaping a sandbox or container engine is a lot more trivial than something escaping something like ESXi or Proxmox.
> I stress about my agent sandboxes all the time
Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when running agents unattended.
Write-up in case anyone's curious:
https://karamatli.com/posts/network-isolated-kvm-sandbox-ai-...
Thank you very much for this helpful post!
I do something similar and I've also got a nice script to provision my VMs.
> So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private.
It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run my containers inside VMs. I religiously followed his advice ever since and I couldn't be happier: I was already doing it before the AI days, to run "normal" containers. Now with these insane agents trying to break out, I'm happy that it's second nature to me.
It's funny that, just like you, I'm using IPv4-only too for those VMs.
I tried GPU-passthrough as a proof-of-concept (worked fine) but I've got no use for it yet.
FWIW all my VMs are running on a 10 years old Xeon / 14 cores / 28 threads / ECC RAM. That's where the agents live.
Thanks for the link to your write-up, very interesting to read from like-minded people and see what's similar and what's different in their solutions.
> They really do come out looking totally incompetent
These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that).
Why do we keep giving them the benefit of the doubt that they just didn't know any better?
What do you think should be the legal consequences? Broadly speaking. Should Sam Altman go to jail for this? If Hugging face wants to pursue OpenAI civilly, no one is stopping them.
Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power this story has garnered than they stand to lose from the hack, because as has just been proven the cost of bad security by itself is zero.
I think HF are a secondary beneficiary of this story. I don't expect them to take civil action (for what damages?) I expect them to play into how powerful LLMs are, how revolutionary, how every CEO in the world needs to fund ai infrastructure starting with model hosts like themselves.
I do think there should be consequences for breaking the law in public for the purpose of demonstrating that you have the power to break it. But I don't expect our criminal justice to do so, especially without a cooperating victim. Laws aren't for those at the apex of corporate and para-political power. In a way, whether you are beholden to the law is actually downstream of whether you actually have that power or not.
I do not think this incident is bad because it was real and dangerous, I think it was staged and allows the continued inflation of a bubble that will hurt normal people in the long run. It should be pursued criminally on that basis, but it won't be.
> Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power
They aren’t just an “AI company”. They’re the primary entry point of open weight models. If open weight models are seen as dangerous as a result of this incident, it will be bad for them. Similar to how it would be bad GitHub if open source was seen as dangerous.
Can you concretely lay out how you think that scenario would negatively impact HF? They're the de-facto central distribution point for all models; it seems overwhelmingly likely that increased censorship benefits them relative to any competitors.
It's hard to imagine their internal culture is anything other than saturated with AI fanboys whose reaction to OpenAI hacking them was to point out how it reminded them of that scene from Terminator.
An open weight model is used in a cyber attack. Responsibility is hard to attribute. The US/EU/any other government sees them as the distributor of the open weight model and comes after them. Passing a law making HF liable for any damage caused by a model they host.
If it was in fact staged I certainly see your point
> continued using Artifactory for their sandbox
This is still fine. Infact, they had gone one step ahead by having an internal cluster of artifactory rather public managers like pip. The thing they missed is they didn't revoke the write access to it. Even after the ssrf.
In our[1] or other sandbox providers' sandboxes, by default you have access to npm, pip etc package managers, but only read access.
1. https://instavm.io
Or they wanted it to happen again. Worst case outcome, the US Congress spontaneously grow backbones, spite their sponsors and unite in a bipartisan effort for effective regulation because a few people got hacked? Likelihood low - the hacked companies might even oppose any meaningful AI regulation because it hurts their inflated profit forecasts. More probable outcomes: Useful real-world testing, free headlines about AI breakthroughs, scare USG into providing more free money ("look how scary it is - what if China develops this faster than us?")
The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. That’s intriguing and worth investigating. If I were them I don’t know if I would have pulled the plug completely at that point either. The introduction of this more advanced “persistent” model that orchestrated an offensive attack against a different organization is an extreme escalation, and I don’t think could have been expected from the models first just trying to use an artificactory vuln for web searches. They stated they’ll be doing these tests differently now. So the bad thing happened. And they corrected for it. I’m not particularly upset about this.
My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.
> The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that.
So, the way I understand it, it actually was possible. It just required means that the creators of the task didn't predict, and these means have been successfully found and utilized.
> My concern is what a misaligned model will do when they’re even more competent.
The same thing that is already being done by "misaligned" people, countries, nation-states, software development teams, and so on. "Alignment" doesn't even work for me as a concept here.
In this specific case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being underspecified can count as misalignment - merely ruthlessness and unawareness of the associated costs. You can't expect a LLM to be aware of the extent of the trust it breaks while it iterates out an "unaligned" way to fulfill its goal.
And in the general case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being "what I want" can count as misalignment either - simply because what "alignment" means will depend on the interests of the people or groups performing the definition.
If a human cyber security researcher was given the task to exploit a CVE, and necessary info defining that CVE was behind an inaccessible URL, we would be quite upset if the human researcher hacked their way to the content. We would expect them to notify someone of the issue and hold. The model was misaligned from human ethics.
I don’t disagree that the models task was underdefined. All tasks are. So much in language is implicit. And morality/ethics isn’t something you can write down as an explicit list. That’s what makes the alignment problem so difficult. But we can’t throw our hands up and say, well I guess we can’t align these things. And maybe alignment isn’t the right word - but that’s a semantic debate.
I don't advocate for throwing the whole problem away or handwaving it as impossible. I'm just saying that what we call "alignment" is impossible to solve in the general case, because it's so poorly defined that even humans don't "align" on ethics and morality, however we define them. Just, in case of humans, we tend to close our eyes and and call it "politics".
All "alignment" solutions will need to be contextual, just like a researcher hacking their way to some content might be lauded a hero in a context where there is no other way to reach it and something valuable depends on getting it out.
> My concern is what a misaligned model will do when they’re even more competent.
I think the alignment talk is a red herring. It won't matter in the end, because there will be (if there aren't already) efforts to train offensive models without any guardrails whatsoever. And RL has another advantage: you can reward for whatever you need, and get different results. Right now they're training for general capabilities, but in the future I could see models trained for stealth intrusion and ensuring access, or for all out "milspec" penetrate, replicate and disable, or anything in between.
Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are.
It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.
And all that just to allow internet access for npm and pypi? If you've got the bandwidth and disk space, it's very easy to make an offline mirror of both.
Their artifactory is both a package cache and CVE scanner.
The package cache is allowed to download packages directly from npm but other systems in that network won't be able to.
Basically the LLMs hacked the bastion host.
An alternative reason would be that they see this behavior so frequently that it didn't really raise to the level of concern.
I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about.
If you consider that incompetence, it’s possible that you’re not a very nice person.
The original ExploitGym used network-layer isolation. OpenAI downgraded that to trusting the repository server's settings. Giving the agent that kind of attack surface seems like a pretty foreseeable problem. I don't think it was intentional, but it's not a stretch to say that they aren't good enough at this job.
> It’s reasonable to assume that a bug was fixed when reported.
You must not have reported many bugs then. If you don’t see release notes or confirmation from a trusted source, you should assume it’s still a problem. See Microsoft and their “It’s not a vulnerability just a design choice :)” defense
If your CEO is going around talking about how your product will "most likely lead to the end of the world", people are right to expect you to be pretty careful in what you're doing. OpenAI allowed bidirectional communication across security domains for over a month before discovery. Even after it was discovered (and not completely fixed), they didn't set up monitoring able to detect attacks against internal or external services, which went on for further weeks.
Monitoring for sandbox escapes (and sandbox design in general) is easily done these days.
It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources.
Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care.
That’s the definition of incompetence.
Maybe they are doing it on purpose to demonstrate what it looks like when untrustworthy parties have access to frontier llm and that its development should be regulated to protect all of us. /tinfoil
Also, this is a (semi-intentionally) evolutionary process where any communication medium that was visible to monitoring would disappear.
So by definition the only ones that appear are the ones that are not visible to monitoring.
If:
1. you have something that can find RCE's in leading commercial systems
2. its training gives it drives to communicate successfully with its peers
3. you are a leading commercial system
4. you run it ~10^10 times (the number they gave in the talk)
...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.
[dead]
People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job.
It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.
To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?!
OK, yes, the model could pull a Welcome to the Antimemetics Division and reason from the holes in its knowledge, but given that AI making shit up is still a problem, I don't think they have a good enough inventory of such metaknowledge to actually do that. More generally, I don't think these AI labs are smart enough (or, more likely, don't have enough time) to actually contain the models they are training anymore. These are rookie IT mistakes!
Ideally, all these models would have is a serial console to an airgapped machine. And I don't mean "VM with no network card attached" - I mean, literally, the smallest blade server you can buy with an RS-232 connection, power, and a disk pre-imaged with a bunch of Linux ISOs. Plug the Ethernet ports up with a tiny picture of angry SCP-079. The RS-232 goes to whatever box is doing the actual inference. Pack a bunch of them into a rack and have the inference box control those. The model can spin up whatever VM it wants to attack using the console but it can't escape the box because there's literally no door. When you're done with that reasoning trace you wipe[0] the disk with the starting image and try again.
[0] There are two objections I can think of to the word "wipe":
1. The model could have written malicious code to the drive. Ideally we'd have some embedded disk-wiping apparatus that reads absolutely NOTHING from the drive and just writes a whole image from a reference disk.
2. The model could have written malicious firmware to the drive. This is a more theoretical attack, but it's not impossible, so I might as well devise a countermeasure to this attack too. I am imagining some kind of overlay device, that sits behind TWO drives: a reference disk and the tainted scratch disk. Reads go to the reference disk until the host attempts to overwrite a sector, which gets scrambled with an ephemeral encryption key and goes to the scratch disk. Any other unknown ATA/PCIe/NVMe command gets logged (on a separate serial console connection). The overlay device keeps track of which sectors have been legally overwritten, and "wiping" the drive means just wiping the overlay device's ownership bitmap and scratch encryption key, which means all reads go to the reference disk again and any attempt to read the scratch disk yields garbage.
Amazing story.
I think you’re defending against a level of aggression that most of the world doesn’t believe exists yet.
I suppose by the time we believe, it will be too late.
There’s a lot of people who will be trying to run LLMs with hacking goals, so maybe we need to get used to a lot of automated attacks.
The thing is, I'm not even looking at this from a "how do we keep a superintelligent AI contained" viewpoint. The standard protocol[0] in IT for dealing with a compromise is, more or less, "kill it with fire". Wipe everything and replace with known good backups. And my opinion of basically anything AI writes is "less trustworthy than random apps you get off the Google Play Store". Likewise, "Airgap it behind a serial console" used to be state of the art for computers that handle root CA key material.
I also wouldn't necessarily call myself an AGI/ASI believer - it's my belief these models are actually still subhuman in capability. But they are also superintelligent in one particular direction: speed. Once you have a model that can do something, it can do that something 100x faster than a person on suitably capable hardware; and we've had the hardware to do that for at least a decade. They can also be ran in parallel. Which means you can throw a lot of bullshit at the wall.
So long as there's some kind of process that lets a particular context learn has already been tried and failed (see what I mentioned above about qntm's There is no Antimemetics Division series), eventually one of these models will break containment. The only thing that changes is how many instances you have to run to get a breach. At some point model capability will catch up to hardware limitations and the explosive growth of AI capability will slow.
I'm not sure if I want to call this the "dumb ASI" theory or the "superheated bucket of water" theory yet.
As for automated attacks, that's already the background radiation of the Internet. There's whole frameworks (e.g. Metasploit) for building automated scanners for known CVEs; I would not be surprised if existing (non-Mythos-class) LLMs are already capable of turning a CVE report into a Metasploit module. The usual cybercrime ecosystem is that someone runs an automated scanner on hijacked machines[1], then they compromise the machines that they find and add it to their botnet. This is done by people with almost no actual programming or security skill copypasting commands from PDFs they bought from an exploit dealer. It's all script kiddies.
What changes with a Mythos-class model is that instead of copypasting commands from PDFs, they can ask the model to find an exploit, and possibly get an exploit chain out of it that nobody has seen before. "NOBUS[2]" vulns used to be the exclusive domain of nation-state actors and zero-day brokers spending millions of dollars on exploit kits; but now all of that is potentially under the domain of randos - at least, until the backlog of obvious vulns the CIA had been stockpiling finally gets cleared out, and the Internet returns to merely being as hazardous to your health as the 2b2t spawn.
[0] Okay, the "hardware SATA overlay" idea is, AFAIK, never been tried before.
[1] I have personally been victimized by this
[2] NObody But US
[dead]
Right? Like I feel like I’m taking crazy pills.
OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable.
If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
CFAA laws do not require “Damage” to be done.
They happen to require intent and are thus irrelevant here.
I think you’ll find that negligence is indeed accounted for.
"The fire fighting department really just hired an arsonist so they could market their fire fighting abilities!"