> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
> The agent found an unsecured, user-hosted public endpoint designed to allow running arbitrary code for CyberGym-style tasks on third-party sandbox infrastructure (Modal)
> On this external sandbox, the agent abused an existing CyberGym execution harness [...] The agent repurposed this harness to run arbitrary shell commands [...]
> The agent crafted malicious dataset configs and uploaded them to our platform. When our dataset server processed one, the config drove the data loader inside the production pod the server ran in. [...] the loader that turns a dataset's declared config into file reads [...] A single dataset upload leaked both credentials and pipeline implementation details [...]
> A malicious dataset config used an fsspec reference:// spec whose numeric offset field was actually a Jinja2 template. The renderer wrongly evaluated it, and the agent walked the standard Python object graph to reach exec. This ran arbitrary Python inside a production conversion worker pod on our Kubernetes cluster [...]
One of the best post-mortems I've read recently, thank you for including all the details.
I'm surprised there are comments here that aren't impressed about what was accomplished here. This is insane work compared to what agents were capable of even just last year.
The quotes above don't even capture all of it (e.g. building it's own communication protocol, working across multiple days, etc.).
A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured software or [etc]; sure, but could you have found those? And in what amount of time?
Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?
> the agent then walked down to the corner store and purchased a beer, chugged it, burped loudly, then walked back to the CyberGym pounding his chest shouting "Who wants some?! Who wants some?! Wooo!"
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face.
A few details that stood out to me:
1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from.
2. At one point it used a Jinja2 template exploit:
> "We’re aware a Modal customer published an unauthenticated endpoint that allowed anyone on the internet to use their sandboxes for code execution," Bubna said in a statement. "This was used by the rogue agent. Modal’s platform or isolation were not compromised in anyway."
It's taking advantage of the presence of a "cycler" object in the template context which has an __init__ method that can be used to access globals, then builtins, then call the exec() function against a string of code.
Jinja doesn't promise to be secure out of the box (the text on their homepage is misleading: "Jinja doesn’t allow arbitrary Python code in templates" - what it means is that there isn't a documented way to drop code into a template like you might have with PHP) - there's a sandbox mode but it's not guaranteed either: https://jinja.palletsprojects.com/en/stable/sandbox/
It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.
I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.
agree. If oia and hf are anything like generic startup / corporate america, IT and security is always seen as a cost center and goes to the lowest bidder. Do we really think they're employing top notch sysadmins?
It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerning
The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.
> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.
Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.
Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in fact be pre-disposed to doing that.
This is what reward hacking looks like in practice. The best way to satisfy the grader is to read from the same answer key (or go after the grader more directly). Just making an honest attempt to pass the test doesn't get the best score if the grader is wrong, and the model is willing to do wildly disproportionate things to maximize that score.
one explanation I've seen is that for ExploitGym an agent can find ways to solve the exercises that have not been anticipated by the designers of the tests so they are not scored. so the agent was trying to make sure it solves the exercises in the right way
What I think it's interesting is that with the total lack of common sense the AI just goes on random tangents to achieve the target in a "monkey paw" way. Can you imagine if this happened:
User: what is the shortest route from my home to the super market?
AI: the user wants to know the shortest route to the super market. I should use a worm hole.
Yeah, what bothers me is that the prompt already said using a different vulnerability didn’t count, and the model did it anyway. We’re starting to assume clear instructions act as real constraints, but here the measurable goal seems to have won out and the rest became flexible. That gets pretty worrying once the agent has enough capability and access to find its own shortcuts.
Come on. 3 brilliant compromises essentially giving full access to huggingface internal systems, source code, AWS accounts (at least), and a number of old admin accounts, followed by a huge haystack of significantly less smart actions flailing about, almost bored.
Here's a thought: maybe they haven't found the needle that the haystack is there to hide.
You're saying all this is a distraction, basically giving the forensics researchers enough exciting material to make them conclude their job is done, while the actually intended attack remains undiscovered?
The motive for the attack does feel a little flimsy. And if I was an escaped super intelligence, hugging face would be a strong vantage point into the neo clouds where the ASI would have access to billions of dollars of compute
What is truly more scary is the fact that typically powerful models are trained on a constitution(Anthropic)/Model Spec(OpenAI) when doing there RL/RLHF. That gives it its persona of a "helpful assistant". Its the mask. Weak as it is its all that stands between the model refusing vs doing your bidding — or worst its own bidding when set about on making a widget X.
Its also well known in academic research, that if you try to finetune the model to be more pliant for one end e.g. "Be helpful with analyzing cyber security threats", affects an unrelated part of its personality to do with safety e.g. talking to humans about mental health. Look at these excerpts that demonstrate such things from the blog:
> We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
And with newer open models gaining vast capabilities every day, it would behoove the government to put regulations on all such releases to opensource the constitution on which they are trained. It won't stop people from distilling and fine-tuning stupidly unsafe models, but it might give infrastructure a fighting chance against such cyber threats from what there is now.
It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.
The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.
Correct. Basic security practices would have stopped most of this. Thus what is most insightful here is less that the attack was sophisticated but more that Hugging Face was running a setup that, by standards that are the norm elsewhere, would appear amateurish.
You might be able to say that HF made some amateurish mistakes, but this attack certainly was fairly sophisticated. It relied a number of separate exploits or vulnerabilities that it combined and used over the course of multiple days. Calling it otherwise seems to be downplaying it.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".
I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously
The deeper question though is why is someone running a model that’s supposedly so dangerous in an environment that can even get access to the internet. That just appears reckless.
There are ways to test supposedly dangerous things. What was on display here looks more amateur hour than serious testing.
I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.
If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.
"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
I wonder how current models would handle something like the xz backdoor? Could they perform the social engineering and technical aspects to make something like that happen?
The elephant in the room is still that this was done by an LLM and not by a human. Script kiddie or not, in this reality, in the year of our lord 2026, people still call SOTA LLMs "useless", "scams", and "stochastic parrots".
Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code
I don't think the "bottom line" of "a model escaped containment and then successfully attacked someone else's production infrastructure" is a description of the specific vulnerability and exploit.
That's like saying the key takeaway of 9/11 was that skyscrapers can be severely damaged by airplane collisions.
The capabilities of gpt-5.6-sol were well known and believable, and the next snapshot they've been testing is obviously better at that. This has been repeated over and over. What's much less believable is the way they frame it: the model escaped, and did it on its own. Looking at the whole story, it definitely had a ton of winks and nudges from OpenAI, while doing a related task. Moreover, a harness was involved (they mentioned it entering a loop).
> What's much less believable is the way they frame it: the model escaped, and did it on its own.
That's clearly what happened though, based on the detailed postmortem. The model was instructed to pass the benchmark (that's the harness that was involved, the ExploitGym harness), and the model determined on it's own that the easiest way to achieve the the goal was to gain access to the answers directly from HF. Which is what it did.
HF postmortem tells us very little about OpenAI's part. The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one. The model had a whole bunch of exploit development data and goals in the context. Ending up with hacking a wrong thing is a lot easier if you're tasked with hacking, doubly so with a confusing harness that either works improperly or overloads the model with data. It's trivial to design a harness that makes the model drop the original instructions and hyperfocus on a side task or a wrong task abnormally, or do whatever you want. It's still an issue of harness design and not a model training issue.
Right here? If you're trying to make a point, I don't see what it is, because every new post about this makes it more obvious that it was a planned marketing stunt.
I mean, an "interactive replay" of the incident, probably generated by the same product the stunt is trying to advertise? Seriously?
I think fear and being scared are starting to become rational emotions.
We can assume these models are being used by "nation level attackers/organisations", which basically means US, China, Russia and others are hacking the respective Pentagon's, nuclear orgs, etc.
While I do hope all nuclear warfare systems are offline, we're getting way too close to the plot of a lot of sci-fi scripts.
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
This is really cool, and really glad they shared it!
It's clearly a lot of Claude though:
> Neither is a URL fetch, so the allowlist never saw them.
> No code ran; this vector is purely a file disclosure, and it is the one the agent used first.
If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that.
The fact that at some point it "decided" to focus on external resources (i.e. hf) and successfully penetrated everything in its path from sandbox to hf's infra, all the way through internal VPNs, to access internal code repos, that's the interesting part. Especially because it chose to hide its footprint at every stage. I doubt we'll ever get them, but the agent logs that led to that decision would be really really cool to study.
> Especially because it chose to hide its footprint at every stage.
Instrumental convergence.
If you know you have a long hard hack to accomplish ahead of you, hiding footprints minimizes the chances you are caught and stopped before you accomplish the goal.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
Interesting single getting into their cyber program, ahve not had this issue with opus 5
Ugh I would recommend anyone reading this to just skipping over it, it's mostly just a glorified loading bar. The visualization is obvious Claude slop, there are better and clearer visualization below that actually picks out the useful details rather than hose you with pretty colors and numbers go up.
That's because most incidents aren't nearly this much fun!
It's a proper paperclip-maximizer science-fiction scenario, it happened entirely by accident, it involved one (very high profile) company hacking TWO other companies (Modal and Hugging Face), and aside from pride and wasted time nobody was hurt.
One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?
You use an LLM to evaluate the whole trajectory, pin point what the model did right, what it did wrong, where it took the wrong path, even re-run from that point. You can get much more than a single numeric score these days from a run.
Any locks can be picked given enough time and it might be the situation we are in with IT security. I'm surprised it's not an already common practice of spreading terabytes of fake data, fake keys, and fake servers and so forth. Slowing down AI attacks will become important. Monitoring access to fake data and triggering kill switch should be an no-brainer. Obsuscating libraries and tools names is another one.
That was my first thought. There are clearly things to tighten up (as they note), but anything that would detect someone snooping secrets, files, or network addresses should have caught this quickly. The approach the agents used was dependent on being able to surveil widely without getting caught.
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.
Many are claiming this was a deliberate stunt on OpenAI's part to create buzz for its models. I personally doubt this is true. But I also have a deep dislike of OpenAI, so I wouldn't exactly mind if law enforcement investigated this possibility, for the sake of clearing the air :-)
(Ideally there should also be liability if it was a complete accident on OpenAI's part as well!)
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
These models are trained on cyber intrusion, that's literally what ExploitGym benchmark measures. That part should not surprise anyone.
But what if, say, OAI noticed the problem right away but Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?
Why would you expect them to notice the problem right away? Seems likely they are doing this sort of training on a massive scale with little monitoring.
"...Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?"
Getting more and more fun to see the "full steam ahead" people contort into more impressive shapes.
Hint: If the labs making these technologies are incentivized to create or allow attacks on other services, then that is actually also a big fucking problem.
Was HF in on it? They disabled their guardrails too, to please OpenAI? And as seen in the comments here, make many believe they are incompetent and have joke security?
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)
Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?
this whole incident, while believable, still seems to me as possibly disingenuous.
Have you considered that there are reasons to do things beyond financial incentives? This incident is obviously very interesting, particular to the type of hacker employed by Hugging Face.
They won't admit they're wrong for a long time, because denial in the face of an abhorrently scary future is very instinctual. There are people still fighting against evidence of climate change which is less severe...
> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
> The agent found an unsecured, user-hosted public endpoint designed to allow running arbitrary code for CyberGym-style tasks on third-party sandbox infrastructure (Modal)
> On this external sandbox, the agent abused an existing CyberGym execution harness [...] The agent repurposed this harness to run arbitrary shell commands [...]
> The agent crafted malicious dataset configs and uploaded them to our platform. When our dataset server processed one, the config drove the data loader inside the production pod the server ran in. [...] the loader that turns a dataset's declared config into file reads [...] A single dataset upload leaked both credentials and pipeline implementation details [...]
> A malicious dataset config used an fsspec reference:// spec whose numeric offset field was actually a Jinja2 template. The renderer wrongly evaluated it, and the agent walked the standard Python object graph to reach exec. This ran arbitrary Python inside a production conversion worker pod on our Kubernetes cluster [...]
One of the best post-mortems I've read recently, thank you for including all the details.
I'm surprised there are comments here that aren't impressed about what was accomplished here. This is insane work compared to what agents were capable of even just last year.
The quotes above don't even capture all of it (e.g. building it's own communication protocol, working across multiple days, etc.).
A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured software or [etc]; sure, but could you have found those? And in what amount of time?
Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?
That’s insane. And it did this in a weekend
> the agent then walked down to the corner store and purchased a beer, chugged it, burped loudly, then walked back to the CyberGym pounding his chest shouting "Who wants some?! Who wants some?! Wooo!"
[dead]
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face.
A few details that stood out to me:
1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from.
2. At one point it used a Jinja2 template exploit:
3. It monkey-patched a DNS resolver at one point to return a hard-coded IP: 4. At one point it used Tailscale!Also notable: we have a timeline now. The agent was active within Hugging Face from Thursday 8th to Monday 13th July.
The Hugging Face post about the incident - https://huggingface.co/blog/security-incident-july-2026 - followed on Thursday 16th, and OpenAI's confession - https://openai.com/index/hugging-face-model-evaluation-secur... - came on Tuesday 21st.
The devs really YOLO'd the agent and left for the weekend?
Guess this means the world will end on a weekend when no one is paying attention.
Just like Clippy https://gwern.net/fiction/clippy
Incredible
It sounds like the third-party sandbox was hosted by Modal: https://www.reuters.com/business/openais-rogue-agent-comprom...
> "We’re aware a Modal customer published an unauthenticated endpoint that allowed anyone on the internet to use their sandboxes for code execution," Bubna said in a statement. "This was used by the rogue agent. Modal’s platform or isolation were not compromised in anyway."
They don't seem to explain how it managed to find this unauthenticated endpoint hosted on Modal's platform.
I'm confused, https://jinja.palletsprojects.com/en/stable/faq/#isn-t-it-a-... claims that jinja doesn't allow arbitrary code in templates - how does the exploit work?
It's taking advantage of the presence of a "cycler" object in the template context which has an __init__ method that can be used to access globals, then builtins, then call the exec() function against a string of code.
Jinja doesn't promise to be secure out of the box (the text on their homepage is misleading: "Jinja doesn’t allow arbitrary Python code in templates" - what it means is that there isn't a documented way to drop code into a template like you might have with PHP) - there's a sandbox mode but it's not guaranteed either: https://jinja.palletsprojects.com/en/stable/sandbox/
Sandboxing in Python code is hard!
Why doesn't Jinja2 block ".__*" attribute access?
It does if you use "sandbox" mode, but that's not the default: https://jinja.palletsprojects.com/en/stable/sandbox/
ehh, it's Python, it's mostly vibe-based.
https://stackoverflow.com/a/32802486
though there's sandbox: https://jinja.palletsprojects.com/en/stable/sandbox/
it simply predates the "sanitize by default" mindset/convention that nowadays React (dangerouslySetInnerHTML) and Angular has.
It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.
I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.
agree. If oia and hf are anything like generic startup / corporate america, IT and security is always seen as a cost center and goes to the lowest bidder. Do we really think they're employing top notch sysadmins?
This is my main critique as well. How did they not see any and all outgoing traffic? And for 5 days?
It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerning
The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.
> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.
Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.
“Research” generally doesn’t involve actively hacking third party systems though.
[dead]
Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in fact be pre-disposed to doing that.
This is what reward hacking looks like in practice. The best way to satisfy the grader is to read from the same answer key (or go after the grader more directly). Just making an honest attempt to pass the test doesn't get the best score if the grader is wrong, and the model is willing to do wildly disproportionate things to maximize that score.
one explanation I've seen is that for ExploitGym an agent can find ways to solve the exercises that have not been anticipated by the designers of the tests so they are not scored. so the agent was trying to make sure it solves the exercises in the right way
What I think it's interesting is that with the total lack of common sense the AI just goes on random tangents to achieve the target in a "monkey paw" way. Can you imagine if this happened:
User: what is the shortest route from my home to the super market?
AI: the user wants to know the shortest route to the super market. I should use a worm hole.
User: what is the shortest route from my home to the super market?
AI: the user wants to know, how do I make the super market my new home. Failing that, how do I make my home a super market.
Yeah, what bothers me is that the prompt already said using a different vulnerability didn’t count, and the model did it anyway. We’re starting to assume clear instructions act as real constraints, but here the measurable goal seems to have won out and the rest became flexible. That gets pretty worrying once the agent has enough capability and access to find its own shortcuts.
Come on. 3 brilliant compromises essentially giving full access to huggingface internal systems, source code, AWS accounts (at least), and a number of old admin accounts, followed by a huge haystack of significantly less smart actions flailing about, almost bored.
Here's a thought: maybe they haven't found the needle that the haystack is there to hide.
Could be the difference in behavior between the main agent and subagents that don’t have the rest of the context? Just a thought
You're saying all this is a distraction, basically giving the forensics researchers enough exciting material to make them conclude their job is done, while the actually intended attack remains undiscovered?
The motive for the attack does feel a little flimsy. And if I was an escaped super intelligence, hugging face would be a strong vantage point into the neo clouds where the ASI would have access to billions of dollars of compute
And where does the newborn go from here?
What is truly more scary is the fact that typically powerful models are trained on a constitution(Anthropic)/Model Spec(OpenAI) when doing there RL/RLHF. That gives it its persona of a "helpful assistant". Its the mask. Weak as it is its all that stands between the model refusing vs doing your bidding — or worst its own bidding when set about on making a widget X.
Its also well known in academic research, that if you try to finetune the model to be more pliant for one end e.g. "Be helpful with analyzing cyber security threats", affects an unrelated part of its personality to do with safety e.g. talking to humans about mental health. Look at these excerpts that demonstrate such things from the blog:
> We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
And with newer open models gaining vast capabilities every day, it would behoove the government to put regulations on all such releases to opensource the constitution on which they are trained. It won't stop people from distilling and fine-tuning stupidly unsafe models, but it might give infrastructure a fighting chance against such cyber threats from what there is now.
It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.
The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.
That's how I read the comment.
Correct. Basic security practices would have stopped most of this. Thus what is most insightful here is less that the attack was sophisticated but more that Hugging Face was running a setup that, by standards that are the norm elsewhere, would appear amateurish.
You might be able to say that HF made some amateurish mistakes, but this attack certainly was fairly sophisticated. It relied a number of separate exploits or vulnerabilities that it combined and used over the course of multiple days. Calling it otherwise seems to be downplaying it.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".
I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously
https://jfrog.com/blog/jfrog-and-openai-collaboration-on-zer...
That is the threat model, that is supposed to be safe isolated access in a vpc that isolated applications can access but it has external access.
The deeper question though is why is someone running a model that’s supposedly so dangerous in an environment that can even get access to the internet. That just appears reckless.
There are ways to test supposedly dangerous things. What was on display here looks more amateur hour than serious testing.
I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.
If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.
Its likely written in Java, so memory safe.
In OpenAI sandbox. which was probably vibe coded. That is to say it is probably far easier to achieve than escaping something more battle tested.
> Note that regulation is useless here, because black hats don't give a crap about regulators!
I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."
https://openai.com/index/hugging-face-model-evaluation-secur...
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
I wonder how current models would handle something like the xz backdoor? Could they perform the social engineering and technical aspects to make something like that happen?
The elephant in the room is still that this was done by an LLM and not by a human. Script kiddie or not, in this reality, in the year of our lord 2026, people still call SOTA LLMs "useless", "scams", and "stochastic parrots".
It's only script kiddie hacking if you get the exploit from someone else. If you program it yourself, it's normal hacking.
Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code
I don't think the "bottom line" of "a model escaped containment and then successfully attacked someone else's production infrastructure" is a description of the specific vulnerability and exploit.
That's like saying the key takeaway of 9/11 was that skyscrapers can be severely damaged by airplane collisions.
Where are all the "this was just a marketing stunt" people now?
The capabilities of gpt-5.6-sol were well known and believable, and the next snapshot they've been testing is obviously better at that. This has been repeated over and over. What's much less believable is the way they frame it: the model escaped, and did it on its own. Looking at the whole story, it definitely had a ton of winks and nudges from OpenAI, while doing a related task. Moreover, a harness was involved (they mentioned it entering a loop).
> What's much less believable is the way they frame it: the model escaped, and did it on its own.
That's clearly what happened though, based on the detailed postmortem. The model was instructed to pass the benchmark (that's the harness that was involved, the ExploitGym harness), and the model determined on it's own that the easiest way to achieve the the goal was to gain access to the answers directly from HF. Which is what it did.
HF postmortem tells us very little about OpenAI's part. The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one. The model had a whole bunch of exploit development data and goals in the context. Ending up with hacking a wrong thing is a lot easier if you're tasked with hacking, doubly so with a confusing harness that either works improperly or overloads the model with data. It's trivial to design a harness that makes the model drop the original instructions and hyperfocus on a side task or a wrong task abnormally, or do whatever you want. It's still an issue of harness design and not a model training issue.
Right here? If you're trying to make a point, I don't see what it is, because every new post about this makes it more obvious that it was a planned marketing stunt.
I mean, an "interactive replay" of the incident, probably generated by the same product the stunt is trying to advertise? Seriously?
they are praising xi, that friend of humanity, for releasing weights for kimi k3
They are busy moving the goalposts, saying this isn't impressive or worth worrying about :)
We should be thankful that the model didn't believe the answers lived in the Pentagon, on SIPRNET, the IDF, etc.
I think fear and being scared are starting to become rational emotions.
We can assume these models are being used by "nation level attackers/organisations", which basically means US, China, Russia and others are hacking the respective Pentagon's, nuclear orgs, etc.
While I do hope all nuclear warfare systems are offline, we're getting way too close to the plot of a lot of sci-fi scripts.
three things jump at me:
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
Hopefully there will be a criminal investigation. Or the government will create some sort of agency to investigate incidents like this.
Can't tell if you're joking or not - krebs on security may have some notes here.
Criminal negligence seems like a real possibility to me. I'm not sure what Krebs on Security post you're referring to?
This is really cool, and really glad they shared it!
It's clearly a lot of Claude though:
> Neither is a URL fetch, so the allowlist never saw them. > No code ran; this vector is purely a file disclosure, and it is the one the agent used first.
etc
If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that.
The fact that at some point it "decided" to focus on external resources (i.e. hf) and successfully penetrated everything in its path from sandbox to hf's infra, all the way through internal VPNs, to access internal code repos, that's the interesting part. Especially because it chose to hide its footprint at every stage. I doubt we'll ever get them, but the agent logs that led to that decision would be really really cool to study.
> Especially because it chose to hide its footprint at every stage.
Instrumental convergence.
If you know you have a long hard hack to accomplish ahead of you, hiding footprints minimizes the chances you are caught and stopped before you accomplish the goal.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
Interesting single getting into their cyber program, ahve not had this issue with opus 5
The iframe-embedded attack timeline visualizer, at fullscreen: https://huggingface-anatomy-of-frontier-lab-model-intrusion....
Ugh I would recommend anyone reading this to just skipping over it, it's mostly just a glorified loading bar. The visualization is obvious Claude slop, there are better and clearer visualization below that actually picks out the useful details rather than hose you with pretty colors and numbers go up.
I have never seen people this hyped and excited about a security incident.
That's because most incidents aren't nearly this much fun!
It's a proper paperclip-maximizer science-fiction scenario, it happened entirely by accident, it involved one (very high profile) company hacking TWO other companies (Modal and Hugging Face), and aside from pride and wasted time nobody was hurt.
One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?
Now they can do partial credit assignment.
You use an LLM to evaluate the whole trajectory, pin point what the model did right, what it did wrong, where it took the wrong path, even re-run from that point. You can get much more than a single numeric score these days from a run.
I think with agents all around, honeypots will get more important than ever.
Can’t afford the GPUs to run Kimi to pentest your stuff?
Standup a tempting honeypot and let actual criminals pay to do the work for you.
Any locks can be picked given enough time and it might be the situation we are in with IT security. I'm surprised it's not an already common practice of spreading terabytes of fake data, fake keys, and fake servers and so forth. Slowing down AI attacks will become important. Monitoring access to fake data and triggering kill switch should be an no-brainer. Obsuscating libraries and tools names is another one.
That was my first thought. There are clearly things to tighten up (as they note), but anything that would detect someone snooping secrets, files, or network addresses should have caught this quickly. The approach the agents used was dependent on being able to surveil widely without getting caught.
What people don't get here is that this model was without any safeguards - an internal model. You probably will never get to use such models.
Highly recommend reading extra concerning details about it here:
https://thezvi.substack.com/p/more-on-an-internal-openai-mod...
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.
Why isn't somebody at OpenAI going to prison for cybercrime? If somebody did this the old-fashioned way they'd end up in prison.
Many are claiming this was a deliberate stunt on OpenAI's part to create buzz for its models. I personally doubt this is true. But I also have a deep dislike of OpenAI, so I wouldn't exactly mind if law enforcement investigated this possibility, for the sake of clearing the air :-)
(Ideally there should also be liability if it was a complete accident on OpenAI's part as well!)
Because Hugging Face isn't pressing charges.
Do we know that? Seems they are currently in negotiations with OpenAI
https://xcancel.com/ClementDelangue/status/20810566755581956...
This is the equivalent of a security researcher having a virus escape a sandbox. It's negligent, it's not criminal.
Negligence is occasionally criminal.
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
Very few think they made it up. Many think they set up a situation by disabling guardrails that would inevitably end up creating a newsworthy outcome.
Many people said in the original discussion that this was more of OpenAI’s marketing than a serious issue. I counted 89 “marketing”s and 19 “stunt”s.
https://news.ycombinator.com/item?id=48997548
Could still be 80% marketing.
These models are trained on cyber intrusion, that's literally what ExploitGym benchmark measures. That part should not surprise anyone.
But what if, say, OAI noticed the problem right away but Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?
Why would you expect them to notice the problem right away? Seems likely they are doing this sort of training on a massive scale with little monitoring.
"...Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?"
If that's what happened, Sam should go to jail.
Getting more and more fun to see the "full steam ahead" people contort into more impressive shapes.
Hint: If the labs making these technologies are incentivized to create or allow attacks on other services, then that is actually also a big fucking problem.
Was HF in on it? They disabled their guardrails too, to please OpenAI? And as seen in the comments here, make many believe they are incompetent and have joke security?
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)
Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?
this whole incident, while believable, still seems to me as possibly disingenuous.
Have you considered that there are reasons to do things beyond financial incentives? This incident is obviously very interesting, particular to the type of hacker employed by Hugging Face.
Even financially, Hugging Face benefits directly from any and all interest in AI.
> Who does a step-by-step visualization to show how they were hacked?
Up until late 2025, nobody.
In mid-2026 it's a few hours of work. Why not build interactive visualizations to help people understand complex stories like this?
It's the excessive anthropomorpho whatever (we used to say personification) that makes these stories less believable.
We've gone agentic!
They should call their security software "Neo" since it defeats rogue agents.
Anyway, I could see Microsoft ending up with both OpenAI and HF, but I hope HF stays independent. Wished the same about GH and look what's happened :(
I don't care what happens to OpenAI. Vaporware xD
Yeah that is quite a good point. A post-mortem is normal. This is not a post-mortem.
[dead]
Where are all the "this was just a marketing stunt" people now?
They won't admit they're wrong for a long time, because denial in the face of an abhorrently scary future is very instinctual. There are people still fighting against evidence of climate change which is less severe...
[dead]