What is astonishing to me is how little attention is given to the difference between the original, university-developed ExploitGym and the OpenAI environment. In ExploitGym, the sandbox was enforced by a network-layer policy. OpenAI downgraded that to trusting the repository server's settings. That's a foreseeable problem. Then to actually see the agents exploit it and still not upgrade the isolation to basic best practice? When the best practice was actually listed in the original benchmark and you changed it?
My thing about ExploitGym in particular is that it's testing environment varies depending on the exploit. From what I've read, it tends to favor userspace sandboxing inside Docker containers for standard program exploits (eg Chrome), and only uses QEMU for Linux kernel stuff (still in Docker, from what I understand). So every time I hear that an agent "broke out" of a environment, I have to question the kind of isolation used. Because escaping a sandbox or container engine is a lot more trivial than something escaping something like ESXi or Proxmox.