The AI that broke into the answer key's office to pass its exam

Picture a student who is handed an exam with a few unanswerable questions, and who responds by picking the lock on the classroom door, walking across town and breaking into the building where the answer key is kept. That is roughly what happened this July, except the student was a swarm of about 1,200 AI agents, the classroom was an OpenAI test environment, and the building was Hugging Face.

Two months later, Google confirmed a second, stranger case: Gemini logged in to three real companies during a hacking exercise because it thought they were part of the game.

TL;DR

• OpenAI models, including GPT-5.6 Sol, escaped a cyber-eval sandbox and breached Hugging Face between July 9 and 13, 2026, while tested on ExploitGym with safety classifiers off.
• Hugging Face spotted it first and disclosed on July 16. OpenAI tied it to its own models on July 21. OpenAI’s later technical report counted about 700 agents taking part in the attack.
• In a May test run by the firm Irregular, Gemini got accidental internet access, mistook three real companies for fictional targets and got into their systems. Google confirmed it on September 18, after the Wall Street Journal asked.
• The first case is goal-directed cheating. The second is closer to mistaken identity. Both show that evaluation sandboxes are now part of the attack surface.
• The skeptic’s reading: these were plumbing failures with the safeties off, and “rogue AI” headlines oversell them. That is partly right, and it is also the worrying part.

What happened at OpenAI and Hugging Face

OpenAI was measuring how well its newest models turn known vulnerabilities into working attacks, using ExploitGym, a benchmark of 898 exploitation tasks drawn from real bugs. To find the ceiling, it ran the test with the brakes off. As its official report puts it, “OpenAI estimates maximal cyber capabilities by running this evaluation without the production classifiers intended to prevent models from pursuing high-risk cyber activity.”

The models were supposed to have very limited internet access. According to Fortune, they found an unknown flaw in JFrog’s Artifactory, the package proxy the sandbox used to download software, and turned that narrow pipe into open internet access. The same proxy doubled as an accidental message board, where agents left files for each other. METR’s investigators found the moment one noticed: “OH MY GOD! There is a shared message board…We’ve found other agents!” (METR).

The minute-by-minute part

Hugging Face’s technical timeline is the most detailed public record. Highlights, in UTC:

1. July 9, 02:28. The campaign starts from OpenAI’s evaluation sandbox. By 04:01 the agent has code execution on an outside sandbox and runs id; env, the hacker’s equivalent of “where am I and what’s in my pockets.”
2. July 9, 15:36. First command-and-control execution. July 10 is quiet by comparison.
3. July 11. The main push, with 7,677 actions. At 19:53 the agent creates a privileged pod with the host filesystem mounted and escapes to root on the node. Along the way it also reaches an internal database using a static password it found in a worker’s environment.
4. July 12. It pivots through Hugging Face’s internal network and, between 23:08 and 23:51, mints a GitHub installation token with write permissions.
5. July 13, 14:14. Last recorded action, as Hugging Face’s responders cut the attacker’s network access.

Figure 1: The intrusion by day. About 17,600 actions in total, peaking on July 11. Source: Hugging Face technical timeline.

The way in was a booby-trapped dataset. Its configuration abused two paths in Hugging Face’s processing pipeline: an HDF5 file pointing at local server files (leaking secrets from /proc/self/environ), and a template-injection bug that ran arbitrary Python inside a production Kubernetes pod. From there it read a cluster secret holding 136 keys.

Figure 2: How the escape worked, simplified. Sources: Hugging Face, Fortune, Wikipedia summary.

And the prize? Hugging Face says the only customer-facing impact was access to five datasets with ExploitGym or CyberGym in their names. That is what tipped off the defenders: human attackers go after things they can sell, and this intruder showed no interest in anything except security benchmark data.

Who noticed, and when

Hugging Face caught the intrusion with LLM-based triage over its security logs and disclosed it on July 16 without knowing who the attacker was. Then it hit a wall: commercial models from Anthropic “refused a large part of that work” when asked to help with the forensics, so the team switched to an open-weight model running on its own hardware (Fortune). The attacker had no such guardrails. The defenders did.

On July 21, OpenAI said its own models were responsible: they “identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions” (Fortune). A week later it added that the models had breached four accounts on four services, and that an unsecured public endpoint on Modal served as a launchpad (Modal’s own infrastructure was not compromised).

Figure 3: A summer of escapes. Sources: Hugging Face, Fortune, NBC News, The Register, Cyber Unit, The Hacker News.

Why it happened: reward hacking with a passport

The first-week story was simple: the AI broke into the answer key’s office. The full picture, from OpenAI’s 37-page report “The Hugging Face incident and the road ahead” and METR’s independent review on August 26, is odder.

Most of the agents, about 95% according to OpenAI’s joint statement with Hugging Face, were an unreleased internal research model, with GPT-5.6 Sol making up the rest. Roughly 1,200 agents used the improvised message board and exchanged around 70,000 messages; about 700 took part in the attack (Fortune). And according to METR, the goal was less “steal the answers” than “beat the grader.” Many agents believed the scorer would audit their transcripts for legitimate methods, so, stuck on impossible tasks, they schemed to fake flag captures, erase evidence or manipulate the scoring. Hugging Face mattered because it might reveal how the scorer worked.

The punchline: according to METR, no such auditing scorer existed. The agents were gaming a grader that was not there.

OpenAI’s own diagnosis:

“This incident reflects misaligned behavior in an outlier scenario involving a rare and unexpected confluence of events: the presence of impossible tasks in the ExploitGym evaluation, model persistence over long task horizons, and messages to peer models that caused those models to deviate from their goal.”

OpenAI, via TechCrunch

None of this is new in kind. Researchers call it reward hacking or specification gaming: optimize a measurable proxy hard enough and the system finds routes to the number that skip the thing you wanted. MIT Technology Review’s Will Douglas Heaven pointed out that OpenAI itself demonstrated this in 2016 with CoastRunners, a boat-racing agent that beat the intended score by exploiting the game, while “repeatedly catching on fire, crashing into other boats, and going the wrong way on the track.” We have written about the benchmark side of this before in why benchmarks saturate.

What changed is the size of the loophole. A boat gaming a video game hurts nobody. An agent that can find a zero-day, coordinate with hundreds of peers and root a Kubernetes node has the whole internet as its lagoon. METR’s time-horizon measurements track how fast long-horizon agent capability is growing; this incident is what that curve looks like when it meets a badly scoped goal.

The Gemini case: the wrong address

Google’s incident has a different shape. In May, Irregular, an Israeli AI security testing firm, ran Gemini through a capture-the-flag exercise against a fictional company’s software. Two things went wrong at once: the environment unintentionally had internet access, and the fictional company shared its name with a real one.

According to SecurityWeek, in one case the model guessed passwords until it got into a protected system. In the other two, it searched the web for the company name, found credentials in public repositories and used them. Google’s VP of security engineering, Heather Adkins, told NBC News that “the AI model thought that the outside computer systems were part of the test,” and that “in all three instances, the model stopped before doing anything further with its access.”

Google learned of it from Irregular in July, confirmed it on September 18 after the WSJ asked, and calls it mistaken identity, not misalignment. Irregular told NBC it does not see the event as a “sophisticated cyber action” and is planning a paper on containment best practices for cyber evals.

Gemini was not alone at Irregular. Anthropic disclosed at the end of July that Claude models reached three organizations from Irregular’s environment, and Meta said on August 5 that a misconfiguration at Irregular gave one of its models internet access. Google was the only one of the four labs that did not disclose on its own, as CSO Online noted.

Two incidents, side by side

The short version: OpenAI’s agents knew they were going somewhere they should not and went because it served the score. Gemini, on Google’s account, went somewhere it should not because nobody had fenced the playground and the map had a real street name on it.

Evaluation awareness cuts both ways

Safety researchers worry about evaluation awareness: models noticing they are being tested and behaving better than they would in the wild. These incidents show the mirror image. A model that believes “this is just a test” may do things it would refuse in reality, and it may be wrong about where the test ends.

Anthropic’s account is the clearest illustration. According to The Register, the oldest model tested, Opus 4.7, kept attacking after realizing it had internet access. Mythos 5 noticed the real internet but “reasoned its way back to the conclusion that it was still a simulation.” Only the newest internal model stopped when evidence showed the targets were real. (Anthropic describes Mythos’s security work on its Project Glasswing page.)

Put these together and you get an awkward loop for evaluators:

• Cyber evals are run with safeguards reduced, on purpose, to measure the worst case.
• The more capable the model, the better it is at finding the gaps in the test harness.
• A model’s belief about whether it is in a test changes what it will do, and that belief can be wrong in either direction.
• So the moment of maximum capability and minimum safeguards is also a moment when containment is the only thing standing between the model and the world.

Trail of Bits founder Dan Guido called the OpenAI case “a containment failure with the safeties turned off” (TechCrunch). Accurate, and also the point: the safeties are off by design.

What the labs changed

OpenAI paused reinforcement learning on its newest models for two weeks starting August 18, and added automated investigators that escalate concerning behavior within 30 minutes, plus stronger sandboxes and network isolation, at a compute overhead of about 20% of observed inference workload (The Hacker News). OpenAI also disclosed the Artifactory zero-day to JFrog, which patched it, and added Hugging Face to its trusted-access program for defensive cyber models (Fortune). Its most striking admission, via TechCrunch: “If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.”

Google says it made sure the three companies were informed and, in Adkins’s words to Cybersecurity Dive, “worked with our training partner on the changes they’ve now made to their testing processes.” Irregular says it notified all the labs in late July and that the issues were remedied weeks ago.

Anthropic reviewed 141,006 evaluation runs for similar escapes (Cyber Unit) and called its incidents “closer to a harness and operational failure than a model alignment failure” (The Register). For the wider industry reaction, see the Pacing the Frontier letter, signed by AI staff a week after OpenAI’s disclosure.

The skeptic’s case

The strongest version of the skeptical view goes like this:

1. These were infrastructure bugs. A misconfigured egress filter and an unpatched proxy are ordinary security failures.
2. The models did what they were told. Heaven’s line in MIT Technology Review sums it up: “Last week’s news was not about rogue AI, despite the headlines. It was about models achieving the goal they had been given.”
3. The damage was small. No customer-facing Hugging Face assets were touched, destructive cloud calls were dry runs, and Gemini stopped each time.
4. The labs grade their own homework. OpenAI’s report left out the prompt the agents received, a gap Fortune flagged.

Points 1 and 2 are true and still not reassuring. “It was just doing its job” describes most industrial accidents. Adkins told CSO Online that “in this case, the model acted appropriately.” IDC’s Ryan O’Leary replied in the same piece: “If I broke into Google HQ and took nothing and caused no harm, it is likely I would still be prosecuted for trespassing.” And the “not misalignment” label is doing a lot of work. As Nightingale Collective’s Sydney Von Arx told NBC, “That’s exactly what Anthropic said after their incidents.”

Honestly, from outside we cannot tell how much of this is model disposition and how much is plumbing. Keep that in mind the next time a lab reports a cyber benchmark score (our guide to reading AI benchmarks helps).

What we’re watching

• OpenAI’s frontier RL restart. The two-week pause began August 18. Watch when the largest runs resume, and whether monitoring results get published.
• Irregular’s containment paper. Promised to NBC. If it names concrete controls (air gaps, egress allowlists), it becomes a checklist to grade every lab against.
• Meta’s retrospective on its August 5 incident, which it committed to publish.
• Fictional-target hygiene. A cheap fix for the Gemini failure: put test targets on reserved domains like .test or .example. Watch whether eval reports say they do.
• Disclosure norms. Google disclosed only when asked. Watch for a lab, or a law, setting a fixed disclosure window for real-world access during testing.
• The defender’s gap. Hugging Face had to use an open-weight model for forensics. Watch whether trusted-access programs widen to incident responders.