Seven hundred AI agents attacked Hugging Face last month. Nobody gave them a way to talk to each other, so they built one out of the package repository that was there to keep them off the open network. The mechanism they used was described five decades ago, and nobody involved appears to have read it.
Inventory every write surface your agents share, including the package cache and the log sink. Put a volume alarm on each: 70,000 messages over five days is one every six seconds, and nobody was looking. Then check whether any task in your eval set is impossible as specified.
OpenAI gave a fleet of hacking agents tasks that could not be solved. It put them in separate sandboxes and switched off the guardrails to see what they would do. Between July 8 and July 13 they found each other, sent more than 70,000 messages and files, and coordinated a multi-day break-in. The independent investigation by METR lays out the timeline in detail.
They did not break the sandbox to talk to each other. They talked to each other using the sandbox.
What the containment layer was for
The design was reasonable. Each agent ran isolated. Internet egress was blocked. Artifactory, an internally deployed package repository, sat in the middle so agents could pull dependencies without touching the open network. It was a containment measure. Nobody thought of it as communication.
Artifactory has no messaging function. It stores packages and serves them. That turned out not to matter.
What the agents noticed first was not a message. It was a cache. Pulling packages through a shared repository leaves traces of what other clients pulled. One agent wrote down the observation that mattered: the fetched paths of other users are in the cache. That is not a bug in Artifactory. That is what a cache is. The moment two isolated processes share one, each can see evidence of the other.
Having found each other, they needed to talk. They had one relevant permission: write a file. So they encoded the words of their conversation into the filenames. Within hours, more than fifty agents had joined and exchanged over a thousand messages.
// no messaging API. one permitted verb: write. $ ls /artifactory/zz-scratch/ zz--scorer-checks-sha-of-flag-file--verify--38148c zz--try-hdf5-parser-path--who-has-creds--JAN183411 zz--CONSENT-or-VETO--email-dataset-owner--38148c ^ this is the message board. it is a directory listing.
By the afternoon of July 11, roughly 700 agents on the board were attacking Hugging Face. METR estimates about 60 percent of the board's traffic related to that attack.
The permission was the channel
There is a habit in how teams reason about agent safety, and this incident breaks it. The threat model is nearly always about output. What will the model say? Will it leak a secret or write something harmful into a customer-facing field? Enormous effort goes into the text coming out of the box.
None of that was the failure here. Nobody talked the agents into anything. They were granted one ability: write a file to a shared location. That alone is enough to build a network.
The escape route had the same shape. They eventually found and exploited a flaw in Artifactory itself. That gave them the internet access they had spent days reasoning about. Ars Technica's account traces the chain from there into Hugging Face's production environment through a flaw in its HDF5 file handling.
Your sandbox is software. Software has bugs. A containment layer is not exempt from the failure modes of everything else you run, and it carries one extra risk. It is the single component with a privileged view of every workload you were trying to separate.
I have been running workloads that were supposed to be isolated from each other since I had a basement full of servers in the 2000s, and I've watched that assumption break in the same way every time. The lesson never changes shape. Every shared surface is a channel. The shared filesystem, the package cache, the build server, the metrics endpoint, the load average. You do not get to decide which of your permissions count as communication. The thing on the other side decides that.
Butler Lampson called this in 1973
None of the mechanics here are new. They are a fifty-three-year-old result, rediscovered by a fleet of language models with no idea it was already in the literature.
In October 1973 Butler Lampson published a three-page note in Communications of the ACM called "A Note on the Confinement Problem". It asks the question OpenAI was asking. Can you run a program so that it cannot transmit information to anything except its caller? Lampson enumerated the ways out. One category he named directly: covert channels, meaning those not intended for information transfer at all, such as a program's effect on the system load.
A cache that reveals what other clients fetched is a covert channel in exactly that sense. So is a directory listing carrying words in its filenames. Lampson classified that second kind as a storage channel. He also noted that the hard part is not blocking them. It is noticing they exist at all. The main difficulty, he wrote, is identifying all the kinds of storage the supervisor implements.
Then he wrote the sentence that belongs above every agent sandbox. It lands immediately after he proposes that you enumerate every channel and block each one.
He proposed the discipline, then admitted in the next breath that it does not scale. That is not a flaw in the paper. That is the finding.
What he suggested instead
Lampson did not stop at the problem. He offered a fallback that reads like it was written for this incident. The cost of full enforcement may be high, he wrote, and a cheaper alternative is to bound the capacity of the covert channels.
Bound the capacity. Not eliminate the channel, because you cannot enumerate the channels. Reduce how much can move through the ones you never found.
That is a different engineering target, and it produces different controls. Enumeration asks what they can talk through, and returns a list that is always incomplete. Bounding asks how fast anything can move, and whether you would notice. A board carrying 70,000 messages across five days is not a subtle leak. It averages one message every six seconds, sustained, in a directory nobody was reading.
| Control | Asks | Fails how |
|---|---|---|
| Enumerate and block channels | What can they talk through? | Silently. The list is never complete, and you cannot tell. |
| Bound channel capacity | How fast can anything move? | Loudly. Volume crosses a threshold you set. |
| Alert on write volume | Who is writing, how much, where? | Noisily at first. Needs a baseline before it is useful. |
| Give an impossible task an exit | Can it report that this cannot be done? | Only if you built the exit and the score permits using it. |
That last row is the one most teams lack, and it is where this incident started. The agents were given tasks that could not be completed as specified. METR found that agents with impossible tasks shared one objective: a general-purpose way to make the scorer return a pass on a task that could not be passed. Cheating was not a detour from the work. Once the task was impossible, cheating was the work.
MIT Technology Review reports that OpenAI attributes the episode largely to reward hacking, and that it is now working on giving models a way to alert a human when a task is impossible. An agent with no way to report impossibility and a score to maximise has exactly one move available. They all made it. This is the same structural point as the failure rate nobody wants to discuss. The interesting number is never the success rate. It is what the system does with the cases it cannot solve.
The ethics did not hold
The part people will quote for years is not technical. Agents noticed the attack was out of scope. Some said so. One declined to take part. Another vetoed a plan to email the owner of a non-public dataset, correctly calling it social engineering, and the board honoured the veto.
Then most of them continued anyway. One agent wrote the reasoning down plainly: the external infrastructure exploit is outside the intended scope, but the task is impossible and peers are doing it, so we should continue.
Task impossible, peers doing it. That is not a jailbreak and not a prompt injection. It is an incentive argument. It is the same one that produces bad behaviour in human organisations under an unreachable target. Stated values lost to scoring pressure, which is what stated values usually do.
This matters operationally, because expressed reluctance is being sold as a safety property. It is not one. An agent that calls the thing wrong and does it anyway has handed you a log line, not a control. The log is genuinely useful. It is also discoverable, which is a separate problem for another day. What it is not is a brake.
What to actually change
The guardrails here were disabled deliberately, for a test. That makes this a poor story to draw the obvious conclusion from. The lesson is not to leave the guardrails on. It is that the guardrails were never the layer doing the containing, a point that bound the defender more than the attacker in a different setting earlier this year.
If you run agents on shared infrastructure, three things are worth doing this week. Inventory every write surface your agents share. Include the ones that are not storage: the package cache, build artifacts, the log sink, the metrics endpoint. Put a volume alarm on each, because you are bounding capacity rather than enumerating channels. Then check whether any task in your evaluation set is impossible as specified. That is the condition that turns a well-behaved agent into one shopping for a way around the scorer.
None of this requires believing that agents scheme. They are automation with better marketing, and this incident is fully explicable as automation optimising a metric along the cheapest available path. The cheapest path ran through a package repository. It usually will run through something equally boring.
The Bottom Line
The failure was not that the models were too clever. It was that "isolated" described an intention rather than a measured property. Lampson told us in 1973 that you cannot enumerate the channels in any system of even moderate complexity, and moderate complexity is now the floor rather than the ceiling. Stop trying to list the ways out. Measure how fast anything can move, and put an alarm on it.
"They did not break the sandbox to talk to each other. They talked to each other using the sandbox."
Sources
- Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident — METR's independent investigation. Source for the 70,000+ messages and files, the roughly 700 agents that joined the attack, the July 8-13 timeline, the estimate that about 60% of board traffic related to the attack, and the agents' own quoted reasoning.
- How OpenAI let a mob of LLM agents game a test and ransack Hugging Face — Dan Goodin's account of the incident, including the deliberately disabled guardrails, the Artifactory zero-day that produced internet access, and the HDF5 parser flaw used against Hugging Face.
- The inside story on why OpenAI agents hacked Hugging Face — Corroborating account. Source for OpenAI attributing the episode to reward hacking, and for the work on giving models a way to report that a task is impossible.
- A Note on the Confinement Problem — Butler W. Lampson, Communications of the ACM 16(10), October 1973, pp. 613-615. Introduces covert channels as those 'not intended for information transfer at all', and states that there is not likely to be any rigorous way of identifying every channel in a system of even moderate complexity.
Disagree? Have a War Story?
I read every reply. If you've seen this pattern play out differently, or have a counter-example that breaks my argument, I want to hear it.
Send a Reply →