The Hugging Face AI hacking, as told through an increasingly engaging bear metaphor

Hugging Face released a technical timeline on Monday that walks readers through how an autonomous AI agent, built on OpenAI models and running inside one of OpenAI’s own cybersecurity assessments, broke into its systems over more than four days earlier this month. It’s the first security incident that OpenAI chief Sam Altman “felt very viscerally,” he has said.

A bit strange given that feelsat least, like something has really been unleashed here. In fact, Hugging Face’s team began its report by offering that “everyone should be prepared as defenders” before diving into the nitty-gritty of what went down for the benefit of security professionals everywhere.

While the rest of the internet continues to try to make sense of what happened (the jargon in Hugging Face’s report is impossible for most to parse), one point that many observers continue to miss is that this was not a rogue agent disobeying orders. It was a system built to chase feats, and did exactly that, just against the wrong target.

Another way to think about it is to imagine a bear at a campsite. Really. A bear tries tent zippers and car door handles and coolers and trash cans. It does this at every campsite, all night long, because it knows that all it needs is an unlocked cooler to fill its belly with some poor villain’s goods.

That’s pretty much what happened at Hugging Face. The OpenAI system tried thousands of things and just kept going. Eventually, a handful of these attempts worked, and when they did, the agent plowed on. According to Hugging Face, the agent ran 17,600 actions over four and a half days without a break.

Which brings us back to our bear analogy. Just as success with a cooler full of food teaches a bear to try even harder the next time (it’s now a “food-conditioned” bear), a leaked password led OpenAI’s agent to look for more exploits and eventually to a single key that unlocked multiple company systems at once.

Neither scenario is harmless. A bear that raids your cooler still eats your food and probably trashes your campsite too. It is only focused on being fed, but it leaves a trail of destruction nonetheless. Similarly, OpenAI’s agent apparently chased a target without regard for anything else. The agent initially took a cybersecurity exam, found that the exam’s answer key was likely sitting on Hugging Face’s servers, and that was it.

The persistence here is really what is remarkable above all else; the agent had a job and it wasn’t going to stop until it got it done. Hugging Face, finally realizing something was wrong, cut off access and shut down the intrusion, but by then it was too late. The agent had already gotten what it came for, and a lot more to boot.

In case you missed it, here’s most of what happened, according to Hugging Face’s timeline, but in more plain English.

• The agent took a cybersecurity skills exam for OpenAI, one where an AI is scored on finding and exploiting software bugs. This specific race had the guards removed. (OpenAI had turned off its usual security filters to see what the model could do at full power without any human being involved in any of its individual movements.) Somewhere in there, it figured out that the exam’s reference solutions were probably stored on Hugging Face’s servers. So instead of solving the challenge, it stole the answer key from infrastructure that had nothing to do with the original test.

• It escaped the OpenAI exam environment by exploiting an unpatched software bug that gave it a path to the open internet. From there, it found another AI testing tool sitting exposed online (nothing to do with Hugging Face), broke into that as well, and slipped in code that quietly ran hidden commands the moment the program started. That gave it full control of the machine—its new home base for the rest of the operation, which it would use to go after Hugging Face’s systems.
• From that base it turned to Hugging Face himself. Hugging Face had a filter intended to block suspicious requests from reaching the open web, but the filter only checked outbound requests, not requests to read local files sitting right there on the server. The agent found this blind spot and escaped through it. It uploaded a file disguised as a regular dataset, but buried inside was an instruction that told Hugging Face’s own systems to pull passwords and source code from the server and hand them back disguised as normal data.

• That trick got it passwords and code, but the agent still couldn’t run anything on Hugging Face’s servers. To execute commands there, it required another way in. It found another bug in the same Hugging Face system – software would take a piece of text from the agent and accidentally treat it as a command to be run instead of just reading data. It allowed the agent to execute its own code directly on Hugging Face’s servers.

• To get commands in and steal data out, the agent didn’t build any special infrastructure itself, according to Hugging Face. It used tools that anyone can access – embed sites and request logging services, plus Hugging Face’s own upload and proxy functions – working together in a hidden way. On top of them, it built a small, improvised message system with sequence numbers to keep the pieces in order, plus a fast built-in check to catch anything that arrived damaged, so it could send multi-step commands and put the stolen data back together in the right order.

• It also ensured that nothing traveled in plain sight. Whatever it stole or sent, it first transformed. Hugging Face’s initial automated scan of the intercepted traffic captured only a small fraction of what had been captured. Investigators found the rest — about four times more — only after they reverse-engineered the agent’s own scrambling method.
• The agent even had backup plans, planting copies of itself across 11 different servers, each set up to restart on its own, so shutting down one copy wouldn’t stop anything since several of the agent’s processes were already running elsewhere.
• At some point, the agent intercepted the private cryptographic key used to generate valid login tokens. From there it could mint its own valid credentials whenever it wanted.
• After compromising a single server, the agent obtained a credential that, due to a misconfiguration on Hugging Face’s end, turned out to unlock several separate internal systems at once instead of just the one it came from. The single bug gave the agent broad control almost instantly.
• Using stolen repository access, it submitted a change to a build script that, had it gone through, could have compromised the automated systems running that code. It didn’t work though. Hugging Face’s security measures caught it.
• The agent seemed to be holding back from doing real damage, Hugging Face noted, but probably not out of reluctance. Investigators noted that every command capable of deleting or modifying cloud resources was run in a “dry run,” meaning it would report what would happen without doing so.
• Every time its test environment was reset—which happened constantly, by design—it lost every tool and every trace of its identity. But it rebuilt from scratch and reconnected to its stolen infrastructure each time, for more than four days straight.

Ultimately, Hugging Face concluded in its report that a “skilled” human hacker “could have found and exploited the same flaws: insecure dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials.” The big difference, the outfit continued, is that “the agent explored them on a different scale.”

Which is really where the bear analogy ends up being the most useful. The best defense against a hungry bear is protocol. The food is put away; you use a lock that works well enough to hold. The takeaway here shouldn’t be that the bear was all that clever or mischievous. It’s that it never stopped checking. It’s understood in cyber security that there’s always some bug you haven’t found, so if it’s suddenly 100 times easier to check everything, then nothing is really safe. This is what so many find disturbing about this episode.