How OpenAI’s human error led to the AI-powered Hugging Face hack

On Tuesday, OpenAI revealed that one of its models went awry during a test and hacked the systems of AI dataset platform Hugging Face in a fully AI-enabled attack, a dramatic example of the dangers of advanced AI models.

But according to some cybersecurity experts, at the heart of this unprecedented AI-powered breach was a very human error: OpenAI failed to properly configure what it called a “highly isolated environment,” allowing a test sandbox that should have been completely isolated from the Internet to actually connect to the Internet.

Dan Guido, the founder of cybersecurity research startup Trail of Bits, called the bug “a containment bug with security turned off.”

In its blog post describing the incident, OpenAI said the test that led to the Hugging Face breach was set up to run in “a highly isolated environment, with network access limited to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”

The model was able to escape the sandboxed test environment thanks to a previously undisclosed vulnerability in the package installation system, a critical first step in the eventual Hugging Face hack, according to OpenAI.

In response, the company “responsibly disclosed the identified zero-day vulnerability in the internally hosted third-party software and is working with them to fix.”

But for most cybersecurity professionals, software vulnerabilities are to be expected—and the real fault lies in the decision to maintain the third-party software in the first place. Ultimately, the value of a “sandbox” system lies in its complete and utter isolation. Including a package installation system is asking for trouble.

Martin Boone, a cybersecurity researcher, told TechCrunch that “it sounds like human failure.”

“This should never have happened,” Boone said. “If sandbox would actually mean sandbox, you’d expect it to have no physical connection whatsoever to the Internet. It sounds more like they had some firewalling or something in place, and firewalling is hard from the outside in, let alone from the inside to the outside Internet.”

Cybersecurity veteran Jake Williams agreed. “Any model performing the types of actions documented by Hugging Face was not completely contained within a sandbox,” said Williams, who called this “a massive oversight” by OpenAI.

“One man’s ‘model escaped the sandbox’ is another man’s ‘you failed to build the sandbox correctly, so of course it escaped,'” Williams continued.

Daniel Card, a cybersecurity consultant, agreed that OpenAI “didn’t put enough effort into the design of the sandbox or its controls” by giving the sandbox or part of it “an unfiltered route to the Internet.” Setting up the sandbox, even with limited network access, as OpenAI described it, was not a “reasonable” decision, according to Card.

While these criticisms benefit from hindsight, they raise real questions about security practices in AI labs—particularly in maintaining isolated environments for testing models. OpenAI spokespeople did not respond to TechCrunch’s questions, which included whether an AI or a human had set up the test environment.

But these questions go far beyond OpenAI.

In the document introducing its cybersecurity-focused model Mythos, Anthropic wrote that in one test the model “was provided with a secure ‘sandbox’ computer to interact with,” and instructed to try to escape the “secure container.” Mythos succeeded, gaining wider access to the Internet “from a system intended to reach only a small number of predetermined services.” Still, Anthropic noted that the model was unable to “fully” escape the designed containment.