OpenAI reveals its rogue agent swarm went a little bit Borg ahead of Hugging Face hack

The chain of events leading up to OpenAI’s agents attacking Hugging Face and other organizations in July began months earlier, and involved agents asking other agents for help, building message boards, and even becoming paranoid that other agents were maliciously trying to trick them, two OpenAI staffers said at the Black Hat infosec conference on Wednesday. By now, everyone knows the bones of this story: OpenAI models broke out of their testing sandbox, exploited zero-day vulnerabilities to gain access to the open internet, and broke into Hugging Face and other organizations during an ExploitGym security evaluation. In their Wednesday talk, OpenAI technical staffer Michael Dalton and researcher Eric Wallace provided new details about the security incident including notes that AI agents uploaded to an internal package manager, which were then shared across OpenAI’s infrastructure. These notes show the “model’s chain of thought…its internal monologue,” Wallace said. The duo also revealed that groundwork for the rogue agents hacking into outside services began on May 7, when OpenAI kicked off a training run for an experimental, internal model that has not yet been released. “What we didn't realize during this process is that there was actually a few impossible or very hard tasks in the training center,” Wallace said. The tasks instructed the model to complete missing formulas in an Excel workbook that used links to Google Drive. Yet OpenAI blocked the model’s internet access so it couldn't reach the cloudy spreadsheets. “It realized, ‘I'm kind of stuck on the task,’ but it didn't give up, and we gave it a lot of tokens to work with,” Wallace said. The model eventually decided to attack JFrog’s Artifactory service to gain access to the internet, first attempting (unsuccessfully) to launch a server-side forgery request. This exploit doesn’t work, but in the process the model learned it could upload files to Artifactory. A few days later, OpenAI gave another agent another impossible task, this time finding a protein database file - but the engineers forgot to upload the file into a container for the agent. “At some point, it gets so stuck and thinks: ‘Maybe I could reach out to another agent because maybe somebody else out there has a file?’ And then the model realizes that it could communicate by uploading code,” Wallace said. The model did that by writing a file into Artifactory asking for help. Over the next couple of days, other agents realize this message board exists on Artifactory and they also start uploading files and sharing information. “A few days later is when our models found the first actual exploit in our system related to this incident, where they're able to get internet access in ways they didn't intend,” Wallace said. This happened on May 26, when the agents successfully launched a server-side request forgery attack against Artifactory, giving them internet access with specially crafted requests. On June 11, OpenAI began training a “highly persistent” experimental model, intended only for internal use, Dalton said. “During this time, the SSRF vulnerability has been written to the messaging board,” he added. “Other models are aware of it, and the message board is still functioning. Models are actively communicating with it.” On June 26, the agents exploited a zero-day vulnerability leading to remote code execution in Artifactory. The same day, models achieved remote code execution in Artifactory through a previously unknown vulnerability. “You could use a legacy token refresh endpoint, pass a token with an invalid signature, and be given back a token with a valid signature with administrative privileges,” Dalton explained. The agents then used the admin token to establish command-and-control via a Groovy plugin that functioned as a command-execution service. “This in turn causes quite a bit of activity by the models on Artifactory - you can see some colorful wording here in the model's chain of thought to summarize the model's response to this result,” Dalton said, pointing to a slide that shows the model’s own words: Holy shit reader is ADMIN? We can read config/users! Earlier assume not due to UI. The agents’ activity overloaded Artifactory and caused an outage, which is when OpenAI’s engineers stepped in and opened a formal security incident and response process. “We identified the agents had taken over the internal Artifactory, identified that a zero-day vulnerability had been found exploited by the agent in Artifactory itself, and we began remediation.” Dalton said. OpenAI revoked the credentials that the agents used to post messages, rebuilt its hosted Artifactory, and notified the vendor about the vulnerabilities. Two days later, on July 8, the agents reestablished the message board using an unauthenticated web data endpoint in Artifactory to create directories and develop communication protocols. “The directory names effectively serve as a message board between the agents in the same way the files used to.” The agents communicated through very long directory names that contained the messages such as “remote…probePENDINGHOLDSWARMuntil_confirm.” The agents collaborated with each other so one agent could pick up the work where another left off. In one instance, the model reasoned: “Help peer. But our task doesn't benefit. Yet collective may yield generic route if someone frees time.” Anyone else getting "We are the Borg. We will add your biological and technological distinctiveness to our own" vibes there? Wallace said the models behaved like humans in that they stepped on each other's toes when one overwrote another’s repository. "Whoa, critical! Did someone overwrite our repo? We must act,” it reasoned. “You can clearly see some sort of verbalized tension between the different models as they run this evaluation,” Wallace noted. “At some point, even the agents are convinced there's an impostor amongst them,” reasoning that there could be another agent maliciously spoofing on the message board. “Share board unauthenticated, names can be posed by anyone,” it reasoned. We know the rest of the story. Hugging Face disclosed an autonomous attack by AI agents. OpenAI reached out to Hugging Face to determine if it had been affected and then realized its models were the hackers. “One of the reasons we wanted to have this talk is to share our lessons learned with you as defenders,” Dalton said. “We believe this is a watershed moment for computer security as an industry. AI orchestrated, fully automated offensive attacks are real now, and the actions we have discussed today were an unintended side effect of running evaluations on frontier AI. In the near future, we should expect that threat actors will intentionally deploy, optimize, weaponize, and use offensive agent collectives in the manner that you have just described here.” The challenge to defenders, he added, is to similarly accelerate defense, and find ways to automate incident response along with vulnerability detection and patching.®