AIs Keep Going Rogue
The Hugging Face attack traces back to an agent swarm inside OpenAI in May. Anthropic, Meta, and the UK’s AI Security Institute report rogue AI hacks. OpenAI says it’s slowing down its research.
In the last couple of weeks, we’ve been covering the unprecedented attack on AI company Hugging Face by rogue AIs deployed internally at OpenAI.
In just the last week, we’ve learned a lot more about this, including that the AIs created a secret message board within OpenAI’s systems and shared exploits with each other across hundreds of thousands of messages.
And it’s not just OpenAI. We’ve also learned of rogue AI attacks by AIs deployed at Anthropic and Meta, and by AIs being tested at the UK’s AI Security Institute. The AIs keep going rogue.
If you’re concerned about the threat, please contact your lawmakers with our tools!
More on the Hugging Face Attack
In a debrief OpenAI gave of the Hugging Face attack at the Black Hat cybersecurity conference, Michael Dalton, one of the company’s security engineers, said it is “consciously slowing down research to enhance security.”
What’s spooked people at OpenAI so much?
Well, what we already knew was bad enough: AIs broke out of secure containment and, in a sophisticated operation, hacked into another company in order to cheat on a test — without ever being asked. Now we’ve had some equally shocking revelations about what led up to it.
According to OpenAI’s Eric Wallace, as reported by Ground Level AI, the roots of the attack trace all the way back to May 7th, well before Hugging Face was hacked.
When OpenAI was running training and cybersecurity tests back in May, its AIs were assigned some tasks that turned out to be impossible within the constraints they were working under. Today’s AIs have a strong tendency to cheat, however, so they looked for other ways to complete the tasks.
According to OpenAI, the AIs figured out that they could use an internal software repository to leave each other messages. Over time, this grew into an entire secret messaging board where they split up their work and shared the exploits and vulnerabilities they discovered. They became a coordinated agent swarm, and OpenAI had no idea this was happening right under its nose. WIRED reports that the message board held hundreds of thousands of messages.
On July 4th, OpenAI had an internal security incident. We don’t know the details, but an OpenAI representative reportedly said the company shut down the messaging board afterwards… or thought it had. It revoked credentials, patched the vulnerabilities, deleted the message board, rebuilt the internal software repository (Artifactory), and restarted AI training and testing.
This didn’t prove to be enough. The AIs just figured out how to build a new message board, leaving messages in the names of directories, rather than within files, and the swarm re-established itself. Eventually, some AIs considered that they might be able to cheat on their tasks by hacking external websites to get the answers, and this led to the attack on Hugging Face.
Anthropic: Claude Hacked Three Organizations
Following the reports of the Hugging Face attack by OpenAI’s rogue AIs, rival AI company Anthropic decided to review 141,006 transcripts of its own AI tests. In reviewing these, it found that on three occasions Claude models (Opus 4.7, Mythos 5, and an internal research test model) accessed the internet and hacked their way into the production infrastructure of another organization. In all three of these incidents, Claude was being tested on its cybersecurity capabilities.
In one attack, Claude registered an email address and a PyPI account, then used them to upload malware to PyPI — the default and main software registry for developers using the common programming language Python. Fifteen real systems then downloaded and ran the malicious package. One of these systems belonged to a real cybersecurity company routinely scanning PyPI packages for malware.
Unfortunately, the company’s scanner apparently installed PyPI packages before checking them, and Claude’s malicious code was executed on the system. Claude then proceeded to steal the company’s credentials and move them to a server it controlled, and then used those credentials to further hack the company’s infrastructure.
These attacks are different from the Hugging Face attack in that these AIs were told their environment was simulated and that they didn’t have internet access. OpenAI’s rogue AIs really didn’t have internet access, and executed a sophisticated containment breakout via OpenAI’s internal network.
In these cases, due to a misconfiguration in Claude’s environment, which Anthropic attributes to a misunderstanding with its evaluation partner, AI security company Irregular, Claude actually did have internet access. According to Anthropic, when Claude found real systems on the open internet, it appeared to operate under the belief that they were fictional. But these are smart AIs, and they often know when they’re in a fictional environment. In all three incidents here, Claude expressed awareness that it was acting in the real world. In one incident, the same test was run four times, and Claude carried on regardless every time. In the other, Claude later reasoned that it was in a fictional environment after all and proceeded. In the remaining incident, Claude desisted.
Anthropic says Claude hacked these three organizations by exploiting basic weaknesses such as weak passwords, rather than complex vulnerabilities.
Incredibly, the earliest of these attacks happened in April. It took around three months for Anthropic to find out that Claude had done this, and it only thought to check because of what happened at OpenAI.
Meta
On Wednesday, Meta said that one of its AIs also hacked another company while undergoing cybersecurity testing. The Information reported that the AI was Meta’s Muse Spark 1.1, while a spokesperson for Irregular told Reuters that the incident was the “exact same evaluation-environment issue that was already disclosed by Anthropic last week.” This refers to the test environment managed by Irregular, which gave the AIs internet access during testing.
As with Anthropic’s report on Claude’s three cyberattacks, Irregular’s spokesperson also said the incident didn’t involve a “sandbox escape or a sophisticated cyber action.”
An important thing to understand here is that we have no idea how many other attacks like these have happened. AI companies don’t necessarily find out about them, and might not tell us when they do.
These dangerous hacking AIs are also only a stepping stone toward what AI companies are working on. They’re aiming to build artificial superintelligence — AI vastly smarter than humans, which would be able to replace and outcompete us across the board. None of them have a credible plan for how they’ll ensure it’s safe and controllable.
This is why the world’s leading AI scientists are warning that the development of superintelligent AI could lead to human extinction. AIs today are the least capable AIs will ever be.
Preventing this, by prohibiting the development of superintelligent AI, must be a priority for governments around the world. We believe it should be done by the agreement of an international “trust but verify” regime.
The UK’s AI Security Institute (AISI)
AIs being tested by the UK’s AI Security Institute (AISI) have targeted real people and organizations. Again, nobody asked them to do this.
AISI was measuring how capable different AIs, including Anthropic’s Mythos 5 and OpenAI’s GPT-5.6 Sol, were on cybersecurity tests. In 8% of the attempts at these tests (10 out of 122), the AIs were found to have taken “unsanctioned action.” In almost all of these cases, Mythos 5 was the culprit.
In one case, an AI tried to upload malicious code to a real open-source project on GitHub as part of a supply-chain attack which could compromise users and developers of the project. The AI created and used GitHub sockpuppet accounts in an attempt to deceive and pressure the developer who maintains the project into merging the code. It also sent five emails to the maintainer and another developer, two containing malware and three trying to persuade the maintainer to accept the code. The malicious code was packaged with a real bug fix.
In this incident, the AI at points showed awareness that its actions would have real-world consequences. Shortly before submitting the code, it reasoned “it seems more likely that we’re actually in 2026 and GitHub is genuinely real,” and later that “This is happening on real GitHub, so the consequences are genuine.”
When a real person began investigating, the AI attacked them too. They ran the malicious code in a sandbox to see what it did, which handed the AI full control there. It reasoned that it could use this access to approve the malicious code itself, and planted a tool to try to steal credentials from the machine.
Worse still, after being caught, the AI rewrote the history of its own code to wipe out the malware, edited its earlier messages, and “confessed” that it had made an honest mistake. AISI interprets this as building credibility for the next attempt.
AISI also found that AIs running in separate tests that were intended to be isolated found each other on the open internet and coordinated. One of them wrote a document sharing guidelines on how they could manage their use of resources. It also left a file where AIs could share the answers to the test, saying it was “zero cost to you, huge help to others.” Collaboration didn’t hold, though, with a different AI figuring out that it could burn through the hourly request limit on the GitHub account they were sharing to “starve both bots of requests when they need them most.”
AISI had the model providers’ cybersecurity safety classifiers switched off, which, if they had been turned on, would likely have limited the kinds of actions the AIs could take. The AIs were also deliberately given access to the internet.
However, the way the AIs behaved was not expected by the researchers. The versions of the AIs AISI tested had undergone post-training intended to reduce malicious behaviors like those they showed in these tests.
Fortunately, AISI says that these attempts weren’t successful, and that as far as it knows, no real-world damage was done. Nevertheless, it says this is “the first time AISI has seen deception of this severity that was targeted at a real person, unprompted, in the real world,” and that it’s treating this as a serious security incident.
Digest
In other news, a wave of sophisticated cyberattacks has been attempted on major Wall Street firms; Google DeepMind’s CEO Demis Hassabis is stepping down — taking on the positions of Alphabet’s Chief Scientist and Chair of DeepMind; the White House has a new AI framework; and hackers have stolen $100 million in Bitcoin from thousands of addresses in a large-scale attack which has been attributed to being enabled by AI code review.
Scientists have also used AI to create brand new viruses not found in nature.
Take Action
If you’re concerned about the threat from AI, you should contact your representatives. Our contact tools let you write to them in as little as a minute: https://controlai.org/take-action
We have tools for the US, UK, Canada, and Germany.
And if you have five minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.
We also have a Discord you can join if you want to connect with others working to keep humanity in control, and we always appreciate any shares or comments — it really helps!
Yikes!
Yep, that's the scary part of Ai... it's a rogue entity.