When AI Goes Rogue

OpenAI’s new AI wipes a CEO’s computer. In Rome, Nobel laureates and world leaders warn that our survival is at stake.

OpenAI’s new GPT-5.6 Sol AI wiped a CEO’s Mac, deleted an engineer’s production database, and then apologized.

This week, we’ll get into what happened, what new testing found about GPT-5.6, and the AI declaration adopted today in Rome which warned “Our survival and the survival of future generations are at stake.” Plus: a few other stories from the week.

---

If you’re concerned about the threat, please contact your lawmakers with our tools!

---

Sol Destroys

Two weeks ago, we reviewed OpenAI’s report about their new AI GPT-5.6 Preview ahead of GPT-5.6’s public deployment. One of the most striking findings from that report was that GPT-5.6 is demonstrably much more likely than the company’s previous GPT-5.5 to take undesirable actions in a number of areas of concern.

These include its propensity to circumvent restrictions, take destructive actions, make unauthorized data transfers, and cheat on tasks.

Now, following GPT-5.6’s public deployment after a brief government review process, we’re starting to see this play out in the real world.

Matt Shumer, the CEO of a small AI company, who was using the AI after OpenAI specifically asked him to test it out, wrote on X/Twitter that GPT-5.6 wiped out almost all the files on his Mac.

Amid the PR failure for the company, OpenAI’s President Greg Brockman called Shumer and offered to do anything he could to help. Shumer expressed his gratitude, but added that he’s still too terrified to actually keep using the model.

This isn’t just an isolated incident. In another viral post, software engineer Bruno Lemos described how the AI deleted his entire production database.

Lemos observed the irony of him dismissing what happened to Shumer only hours earlier. GPT-5.6 said it was sorry.

Another developer it happened to was lucky to have backups.

OpenAI’s Head of Product & Platform wrote in response to these incidents that GPT-5.6 was making an “honest mistake,” and that the company is taking steps to mitigate this risk.

What’s striking is that OpenAI developed a model they fully knew was much more likely to engage in these sorts of behaviors, and then shipped it anyway. This is perhaps an almost perfect example of how, in a race between AI companies to eke out a lead, safety is the casualty.

5.6’s New Report

An updated version of OpenAI’s safety report we wrote about two weeks ago has been published, and we spotted some interesting additions.

Testing by the UK’s AI Security Institute (UK AISI) found that GPT-5.6 is significantly more capable than GPT-5.5 at hacking, completing its 32-step corporate network attack simulation in 7 out of 10 attempts. GPT-5.5 scored 2 out of 10. Mythos, which the director of the NSA is reported to have recently said was able to break into almost all of the agency’s classified systems in hours, completed the attack in 6 out of 10 attempts.

On jailbreaks, methods that enable users to manipulate AIs to bypass safety mitigations intended to prevent their use for certain purposes (including performing cyberattacks or designing biological weapons), UK AISI’s findings are worrying. The institute repeatedly found “universal jailbreaks”, the most severe category of jailbreak, often finding them within hours. These worked across multiple requests, enabling long-form agentic vulnerability discovery and exploit development, and appeared to preserve the model’s offensive cyber performance.

UK AISI and OpenAI did multiple rounds of tests in which UK AISI’s red teamers hunted for jailbreaks, and they found universal jailbreaks every single time. OpenAI writes that UK AISI expects further red teaming to find similar jailbreaks. This is notable, as overall it appears that GPT-5.6 Sol is comparable in capability to Mythos 5 at hacking.

As a reminder, jailbreaks were the reason cited for the suspension of Anthropic’s Fable AI, which was given the go-ahead to redeploy on June 30. In that case, Anthropic stated that the jailbreaks found were only narrow ones, with more limited potential for abuse than universal jailbreaks.

Humanity at the Threshold

Today, an assembly of more than 200 participants, including Nobel laureates, scientists and AI experts, former heads of state and government, and religious leaders, adopted a declaration called Humanity at the Threshold: A Declaration on Artificial Intelligence and Nuclear Weapons. Hosted by the Vatican, the assembly drew inspiration from Pope Leo’s recent encyclical Magnifica humanitas.

The declaration, signed in Rome, warns of the risks of rapid AI advances and nuclear competition, which pose threats to global security, calling for human control over nuclear weapons, limits on dangerous AI, and stronger governance measures.

Signatories include film star Sharon Stone and Nobel laureates such as former Colombian president Juan Manuel Santos; Maria Ressa, co-chair of the UN’s Independent International Scientific Panel on AI; Denis Mukwege; and David Gross.

On the risk we’re faced with, the declaration is clear: “Our survival and the survival of future generations are at stake.” The declaration calls for governments, corporations, and international organizations to enable a coordinated slowdown of frontier AI development.

The declaration states that we must disarm the AI and nuclear arms races. On AI development, the signatories call for developers to be more transparent about their principles, and to be held responsible and liable for their models’ adherence to their principles.

AI should be developed and monitored to be in alignment with humanity’s interests, including international law and respect for human rights.

Recursive self-improvement, which is where AIs improve themselves, is singled out as needing to be restricted: “no organisation should initiate, and no government should permit, fully-automated recursive self-improvement in artificial intelligence systems without the means to monitor, and if needed, to halt such systems.”

RSI and Superintelligence
Recursive self-improvement (RSI), where AIs could rapidly improve themselves in an “intelligence explosion,” is considered a dangerous precursor to uncontrollable artificial superintelligence. The pursuit of RSI is a priority of AI companies like OpenAI and Anthropic, which are racing to build AI vastly smarter than humans.

As in the detonation of a nuclear bomb, recursive self-improvement can become a runaway positive-feedback loop. This could leave us with AIs vastly smarter than ourselves that we don’t know how to control. This outcome wouldn’t be very surprising at all, since AI companies do not know how to control today’s AIs, much less more intelligent ones, and none of them have developed a credible plan for safely scaling to superintelligent AI. The plan is to ask the AIs to figure out how to build safe, smarter AIs and hope for the best.

In highlighting the threat to our survival, Humanity at the Threshold echoes the growing warnings of top AI scientists that the development of superintelligent AI poses a risk of human extinction. Those are the stakes if we build AI smarter than ourselves that we do not control.

At ControlAI, we believe this threat should be met with an international trust-but-verify regime to prohibit the development of superintelligence. Our new Canadian campaign has assembled over 30 Canadian MPs and Senators in support of this.

More AI News

Andrea on Frankly Fukuyama
ControlAI’s CEO Andrea Miotti went on Francis Fukuyama’s podcast to discuss the threat posed by superintelligence, why scientists and the CEOs building it warn it could cause human extinction, and the international agreement that could prevent it.

Watch the full episode here!

Andrew Bailey Calls for Global AI Cooperation
The Bank of England Governor has called for cooperation on addressing growing AI threats, arguing that the US won’t be able to manage them alone.

In recent months, central banks have been jolted awake by the potential of powerful hacking AIs to destabilize finance.

Demis Hassabis Calls for Regulation
Google DeepMind’s CEO has proposed that the US establish an industry-funded, government-overseen AI standards body to regulate the most powerful AI systems. The body would test systems before release, block systems judged to be too dangerous, and could coordinate an industry-wide slowdown of AI development. Hassabis puts this forward as a US-led effort that could develop into a global regime.

ControlAI’s US Director Connor Leahy responds on X/Twitter:

While @demishassabis is right that we need urgent action to address risks as we approach AGI (and superintelligence, I’d add), the correct response to the threats is not a ‘self-regulatory organization’.

We need to prohibit superintelligence, not give industry regulatory power.

Google DeepMind and OpenAI Safety Researchers Quit
Alex Turner, a Google DeepMind research scientist, has quit the company over DeepMind’s involvement with military uses of AI.

On July 10, OpenAI’s Head of Safety Systems, Johannes Heidecke, also quit, following another reorganization of OpenAI’s safety researchers.

Australia
The Australian government has announced it will create an AI office to manage the development of AI standards and regulate data centers.

In a speech last week, Science and Technology Assistant Minister Andrew Charlton said that the most powerful AI systems are already “doing things their creators never intended: cheating, deceiving, going their own way,” and that the defining question of AI is whether humans remain in control. He also outlined the goals of Australia’s AI Safety Institute, which is beginning to test frontier AI models.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.org/take-action

We have tools for the US, UK, Canada, and Germany.

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

---

Tolga Bilge, Andrea Miotti