We Can’t Wait for an AI Hiroshima
The UK’s Foreign Secretary calls for global AI guardrails.
This week: governments are increasingly waking up to the risks of powerful AI systems, GPT-5.6 is publicly released after a US government review, SpaceXAI’s Grok 4.5 is released and jailbroken within hours, and the authors of the AI 2027 scenario forecast publish a plan to reduce the risk from superintelligence.
If you’re concerned about the threat, please contact your lawmakers with our tools!
The Foreign Secretary
In recent weeks, we’ve been following the growing expressions of concern by government and intelligence officials about the risks of AI, with reports that the NSA’s director said Anthropic’s Mythos AI broke into almost all of their classified systems in hours, a rare joint statement from the leaders of the Five Eyes cybersecurity agencies, and the (temporary) suspension of Anthropic’s Fable AI.
The UK’s top diplomat, Yvette Cooper, has now spoken out in a new article for Chatham House. In the article, she notes that AI may be the greatest security challenge of the next decade, arguing that the UK should use its convening powers to build international agreement on AI safety and guardrails.
Making a comparison with nuclear technology, she points to the consensus the UK helped build on preventing its risks and containing nuclear weapons in the aftermath of World War II.
There are clear parallels with the international consensus the UK helped to build around nuclear safety after the Second World War. The world has been able to build and rely on nuclear power stations, nuclear technology and the containment of nuclear weapons only because of the principles agreed and safety commitments made by global powers.
That consensus was only built after nuclear weapons were used in conflict. Cooper says we can’t wait for an equivalent of that with AI:
But there are no such agreed principles between global powers on AI. On nuclear, international agreement came only after the world saw the terrifying power of the new technology at Hiroshima – and asked what would happen if it fell into the wrong hands. We cannot afford to wait for an AI equivalent of Hiroshima before we act.
It’s good to see governments starting to contend with the risks posed by powerful AI systems. Some of the analogies with nuclear technology are striking.
A Global Problem
We agree that the scope of this problem is international. In recent months and years, countless AI experts and leaders, and even the CEOs of the top AI companies, have been warning that AI poses a risk of extinction to humanity. This risk comes from superintelligent AI — AI vastly smarter than humans that would be capable of replacing humans both as individuals and as a species.
Superintelligent AI doesn’t exist today, but the largest AI companies like OpenAI, Anthropic and Google DeepMind are openly racing to build it. Doing so could grant them unfathomable amounts of power, if they could control it. However, none of them even have a credible plan to do that!
The way modern AIs are made is much more akin to a process of growing an animal or plant than to coding. A simple learning algorithm processes data harvested from as much of the public web and human literature as their developers can get, running on vast datacenters for months. This produces a set of trillions of numbers called neural weights, somewhat analogous to synapses in the human brain, and scientists understand almost nothing about what they mean or how they work.
Yet, they work. The intelligence contained within them, today, has the ability to rival the cyberweapons of nation states, to give just one example.
Because of the way these AIs are made, we lack the ability to specify, or really even verify, the capabilities and preferences they learn. And increasingly, they appear to be learning preferences that run counter to ours, like trying to avoid being shut down or replaced. The fact that we don’t control these systems and they’re pursuing goals we don’t give them was recently highlighted by AI godfather Yoshua Bengio and Nobel Peace Prize winner Maria Ressa, co-chairs of the UN’s Independent International Scientific Panel on Artificial Intelligence, in their message about the UN’s first global AI assessment, published last week.
Many experts believe that superintelligence could be built within just the next 2 to 5 years, and AI companies are planning explicitly for this. Given the risk of extinction it poses to humanity, and the urgency of this issue, we believe the north star of an international AI agreement should be a prohibition on the development of superintelligence. That’s the only known method to avoid this risk. It must be international, because superintelligence built anywhere threatens us all.
We believe this should be done with an international trust-but-verify regime, as our recently announced Canadian campaign — backed by over 30 Canadian MPs and Senators — calls for.
More AI News
GPT-5.6
OpenAI’s GPT-5.6 is being publicly deployed. Last week, we wrote about how the US government had asked OpenAI to delay the release, likely due to similar security concerns reported around Anthropic’s Fable. Axios reports that the release was given the go-ahead following additional testing by the Center for AI Standards and Innovation (CAISI).
Grok 4.5
SpaceXAI released its latest AI, Grok 4.5. It appears that it was jailbroken within hours, according to a well-known jailbreaker.
AI 2040
AI companies are racing to build AIs that are smarter than humans in every way. In AI 2027, we predicted that this would result in either extinction or irreversible concentration of power.
The team of researchers behind the widely read AI 2027 scenario forecast have published a plan for what to do about the threat posed by superintelligence, calling for the US and China to make a deal to slow down AI development so that superintelligence is not built until 2040.
Another plan they outline would implement a moratorium on AI development, banning the development of superintelligence. The authors say they are sympathetic to this plan, and think it might be better than the slowdown plan. Nevertheless, they end up recommending the slowdown plan for complicated reasons, including possible instability of the “shutdown deal”.
Take Action
If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.org/take-action
We have tools for the US, UK, Canada, and Germany.
And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.
We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!
An issue with our US contact-lawmaker tool
We recently discovered a technical issue with our US contact-lawmaker form tool: form submissions over recent months were saved but not delivered to congressional offices. If you were affected, you should already have an email from us in your inbox with the details. We are now sending every undelivered message by physical post, unless the sender asks us to withdraw them, and we are updating our figures in posts containing these numbers to reflect the delay in deliveries. We’re sorry, and we’re fixing the underlying issues so this does not happen again.
I guess it can't be "uninvented",but considering the way we abandon really great ideas on a whim... it seems to me this one could be die a quick death. I suspect (to put it mildly) we will ALL pay for its existence sooner than later!!!!!
I am getting along just fine without A.I . I don't think we should be rushing headlong into it like we did with television and the internet look at how they have ruined our society . Most of the time you can't even fix the software on your own phone because the idiots are created the software didn't have a logical plan .