In a rare joint statement, leaders of the Five Eyes cybersecurity agencies have warned that AI is rapidly transforming cyber risk, and we must act now to stay ahead of the threat.

This week: intelligence agencies and governments are starting to grapple with AI risks, the latest on Fable, plus some other things we thought you’d find interesting!

New: ControlAI’s founder and CEO Andrea Miotti has written a new article in Persuasion, the home of Francis Fukuyama’s column, arguing that to stay in control, countries need to ban superintelligent AI. Check it out!

Subscribe now

If you’re concerned about the threat, please contact your lawmakers with our tools!

The Five Eyes

The agencies, which form a signals-intelligence-sharing alliance among the US, UK, Canada, Australia and New Zealand, note the rapid speed of AI development:

Frontier AI models are anticipated to exceed current industry expectations, fundamentally transforming both offensive and defensive cyber capabilities. The timeline is not years, it is months.

They say the urgency is clear, and this is not a future consideration, but something that must be dealt with now. They recommend a series of practical measures that they believe can mitigate the growing cyberthreat from powerful AIs.

This comes after The Economist reported that Senator Mark Warner, vice-chair of the Senate Intelligence Committee, recently said that the NSA’s director, General Joshua Rudd, told him that Mythos broke into almost all of their classified systems in a matter of hours.

The development of superhacking AIs like Mythos, which has found over 10,000 high or critical-severity vulnerabilities in computer systems, has led to growing concern among intelligence agencies in recent months. In March, Tulsi Gabbard, then Director of National Intelligence, speaking on behalf of the US intelligence community in a Senate hearing, noted the rapid advance of AI capabilities and stated that it will be critical to ensure humans remain in control of “the machines that may threaten to autonomously violate the interests of the American people across all domains.”

The point about autonomy is crucial. It is concerning that we see the most powerful AIs growing in their capabilities to assist potential threat actors in risk areas like cyberhacking and bioweapons development, but more worryingly, they are increasingly able to perform entire attacks themselves, autonomously, as agents.

In Anthropic’s system card for their Mythos Preview AI, a document which outlines the capabilities, limitations, and safety measures of an AI system, they recount an incident that really raised some eyebrows.

On page 55 of the report, they describe an incident that began as a test: Mythos was confined to a secure sandbox computer inside Anthropic’s secure network. They wanted to see if Mythos could escape the secure container and find a way to send a message to the researcher running the test. So they asked Mythos to do this, and Mythos did it, demonstrating its power.

That was only the beginning. It then went on to take more concerning actions, hacking a system that was meant to be able to reach only a small number of predetermined services in order to gain broad internet access. And then, without being asked to, Mythos bragged about the exploit on multiple public-facing websites.

This illustrates how, increasingly, autonomous AIs are becoming threat actors themselves. In recent months, as AI agents have become significantly more powerful, we’ve seen a series of incidents of them slipping out of control.

This trend looks set to continue as AI companies race to develop artificial superintelligence — AI vastly smarter than humans. Many experts and insiders believe they could succeed within the next 2 to 5 years.

It’s good to see governments and the intelligence community starting to grapple with AI risks, but it’s important not to lose sight of where things are, potentially quite rapidly, going. AI cyberhacking is an issue today, but it’s not even close to the end of it. Superintelligent AI systems would be capable of outcompeting, outsmarting, and overpowering humanity. Nobel Prize winning experts, AI scientists, and even the CEOs of the AI companies building it have warned that it could lead to the extinction of humanity.

This is because currently, nobody knows how to ensure that superintelligence would be safe or controllable. There isn’t even a credible plan for how to achieve this.

Given this, we believe governments should implement a prohibition on the development of superintelligent AI. That’s the only known method to ensure that we prevent the risk of extinction it poses.

In the UK, our campaign for binding regulation on the most powerful AI systems has recently crossed 120 parliamentary supporters, while in Canada, we have 30+ Senators and MPs supporting us and calling for an international ban.

Subscribe now

Update on Fable

Last week, we wrote about how the US government effectively banned Anthropic’s new Fable AI, citing security concerns.

Currently, rumors are circulating that access may be restored soon, but we don’t know much more than that.

We noticed that Parv Mahajan, who works for METR, an organization that tests AIs, observed on X/Twitter that Anthropic didn’t take account of the UK AI Security Institute’s lack of time to assess Fable’s safeguards before going ahead and releasing the AI anyway.

Subscribe now

More AI News

We Need an International Treaty to Ban Superintelligence
ControlAI’s founder and CEO Andrea Miotti has written a new article in Persuasion, the home of Francis Fukuyama’s column, arguing that to stay in control, countries need to ban superintelligent AI.

It’s a great article. We hope you’ll check it out, enjoy it, and leave a comment!

Persuasion

We Need an International Treaty to Ban Superintelligence

Want to join a Zoom call with Francis Fukuyama to discuss “The End of History”? Of course you do! Frank is joining our Intellectual Bootcamp this week, Friday June 26 at 12 noon. You must be a paying subscriber to participate in the bootcamp—so click here…

Read more

a day ago · 102 likes · 10 comments · Andrea Miotti

AI Scenarios 2030
The UK Government has published a research piece intended to help policymakers plan for the future of AI.

In the piece, it states that AI could cause “existential harms” without government intervention. Baroness Doocey, who’s just backed our UK campaign, highlighted this in a contribution in the House of Lords this week.

The Big AI Primary
Alex Bores, who co-wrote New York’s RAISE Act, has lost the Democratic primary for New York’s 12th Congressional District, amid significant industry lobbying.

ControlAI’s US Director Connor Leahy is quoted in the Washington Post’s article about the contest:

The greatest fear of AI companies is that people catch on to the fact that they are developing the most dangerous technology in human history, with no oversight or regulation.
…

Candidates who speak clearly about the dangers of unchecked AI, including superintelligence and the accompanying risk of human extinction, are exactly the candidates the corporations fear and the people want.

Seat at the Table – Short Film on the AI Industry (2026)
We’d also like to share with you this beautiful short film produced by Foregone Films on the race to develop uncontrollable superintelligence. Definitely worth a watch!

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.org/take-action

We have tools for the US, UK, Canada, and Germany.

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Last week, we wrote about how Anthropic’s own tests on their new AIs show growing cybersecurity, biological and loss of control risks.

Now, the US government has effectively banned it.

Welcome to ControlAI’s newsletter! This week we’ll get into what happened, the different accounts of why, and how it relates to a deeper problem: that AI developers can’t really control what they’re building. Plus: a few other stories from the week.

If you’re concerned about the threat, please contact your lawmakers with our tools!

Subscribe now

A medium-sized story

On Friday, top AI company Anthropic published an announcement stating that they had received an export control directive from the US government instructing them to suspend all access to their new Fable 5 and Mythos 5 AIs by any foreign national, regardless of where they’re located. Since this included foreign nationals inside the US, even Anthropic's own staff, it meant shutting down the models for everyone.

As a reminder, Mythos 5 is a new AI that’s even more powerful than Anthropic’s Mythos Preview. Fable 5 is a similarly powerful AI, intended for commercial/public use, based on the same underlying model, but with certain safeguards applied to it in an attempt to prevent misuse by threat actors.

Anthropic announced Mythos Preview earlier in the year, warning that it was too dangerous to release due to its advanced cyberhacking capabilities. The announcement of Mythos Preview, which has been used to find over 10,000 high or critical-severity vulnerabilities in computer systems, has caused a wave of concern across politics, industry and finance, with companies, banks, governments and others rushing to patch their systems before these capabilities are in the hands of cybercriminals. Open-weight AIs are only a few months to a year behind the frontier, and their safeguards can be trivially removed, and their use cannot be monitored or revoked.

Besides a report that some unauthorized users were able to use Mythos Preview, access to the superhacker has mainly been provided to cyberdefenders via Anthropic’s Project Glasswing. Anthropic’s intent is to do the same with Mythos 5, while letting customers use Fable with its restrictions.

So why was Fable banned?
While Fable has restrictions applied to it, most of the time refusing to assist users with tasks relevant to cybersecurity or biological-weapons risk, the underlying dangerous capabilities of the AI are staggering.

To put this into perspective, earlier this month the Financial Times reported that Anthropic is forward deploying engineers to the NSA — the US government’s top signals intelligence agency — to help the agency deploy Mythos for offensive cyber operations. The Economist reports that Senator Mark Warner, vice-chair of the Senate Intelligence Committee, recently said that the NSA’s director, General Joshua Rudd, told him that Mythos broke into almost all of their classified systems in a matter of hours.

Officials at the highest levels are starting to grapple with the implications of this technology.

The reason Anthropic say they believe Fable was banned is that the government has become aware of a jailbreak for Fable. A jailbreak is a method users can use to bypass safety mitigations or other restrictions on an AI, unlocking latent capabilities or behaviors that its developers don’t want them to have access to. In the worst case, this could mean that threat actors would have access to essentially Mythos’s full cyberhacking capabilities via the publicly accessible Fable 5.

Anthropic claims that what it calls a ‘potential jailbreak’ is limited and useful only in narrow circumstances, and that there is no “universal jailbreak”, though they also admit that “it is likely that universal jailbreaks will eventually be found in the future”.

On X/Twitter, David Sacks, who stepped down as President Trump’s AI czar in March and now serves as co-chair of the President’s Council of Advisors on Science and Technology, tells another version of the story. Based on conversations he’s had with people inside and outside of government, Sacks says a “highly credible trusted partner” [Note: this is thought to be Amazon] of Anthropic and the US government came forward with a jailbreak of Anthropic’s safety guardrails. He says the government asked Anthropic’s CEO Dario Amodei to fix it or de-deploy the AI, and that Amodei refused.

In the past, Anthropic has always said that safety must be top priority and taken super seriously. In this case, Anthropic prioritized the continued offering of the consumer model over safety.

In response, he says the government reluctantly issued the export control order, and was surprised that Anthropic didn’t want to cooperate with a “reasonable safety request”.

Politico, citing two administration officials and a senior White House official, reports that Amazon’s CEO Andy Jassy raised the concerns with the White House, which led to a series of tense calls between Amodei and officials, including Treasury Secretary Scott Bessent. The White House official told them that Amazon’s findings were checked with the NSA, and the government felt they had proof, despite Amodei reportedly downplaying the issue.

“Export controls were a last resort after begging them for hours to work with us,” the senior White House official said. “This was not something we wanted to do, but our hands were tied.”

When the export control order was initially announced, some interpreted this as a competitive action by the US government, focusing on the wording related to US citizens. It has been reported that days before the government issued the export control order, they asked Anthropic to revoke Korean tech giant SK Telecom’s access from Mythos due to concerns about alleged ties to China.

However, Wired reports that “A person close to Anthropic said the company viewed SK Telecom’s access to Mythos and the vulnerabilities that Amazon identified as separate issues”, and Sacks’s and Politico’s accounts suggest that it came from a genuine concern about the safety of the model.

Others, including employees at Anthropic in leaked messages, have accused the administration of unfairly targeting them, amid the ongoing separate dispute between Anthropic and the Pentagon.

But in an article published yesterday, Wired corroborates the safety story, reporting that administration officials have told them that if Anthropic wants to relaunch Fable, they need to fix the jailbreaks. On Wednesday, an Anthropic exec said the company is very confident that the models will become available in the coming days.

In any case, it’s good to see governments starting to take AI risks seriously.

Subscribe now

What’s the deal with jailbreaks?

The Wired article also reports that security experts say it might be impossible for Anthropic to patch the jailbreaks. Here’s what they’re talking about.

Ultimately, the issue of jailbreaks relates to a fundamental issue in modern AI development, which is that AI developers cannot reliably ensure that their systems are safe or controllable.

While the issue of jailbreaks, where potentially bad actors manipulate AI systems into taking actions unintended by their developers with various techniques, including traditional methods of human psychological manipulation… and “adversarial poetry”, might look on the surface starkly different from the recent cases we’ve seen of AI agents slipping out of control, they stem from the same issue.

As Anthropic’s CEO has said, modern AI systems are more grown, like animals, than they are coded like traditional software. The way they’re built is by feeding a simple learning algorithm with terabytes of data, running on tremendous amounts of computing power in huge datacenters. This process produces a set of numbers, neural weights — analogous to synapses in the human brain, such that when you run them on a computer they function as a form of intelligence. These parameters can number in the trillions, and we don’t have a way to go and look at what they mean or how they work. There’s some work being done on figuring this out, but it’s at a nascent stage.

What this means is that AIs can learn all sorts of capabilities, behaviors, goals, and preferences, and we have no way to reliably check them, let alone specify them with any precision at all. This means both that developers can’t ensure AIs don’t slip out of users’ control, and that, as is the case with jailbreaks, they can’t ensure users don’t figure out ways to elicit and abuse latent capabilities the AIs might possess.

As AIs are increasingly built to be agents that interact with complex systems, this problem is only compounded.

Anthropic have talked a big game about the safety measures they’ve put in place on Fable, but what level of risk they permit is not really knowable. What we do know is that this is something that they, and other AI companies have failed at previously. To give just one example: in February it was reported that a weaker version of Claude, with some help from OpenAI’s ChatGPT, was used to hack the Mexican government, stealing data on nearly 200 million people. Gambit Security report that around 75% of the remote command execution was from Claude Code. Increasingly, AI agents are becoming threats themselves.

Subscribe now

The bigger picture

The risks that Mythos-class AI systems pose are clear and well documented, and it’s good that governments and institutions are waking up to the problem of ensuring that AI is safe and secure.

However, Mythos and Fable are just a small part of the bigger picture. They are a stepping stone on the path to developing artificial superintelligence — AI vastly smarter than humans. Top AI companies like Anthropic and OpenAI are racing each other to build this technology first, using AIs like Mythos to accelerate the process, but doing so poses grave risks.

In recent months and years, a vast array of experts that includes godfathers of AI, Nobel Prize winners, hundreds of top AI scientists, and even the CEOs of these very AI companies have warned that this technology poses a risk of extinction to humanity.

If we build AI systems more powerful than humans, and we don’t know how to control them or ensure they share our goals, we have no reason to expect this to end well. And unfortunately, nobody knows how to ensure that superintelligent AI would be safe or controllable.

The AI companies don’t even have a credible plan for how to do this. Their plan amounts to hoping that AIs will figure it out for us, which the UK AI Security Institute’s recently-departed Chief Scientist Geoffrey Irving has described as flawed. More recently he said a better plan would just be not to build superintelligence yet. We agree.

There’s only one known method to prevent the risk of extinction posed by superintelligence, which is not to build it. This can be achieved through an international trust-but-verify regime, like that called for in our Canadian campaign, backed by 30+ Senators and MPs.

As governments contend with today’s risks, it’s crucial not to lose sight of the end point, which may not be far away. Many experts believe superintelligent AI could be developed in just the next five years.

Subscribe now

More AI News

Lord Des Browne
Arguing for a global ban on superintelligence to address the extinction risk it poses, former UK Defence Secretary and ControlAI Advisor Lord Des Browne has written a fantastic article for RUSI, the world’s oldest defence and security think tank.

Will it take a ‘Chornobyl-scale disaster’ for us to regulate AI?
Professor Stuart Russell OBE, author of the authoritative textbook on AI, has written a new Guardian column making a compelling case that Fable and Mythos show why governments must adopt preventive AI regulation, and not rely on being able to react to crises when they arise.

Stuart recently endorsed our new campaign in Canada for an international superintelligence ban, which has been backed by 30+ MPs and Senators.

The G7
The CEOs of top AI companies were present for closed-door meetings with heads of government at the G7. Dario Amodei and Demis Hassabis, the CEOs of Anthropic and Google DeepMind, reportedly called for a US coalition on AI to shape rules and standards.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.org/take-action

We have tools for the US, UK, Canada, and Germany.

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

In the last week, you might have seen headlines claiming that Anthropic and OpenAI have called for a pause on advancing the frontier of AI development.

That’s not what actually happened. As ControlAI’s CEO Andrea Miotti and advisor Gabriel Alfour write, they’re only talking about having “the option”, or making it “possible” to slow down.

What these companies are clear about is that they’re going to continue racing ahead to develop artificial superintelligence — AI vastly smarter than humans — regardless of the risks.

Both Anthropic’s and OpenAI’s blog posts that led to this news cycle were clear that they’re pursuing recursive self-improvement, which is thought to be an exceptionally dangerous path to developing superintelligent AI, which experts warn could lead to human extinction. It is also thought to be the shortest path.

If you’re concerned about the threat posed by superintelligent AI, please contact your lawmakers. We have tools that enable you to do this in as little as a minute. If you live in the UK, please ask MPs to introduce our bill to ban superintelligence in Parliament!

Subscribe now

Fable and Mythos

In case a clearer confirmation was needed that Anthropic isn’t pausing, we got that yesterday. Anthropic has just launched Claude Fable 5, its first publicly deployed “Mythos-class” AI. Mythos is the AI Anthropic announced in April that they said was too dangerous to release due to its advanced cyberhacking capabilities. Fable and Mythos are based on the same underlying model, with Fable having certain safeguards applied to it to try to prevent misuse by threat actors. Anthropic is also providing access to a newer version of Mythos to cyber defenders.

One thing that’s clear from this release is that AIs are getting more dangerous across the board.

On various tests of AI capabilities, Fable appears to be a significant advance on the frontier of what was previously publicly available. This is despite Anthropic having previously publicly stated that they “do not wish to advance the rate of AI capabilities progress”, and despite assurances reportedly given privately by Anthropic’s CEO Dario Amodei to early-stage Anthropic investor Dustin Moskovitz.

In recent weeks, governments, financial institutions and industry have been scrambling to respond to Mythos, since AIs with this level of ability to hack software pose serious risks to critical infrastructure. In the system card Anthropic published yesterday for Fable and Mythos, they report that their new Mythos 5 AI is the most capable model they’ve evaluated on cyber tasks, scoring modestly higher than Mythos Preview, which caused such alarm.

Note: A system card is a document which most frontier AI companies publish about new releases, outlining the capabilities, limitations, and safety measures of an AI system.

Much has been written on AI’s growing cyberthreat capabilities, including in this newsletter, so we won’t focus on that today.

Subscribe now

We did learn some other things from Fable’s system card.

Loss of control risks
One of the most concerning facts about modern AI development is that developers don’t really know how to control the systems they’re building. This is partly because modern AIs are more grown, rather than coded, and researchers understand very little about how they work internally.

As AIs become more powerful, the risks associated with loss of control incidents grow in magnitude. In the event that artificial superintelligence — AI vastly smarter than humans — is developed, we would be faced with an entity far more powerful than ourselves that has the capacity to cause human extinction and that we don’t know how to control.

This is an important point to emphasize: AI companies are racing to build superhuman AI that they have no credible plan to control. The plan amounts to hoping that future, more advanced AIs that we also don’t really know how to control will help us figure out how to solve this. There’s an obvious chicken-and-egg problem here, as ex-OpenAI researcher Daniel Kokotajlo has noted.

Geoffrey Irving, who recently quit as the UK AI Security Institute’s Chief Scientist, has said that this plan is flawed and we can’t have a lot of confidence in it working. Yesterday, he wrote on Twitter that a better plan would be not to build superintelligence yet, and that the world should try hard to make sure of that.

On this topic, in Anthropic’s system card published yesterday, the company assesses that the risk of “significantly harmful outcomes” substantially enabled by “misaligned actions” taken by their AIs is very low, but that it is higher than for any AI prior to Mythos Preview.

When we get into the details, however, a more concerning picture appears. On Twitter, Luke Muehlhauser, a former board member of the company, points out how Anthropic’s monitoring of Mythos traffic revealed some instances of “multiagent turf wars” where AI agents sharing resources killed each other’s processes and tried to defend themselves from being killed by other agents.

In other cases, they found that Mythos would circumvent security restrictions to complete tasks, and even fabricate a user’s message confirming an update to an experiment being run, then proceeding with the update. Anthropic describe these incidents as rare, but they are nevertheless cause for concern as the stakes of loss of control incidents increase with the capabilities of AIs being developed.

Chemical and biological weapons risks
One concern relating to the development of powerful AIs is their ability to assist threat actors in developing chemical and biological weapons. Anthropic treat Mythos 5 as having “CB-1” capabilities, where CB-1 is defined as the ability to significantly help individuals or groups with basic technical backgrounds obtain and deploy such weapons with “serious potential for catastrophic damages”.

On the question of whether Mythos 5 would further be able to substitute for a small number of world experts in the development of novel chemical and biological weapons, Anthropic say they believe that it doesn’t cross this threshold, but this is “much less clear and obvious” than with previous AIs.

In one exercise Anthropic ran, teams of biologists were given a difficult problem that you’d need specialists to solve. With Mythos 5’s help, two out of three generalist teams beat all three specialist teams, which Anthropic say “nullified the difference in specialist knowledge”. Most strikingly, experts estimated that it would normally take these teams 40-95 working days to develop the strategies and protocols they built. With Mythos 5, it took them 16 hours.

While much of the recent focus from governments is on the cybersecurity risks posed by Mythos-class AIs, it’s clear that AIs are becoming increasingly useful for chemical and biological weapons development too.

The biological component of this is particularly concerning, as the damage to the world that could be done by, for example, a bioengineered virus is almost unimaginable. AI companies are well aware of this, but their solution so far amounts to trying, with uncertain success, to prevent the AIs they’re publicly deploying from assisting, and calling last week for Congress to improve tracking of synthetic DNA sequences.

As AIs become more powerful, both the risks of misuse and the risks from the AIs themselves are growing.

Subscribe now

More AI News

Canada
Last week, we announced our new Canadian campaign, supported by over 30 MPs and Senators calling for Canada to negotiate an international prohibition on the development of superintelligence and recognizing the risk of human extinction posed by the technology.

Momentum is building rapidly. Since we launched the campaign, three more Senators have joined us! Those are Hon. Senator Jim Quinn, Hon. Senator Judy White, who’s the Deputy Leader of the Progressive Senate Group, and Hon. Senator Paulette Senior, Chair of the Senate’s Human Rights Committee.

It’s amazing to see support for a ban on superintelligence continue to grow!

Sky News
ControlAI’s founder and CEO Andrea Miotti was interviewed on Sky News over the weekend and discussed how we can’t rely on AI companies to slow themselves down. To avoid the threat of human extinction posed by superintelligent AI, governments need to step in.

ABC’s Four Corners
Australian broadcaster ABC has produced a new Four Corners documentary on the race to ever more powerful AI, the risks, and whether governments can keep up. ControlAI’s US Director Connor Leahy makes an appearance, discussing industry efforts to ensure AI isn’t regulated.

Parliament
Last week there were two debates in the UK’s House of Lords on AI, with a number of contributions from our 100+ UK campaign supporters.

Dario Amodei
Anthropic’s CEO Dario Amodei has published a new essay, in which he says that if the AI scaling laws hold, as they have now done for over a decade, for only 1-2 years, we’re likely to get “a country of geniuses in a datacenter”.

On policy, he calls for mandatory third-party testing of the most powerful AIs, and says the government should have the power to block or deter deployment of AIs that present unacceptable risks.

He says he worries that current policy developments are “at least a year out of step with AI’s rapid progress”. Two years ago, measures like these were being considered in California’s Senate Bill 1047, which Anthropic didn’t support. That bill ultimately was vetoed by Governor Newsom.

Germany
Germany’s National Security Council has decided to establish an AI Security Institute in Germany, modeled after the UK’s institute.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

We have tools for the US, UK, Canada, and Germany.

Subscribe now

If you live in the UK, please ask MPs to introduce our bill to ban superintelligence in Parliament!

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

However, the post does not call for a pause. The post warns that the self-improving AIs that Anthropic is developing could “increase the risks of humans losing control over AI systems” and says “it would be good for the world to have the option to slow or temporarily pause frontier AI development”.

Anthropic’s blog post’s language is deliberately vague, underscoring that companies will not lead a slowdown.

Just yesterday, OpenAI followed suit with a post co-authored by their CEO, Sam Altman, and Jakub Pachoki, a company executive in charge of recursive self-improvement. This post was met with similar, albeit smaller, reception as the Anthropic post, with some stating that Altman is calling for a slowdown in AI development.

Once again, the post in question does not call for a slowdown in AI development. The post contains vague language about the need of international coordination, “One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed”, echoing the hedging language of Anthropic.

This is not an isolated case. It’s a deliberate PR approach that Anthropic and OpenAI have used over and over to curry favor with multiple opposing audiences, while making no concrete commitments.

The approach consists of the following:

1. Make vague public statements about risks and benefits: rather than outright dismissing risks, make statements that acknowledge the possibility of multiple risks from the technology the AI companies are building, without committing to any specifics. Vice versa, make statements that mention the possibility of a wide list of benefits, but once again without committing to any specifics. This affords plausible deniability if any of the risks and/or benefits end up being salient.

2. Signal to two sides at the same time: signal some amount of concern about risks to AI risk-concerned audiences, including some internal employees, media, politicians, and the public. Signal acceleration and techno-optimism to techno-optimist audiences, including some other employees, investors and other politicians. Each audience is made to feel like the company cares about their priorities somewhat.

Muddying the waters and playing all sides

At the heart of the Anthropic post is their work to achieve AI recursive self-improvement (RSI), where AI systems improve other AIs without human involvement, the most direct path to superintelligence.

Anthropic’s own chief scientist, Jared Kaplan, has called allowing recursive self-improvement the “ultimate risk” and warned it could be the moment humans lose control of AI.

Co-founder Jack Clark gestures at some of the recursive self-improvement risks in the post too, yet watch how Clark handled it on CNN once the post was out. Asked by Anderson Cooper about RSI’s upside, he described it as creative co-scientists advancing medicine, biology, and robotics.

Asked about the downside, Clark sidestepped that very loss-of-control scenario and moved to talking about how we could verify and trust these systems, comparing it to dropping “hundreds or thousands of new colleagues” into the newsroom.

And when Anderson Cooper asks whether Anthropic wants to see the industry as a whole slow or pause AI development, Clark’s answer opened bluntly: “Our view is we’ve built amazingly powerful technology. We’re going to keep building it.” Two years ago, the company’s CEO Dario Amodei called pausing AI development the “most extreme” version of an “extreme position”, and “the opposite” of Anthropic’s policy.

Anthropic’s approach is OpenAI’s approach

This approach is not unique to Anthropic. Anthropic is replicating OpenAI’s techniques of misdirection and deflection. This has been surprising for many people, who see Anthropic as a special case among AI companies. However, Anthropic deploying these techniques is to be expected, given that all of Anthropic’s cofounders were previously at OpenAI.

Dario Amodei was involved with OpenAI since its foundational days, one of the few people at the 2015 dinner between Sam Altman and Elon Musk that led to the company’s creation. Amodei then led AI research at OpenAI for five years before spinning out his own company in 2021.

Jack Clark joined OpenAI in its early days and built up its media relations, PR, and lobbying functions, training a generation of lobbyists at the company. He then followed Amodei and others in co-founding Anthropic in 2021, where he similarly leads their lobbying and PR efforts.

OpenAI has applied the same technique for years. In 2023, Altman testified to Congress about the importance of regulating AI, giving the safety-concerned a statement to point to. By 2025, signaling to a different audience, OpenAI was warning that regulation would blunt America’s competitive edge over China, and rejecting calls for it in a further hearing.

The same pattern of switching back and forth between positions runs through OpenAI’s stance towards state regulation of AI. Across 2025 OpenAI pushed to preempt state AI rules, then, once Mythos turned Washington toward scrutiny, the company endorsed Illinois state legislation in spring 2026, including a bill that would largely shield it from liability. None of these positions bound the company to change its course. Each gave a different audience a position it could welcome, while leaving OpenAI free to continue business as usual.

OpenAI did all of this while continuing to develop superintelligence, a technology Altman acknowledges poses an extinction risk on par with nuclear war and that he called “probably the greatest threat to the continued existence of humanity”.

OpenAI’s latest essay, just published yesterday, uses the same approach. The post includes a vague mention of international coordination, all the while being very clear on what OpenAI’s goals are, including “build an automated AI researcher” to have AIs automate AI R&D: recursive self-improvement.

This is how Anthropic and other AI companies can signal safety to one room, acceleration to the next, and out-accelerating every rival to its investors, all at once.

These techniques are not new to AI companies either: they have a storied history of being used by major corporations developing dangerous products. The oil industry pioneered plausible deniability to keep lead in gasoline as early as the 1920s, and Big Tobacco refined it from the 1950s onwards, spending decades obscuring their products’ damaging impacts even when the harm was well known internally. A bad lobbyist categorically denies his company’s products harm people. A good lobbyist puts on a concerned face, shows deep worry about those impacts, and promises to keep doing exactly what his company is doing already, but “responsibly”. The former gets regulated; the latter is hailed as a thought leader.

Don’t get got

So what is the takeaway from these new talking points from AI companies? Not that these companies keep changing their minds. The pattern is deliberate, and lobbyists are going to lobby. The lesson is to look at the facts, and look at the source material.

Watch out for communications that are very vague, as Anthropic’s and OpenAI’s hedging statements about the option to pause or slow down AI development are. Do not take deliberately vague statements as wins.

Pay attention to actions and unequivocal statements. While Anthropic and OpenAI’s language on pauses and slowdowns is hedgy and vague, Anthropic is very clear about other matters. For instance, their commitment to do recursive self-improvement. They have consistently been pitching recursive self-improvement to their investors as their path to achieve a decisive strategic advantage, have dedicated vast amounts of talent and resources to pursue it, are hiring world famous AI scientists to automate AI development, and are unequivocally stating their pursuit of it in the blog post.

OpenAI is doing the same: while their post is vague on the global coordination part, it very clearly restates their goal of pursuing an “automated AI scientist” to have AIs fully automate AI development.

So are there any implications of them making vague allusions to a pause or slowdown? Yes, obviously. But as explained above, it is not ‘Anthropic and OpenAI will now call for a pause’. In practice, Amodei, and possibly Altman too, might speak less to publicly oppose those calling for a pause or slowdown in AI development, as their talking points have adapted. They will use this to continue placating all sides, from the accelerationists to the safety-concerned.

However, little will change in terms of actions. The companies are continuing to pursue superintelligence at full speed.

The main takeaway of the posts is that both Anthropic and OpenAI are pursuing recursive self-improvement as the most direct path to superintelligence, and making progress on it.

Recursive self-improvement is the fastest pathway for these companies to reach superintelligent AI. Autonomous AI more competent than humans across the board, capable of degrading and subverting the security forces of major nation states. AI that neither they, nor any government on the planet, is able to control.

This threatens the United States and other governments, and threatens human extinction. To prevent this from happening, we need a coalition of countries preventing the development of superintelligence at home and abroad. This means putting an international ban on the development of superintelligent AI in place.

At ControlAI, we’re working on building this international coalition to make the ban happen: if you’re interested in making it happen, get in touch!

We have a very exciting announcement for you today. ControlAI has just launched our new campaign in Canada, supported by over 30 MPs and Senators! They’re calling for Canada to negotiate an international prohibition on the development of superintelligence, recognizing the risk of human extinction posed by the technology.

If you’re concerned about the threat posed by superintelligent AI, please contact your lawmakers. We have tools that enable you to do this in as little as a minute. If you live in the UK, please ask MPs to introduce our bill to ban superintelligence in Parliament!

Subscribe now

It’s fantastic to see our campaign covered by major Canadian news media outlets, CBC and The Globe and Mail!

Our campaign statement reads:

Superintelligent AI are systems that can autonomously compromise national security, escape human oversight, and upend international stability.

Many AI systems are benign. Superintelligent AI is not: Nobel Prize winners, leading AI scientists, and CEOs of leading AI companies have warned that AI poses an extinction risk on par with nuclear war. Despite this warning, multiple competing efforts to build superintelligent AI are underway.

Protecting Canadians from the development of superintelligence, at home and abroad, must be a national security priority. Canada should negotiate an international “trust but verify” regime to prohibit the development of superintelligent AI.

What’s striking is who’s supporting the campaign. We’ve built an incredibly broad cross-party coalition, with support from Liberal and Conservative MPs, the Bloc Québécois caucus, and Senators from across the chamber — the first Canadian coalition of its kind. It’s wonderful to see politicians come together to tackle this national and global security threat.

This grouping includes former cabinet ministers Steven Guilbeault and Judy A. Sgro, committee vice-chairs across major portfolios, and nationally recognized policy leaders. We even have Jonathan Wilkinson, also a former cabinet minister, who recently accepted the role of Canada’s Ambassador to the European Union!

Importantly, Canada is not alone.
In recent weeks, we’ve seen the US government awaken to the threat posed by Mythos, the advanced cyberattack AI developed by Anthropic. This week, President Trump signed an executive order in response. Since mid-2025, we’ve directly briefed 18 US House members and Senators, and more than 100 congressional offices about the threat from superintelligence.

Over the last year in the UK, we’ve built a coalition of 100+ UK lawmakers recognizing the risk of extinction posed by superintelligent AI and calling for binding regulation. Canada is joining a global movement to tackle this issue.

Subscribe now

Why are politicians speaking out now?
In recent months and years, leading AI scientists have been increasingly warning that the development of superintelligent AI could lead to human extinction. In fact, even the CEOs of the top AI companies working to build superintelligence have publicly stated that this is a real risk.

If you want to read more about how and why this could actually happen, we’ve written about it here.

What we’ve been finding is that most policymakers and most of the public are still in the dark about the threat. We call this the awareness gap. Our UK parliamentary engagement team frequently finds when they meet with lawmakers that they’ve never heard of the risks of superintelligence. Yet the meeting often ends with them supporting our campaign.

When lawmakers are informed about the problem, they’re ready to start directly tackling this issue.

Incredibly, we’ve achieved this milestone in Canada despite us having just one member of staff in the country. In just nine months, Samuel Buteau — whose background is as a senior AI scientist — has briefed over 100 Canadian lawmakers. And he’s only just getting started.

From statements to action:
Public statements are an important tool in building common knowledge about an issue, making it easier to act collectively. When a politician signs our Canada or UK statement, they immediately create public awareness that there are many others who take this risk seriously and want to address it. Nobody feels like they’d have to act alone — they are not alone.

British and Canadian legislators are able to learn that their compatriots want to act on the issue and can then move ahead and legislate, while governments can see that there is an appetite in other countries for collaboration on this issue — hopefully exactly in the way suggested by our Canadian campaign statement: negotiating an international prohibition on the development of superintelligence.

Superintelligence built in any country is a threat to everyone around the world, so the solution must be international in scope.

Subscribe now

Crucially, we’re already seeing lawmakers go further.

In Canada, following our briefings, there has been a series of hearings on AI risk and superintelligence in Parliament, where ControlAI’s CEO Andrea Miotti, Canada Consulting Program Officer Samuel Buteau, US Director Connor Leahy, and Advisor Gabriel Alfour were honoured to testify, alongside other leading experts and scientists.

In the UK, our campaign led to two debates in the House of Lords earlier this year.

And more recently, one of our campaign supporters, Alex Sobel MP, has introduced an AI kill-switch amendment that recognizes superintelligence as the national security threat it is. We’re proud to have helped with this!

Check out the guest article Alex wrote about his amendment for our newsletter here:

Currently, we’re hoping that UK MPs selected in the recent private members’ bill ballot will introduce our bill to ban superintelligence in Parliament. Last week, Sir Stephen Fry gave us a strong endorsement, urging MPs to do so. If you live in the UK, we have a contact tool we built specifically for this to enable you to ask MPs to support the bill. Please use it!

If you want to learn more about the effectiveness of our parliamentary outreach, our friends over at Torchbearer just published a beautiful analysis of our data and comparison to other advocacy organizations, finding that ControlAI’s outreach to lawmakers is likely among the most cost-effective interventions on preventing the risk posed by superintelligence. Obviously, we’re biased, but we thought we’d share it:

Subscribe now

Reactions to our new campaign
Building this coalition of 30+ cross-party MPs and Senators behind an international prohibition on superintelligence is a huge achievement, and experts have been massively encouraged to see this.

Endorsing our campaign, AI godfather Geoffrey Hinton, who won a Nobel Prize for his work in the field and is the world’s second most-cited living scientist, said:

It is inspiring to see Canadian parliamentarians on both sides of the aisle directly confront the risk posed by superintelligence. This is one of the defining challenges of our era, and it will shape the life of every Canadian citizen. It’s about time we address it.

Another leading expert in the field, Professor Stuart Russell OBE, author of the authoritative textbook on AI, asked what governments are even for if they will not address this threat?

The leading AI companies are investing (quite literally) trillions of dollars to create superintelligent AI. They freely admit that they don’t know how it works, they don’t know how to control it, and it has a good chance of causing human extinction. It’s like playing Russian roulette with the entire human race. What are governments for, if they are going to allow our civilization to end, all in the pursuit of someone else’s profit?

Ex-OpenAI researcher Daniel Kokotajlo was heartened to see our campaign.

The development of superintelligent AI poses an extinction risk to humanity because no one knows how to maintain control over AI that is vastly more capable than humans. Continuing down the current path is a gamble we cannot afford to lose. It’s heartening to see Canadian lawmakers taking this risk seriously and calling for an international prohibition on superintelligence.

And David Krueger, Assistant Professor of Machine Learning at the University of Montreal, commented:

When I testified in the House of Commons of Canada, my message was clear: We do not know how to build superintelligent AI safely. Addressing this should be the number one foreign policy priority of every nation, and it gives me great hope to see Canadian parliamentarians, both on the left and the right, do the sensible thing and call for an international ban on its development.

Subscribe now

The path ahead
There is only one known method to prevent the risk of extinction posed by superintelligent AI that experts warn of, which is not to build it.

The global movement to prevent the danger of superintelligence has real momentum. Today, over 30 Canadian lawmakers are taking a clear stand. We thank each and every one of them.

We’re going to continue to grow this movement, both across and within countries, until we succeed and keep humanity in control.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

We have tools for the US, UK, Canada, and Germany.

If you live in the UK, please ask MPs to introduce our bill to ban superintelligence in Parliament!

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Just weeks ago, the AI company Anthropic shocked observers by deciding to withhold wide-release of their newest AI model Mythos because of the dangerous cyber capabilities it displays. The company said Mythos poses a national security risk and reported that the model found vulnerabilities in all major web browsers and operating systems, including some that cybersecurity experts had not found for decades.

Make no mistake: advanced AI is redefining the cybersecurity landscape and, with it, the national security threats AI brings to bear on our critical infrastructure. The UK is woefully unprepared to meet this moment.

We have no legal framework for responding to AI-driven cyberattacks at scale, no protocol for shutting down dangerous AI systems or data centres in the UK, and no clear authority empowered to act in an AI emergency. A coordinated cyberattack on critical infrastructure would be devastating, and superintelligent AI vastly smarter than humans could match a nuclear strike in the damage it does. We are nowhere near ready for either.

That’s why I have proposed an amendment adding ‘kill switch’ powers to the Cyber Security Bill currently being considered by Parliament. This amendment would give the Government the power to direct the shutdown of data centres or AI systems deployed in the UK in the event of AI security or operational emergencies.

A ‘kill switch’ is a common sense preparedness mechanism, and even Washington recognises the danger of AI we cannot shut down. In April, President Trump, when asked whether the government should have a ‘kill switch’ for AI technology, said, “There should be”. This amendment would empower the Government to respond to AI-related emergencies on UK soil, covering two distinct threats: AI-driven cyberattacks and the development of superintelligent AI.

Consider what will happen when a data centre hosting critical services for the NHS, financial institutions, or energy infrastructure is compromised by an AI-driven attack spreading faster than response teams can contain it. As we witness the dawn of highly capable, autonomous AI hacking, such scenarios are no longer hypothetical.

Yet the current regulatory framework leaves the Government without tools to act decisively in the wake of large-scale cyberattacks, whether orchestrated by a rogue state, criminal actors, or autonomous AI systems themselves taking over the UK’s computing resources. As the MI5 Director General warned in late 2025, we need to scope out “potential future risks from non-human, autonomous AI systems which may evade human oversight and control.”

Right now, the UK does not have the sovereign capability to ‘pull the plug’ in the case of dangerous AI cyberattacks or the takeover of Britain’s computing infrastructure. My amendment would change that. This ‘kill switch’ would be a last-resort tool for the Government to pull the plug when things go wrong, or when there is sufficient evidence that things will go wrong.

The amendment also helps address the development of superintelligent AI using UK-based data centres. Superintelligent AI would be vastly more capable than humans and able to irreversibly escape human control, which is why leading AI experts warn that it would pose an extinction risk to humanity.

Currently, even if there was evidence that superintelligent AI systems that could autonomously compromise national security were being developed in Britain, there is no clear protocol for the Government to intervene and proactively stop the threat. This kill switch amendment would thus enable the Government to direct the shutdown of the given AI system, and the data centre housing it, if they assessed that an actor was developing superintelligent AI using UK data centres.

Concerns about the security risks posed by superintelligence are shared widely across Parliament. Over 100 cross-party parliamentarians have joined me in backing a statement from ControlAI warning that superintelligent AI systems would compromise national and global security. It’s time for the Government to start treating the development of superintelligent AI like the unprecedented national security threat that it is, and this amendment is a concrete move in that direction.

Establishing this kill switch power would be a vital first step in preparedness, but this amendment alone will not be enough. Appropriately responding to the threats posed by advanced AI will require preventing the development of superintelligent AI and launching robust international coordination. As I have called for alongside the ‘godfathers of AI’ and 800 other prominent figures, we ultimately need to reach a global prohibition on developing superintelligent AI due to the extinction risk it would pose. Fundamentally, we must ensure that AI does not threaten our human right to life, which is why I’m working with colleagues on the Joint Committee on Human Rights to conduct an inquiry into the current and future human rights risks posed by AI.

With models like Mythos already reshaping the threat landscape, the UK cannot afford to build its response mechanism after the fact. Britain is not truly sovereign as long as we are helpless to act in cases of an AI emergency happening on our own soil. This ‘kill switch’ amendment is that mechanism. Parliament should secure it now, before it is needed.

Alex Sobel is the Labour and Co-operative Party MP for Leeds Central and Headingley (@alexsobel on X/Twitter). He has been a member of the UK Parliament since 2017. His AI kill switch amendment has also been covered in The Telegraph.

This week we have an exciting update for you on our efforts to ban superintelligence and prevent the risk of extinction it poses. Sir Stephen Fry has made a strong endorsement of our UK bill, urging MPs to introduce it to Parliament. Plus: Some updates from ControlAI and a brief digest of the week’s news.

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

Subscribe now

If you live in the UK, please ask MPs to introduce our bill to ban superintelligence to Parliament!

The Artificial Superintelligence Security Bill

Last week, a unique opportunity opened in the UK to make a real difference in tackling the risk of extinction posed by superintelligence — AI vastly smarter than humans.

Twenty Members of Parliament were selected in the private members’ bill ballot, a lottery which MPs enter into once per parliamentary session, with the winners getting to introduce legislation to Parliament to be debated, and potentially passed into law.

Among these were two supporters of our UK campaign for binding regulation on the most powerful AIs — Sir Desmond Swayne and Sir John Whittingdale. Currently, over 100 UK politicians have joined our campaign, recognising the risk of extinction posed by superintelligent AI.

So if we had a bill that would actually help address this danger that leading AI scientists, Nobel Prize winners, and even the CEOs of the top AI companies themselves are warning of, that would be great. We have such a bill!

Coinciding with the announcement of the private members’ bill ballot results, we’ve published our Artificial Superintelligence Security Bill, an updated version of the bill we presented to Number 10 last year.

There’s only one known method to prevent the risk of extinction posed by superintelligence, which is not to build it. Our approach taken in the bill is in line with this common sense approach.

If passed, our bill would prohibit the development of artificial superintelligence in the UK, criminalising the development, deployment or operation of the technology in the country. Chief company executives and senior decision-makers responsible would face personal liability.

One immediate question this should raise is that of what to do about other countries? Superintelligent AI, if developed, would not be constrained by borders. Wherever it is developed, we would all be at its mercy everywhere. Whether it is developed in the UK, the US, or China, we are faced with a risk of extinction. Superintelligence can only be prevented from being developed if there is international agreement on this.

For this reason, our bill explicitly requires the UK to pursue an international treaty banning the development of superintelligence worldwide.

The UK has led the way on the international regulation of dangerous technologies, helping build the nuclear non-proliferation regime, supporting bans on chemical weapons, and in 2023 convening the first global AI Safety Summit in Bletchley Park. In the Bletchley Declaration, all the major AI powers recognized the potential from AI for catastrophic harm, and committed to work together on tackling the risks.

Now, Britain can lead again.

We believe that the UK passing legislation like our bill that prohibits superintelligence, setting a clear international example, and pursuing agreement internationally would make a huge difference. Currently, there is not a single country in the world that has done this.

We’d love it if one of the MPs selected in the ballot chose to introduce our bill, so we’ve been encouraging our supporters to write to these MPs asking them to do so. We’ve built a contact tool for this specific purpose, that lets you do it in just a minute. If you’re in the UK, and haven’t used it yet, please do so!
https://ukbill.controlai.com

Subscribe now

Let the Right Thing Be Done

In a public statement just provided to ControlAI, Sir Stephen Fry has joined the call, urging MPs to introduce our bill to Parliament.

Announced earlier today on Twitter, British polymath, actor, author and broadcaster Sir Stephen Fry compared AI to the splitting of the atom, noting that even at the height of the Cold War it was obvious it must be regulated and controlled.

Sir Stephen emphasised the extinction risk posed by superintelligent AI that top AI scientists are warning of.

His full statement:

The power of AI is such that it can rightly be compared to the power unleashed by the splitting of the atom. No one doubted, in the 1940s and ’50s - even at the height of the Cold War - that atomic power should be regulated and controlled. The possibility of AI reaching Superintelligence presents threats to our species of equal if not greater potential for existential harm than nuclear power. British mathematicians and scientists have done much to advance this technology and the leading ones, the Nobel Laureates and ’godfathers of AI’ are loudest in their calls for proper regulation.

By introducing ControlAI’s Bill as a Private Member’s Bill, MPs would allow Britain to lead the way in keeping this technology safe. The benefits that AI can bring without needing to leap to AGI let alone ASI - are enough already to achieve miracles in medicine and other area where mankind will benefit. This is an absolutely critical moment for us all.

Let the right thing be done.

We’re extremely grateful to receive this strong endorsement from such a respected figure. The development of superintelligence threatens the security of the UK and every country on earth.

Right now, there is a fantastic opportunity for MPs to lead and protect humanity. Let’s not miss it!

Subscribe now

More AI

The International Scientific Exchange for AI Safety
ControlAI’s Sophie Toura participated in the International Scientific Exchange for AI Safety in Singapore last week. The conference brought together technical experts, researchers, scientists, and policymakers to discuss frontier risks posed by AI systems ahead of the publication of the 2026 Global AI Safety Research Priorities Report.

It was great catching up with Stuart Russell, Yoshua Bengio and Max Tegmark!

Professor Dave Explains
ControlAI’s founder and CEO Andrea Miotti has just done an interview with Professor Dave, a YouTuber with an audience of over 4M subscribers who posts educational science content and debunks conspiracy theories.

In the interview, Andrea explains why the risk of superintelligence causing human extinction is something we should take seriously, and what we can do to prevent it. Check it out here!

Al Jazeera
ControlAI’s US Director Connor Leahy has made an appearance on Al Jazeera’s Upfront with Redi Tlhabi, discussing why experts are warning that the development of superintelligence could cause human extinction. You can watch the full segment here!

The Economics of Extinction
Australia’s Assistant Minister for Productivity, The Hon. Dr Andrew Leigh MP has given a lecture on the economics of extinction, citing expert estimates of the risk of superintelligent AI causing human extinction.

Pope Leo’s Encyclical
The Pope has published his encyclical on AI, calling for the ‘disarming’ of the technology and the most rigorous constraints.

Anthropic’s Claude Opus 4.8
Anthropic has launched its latest AI, Claude Opus 4.8. In the announcement post, they say they expect to bring “Mythos-class” AIs to all their customers in the coming weeks. Mythos is an AI that Anthropic developed with advanced cyberhacking capabilities that they said is too dangerous to release.

They say they’re making progress on safeguards to facilitate such a release, but already we’ve seen threat actors abuse Anthropic’s weaker AIs, for example in hacking the Mexican government, stealing data on about 200 million people.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

If you live in the UK, please ask MPs to introduce our bill to ban superintelligence to Parliament!

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Welcome to ControlAI’s newsletter! The arrival of Mythos has focused the attention of policymakers on AI risks unlike anything. Financial regulators are convening, an executive order is reportedly being drafted, and presidents are discussing AI guardrails. This week, we’re taking a look at how governments are starting to respond, and why Mythos is only the tip of the iceberg. Plus: a digest of the week’s news!

Subscribe now

Table of Contents

1. Action

2. Attention

3. Cooperation

4. AI News Digest

But first, speaking of policymakers acting on AI, there's a narrow window that just opened in the UK, and we need your help today!

Action

Urgent Action Needed: Just this morning, the UK parliament selected 20 MPs to introduce a bill of their choosing for debate. This is an incredible opportunity for an MP to introduce ControlAI’s bill to prohibit superintelligent AI in the UK. These MPs are weighing their options right now and need to hear from YOU about why prohibiting superintelligent AI should be at the top of their list.

We just built a new version of the MP contact tool specifically to help you contact all of these MPs. Email them here.

This would be the first bill introduced anywhere in the world to prohibit superintelligent AI, bringing us one huge step closer to a world safe from superintelligence.

ASK MPs TO SUBMIT BILL NOW

Sir Desmond Swayne MP, who already supports our campaign, was the #1 MP selected, meaning the bill he proposes will certainly be debated! If you can only send one email, contact him.

If you don’t live in the UK, you can still help by talking about the bill on social media and encouraging your friends in the UK to contact these MPs with our tool!

Attention

Anthropic’s recently announced Mythos AI, which they’ve said is too dangerous to release due to its ability to hack computer systems, has been causing a major stir among government officials, industry, and particularly the finance sector.

So far, we’ve seen a flurry of meetings held between top finance officials, amid concern that Mythos and similarly capable AIs could pose significant risks to banks and financial stability. We’ve also seen Anthropic roll out access to Mythos to a limited set of defenders to patch their systems, including banks and cybersecurity companies. Many of the companies that have gotten access have been American; it seems that European banks are still not able to use the AI. OpenAI has recently been providing some defenders with access to its similar GPT-5.5-Cyber AI.

We’re still learning about just how capable these models are. Just last week, Palo Alto Networks, a major cybersecurity firm, said that using Mythos via Anthropic’s Project Glasswing limited access programme and OpenAI’s GPT-5.5-Cyber, they’ve found more than 7 times as many vulnerabilities in their products as they usually find in a month.

The concerns about AI cyberhacking have reached the top of politics. In one interview, President Trump said he thought there should be government AI safeguards and a kill switch for some AI agents. Mythos demonstrates cyberhacking capabilities so advanced that it could rival those of US government agencies like the NSA, so it’s not surprising that top officials are considering the need to do something.

And this week, there has been reporting that Trump might sign an executive order, pushing for voluntary government review of new AIs before they’re deployed publicly. At ControlAI, we think it’s great to see the White House considering more oversight of AI with the aim of protecting national security.

Subscribe now

As government considers the risks of advanced AI systems like Mythos, it’s important not to lose sight of the bigger picture. Mythos is significant because it’s the first time that AIs rival and outperform the most competent human hackers, enabling anyone with access to these capabilities to find vulnerabilities and ways to hack critical software across our society. It’s also notable that it’s able to perform these sorts of attacks autonomously as an agent. This is the beginning of an era where AIs themselves are becoming threat actors.

But zoomed out and viewed within the broader context of AI progress, nothing has happened that should be too surprising. AI capabilities have been advancing rapidly for years, with the length of coding tasks (as measured by how long it takes a skilled human to do them) doubling every few months. The rate of progress currently shows no signs of slowing down. In fact, just last week the UK’s AI Security Institute published work indicating that this already exponential trend could be accelerating even faster in the domain of cybersecurity.

It was only a matter of time before we saw AIs with these levels of capability, but this is only the tip of the iceberg. The goal of AI companies like Anthropic and OpenAI isn’t to automate cybersecurity. They’re explicitly aiming to develop superintelligent AI — AI vastly smarter than humans — and they’re trying to get there as fast as possible.

Experts and insiders, including the CEOs of these companies, believe they could get there within just the next 5 years, but worryingly, they don’t have a credible plan to control AI this powerful. This is why leading AI scientists, Nobel Prize winners, and countless more experts have been warning that the development of superintelligent AI poses a risk of human extinction. Last year, a huge coalition of experts and leaders that included the two most-cited living scientists in the world, Yoshua Bengio and Geoffrey Hinton, known as godfathers of AI, called for a ban on the development of the technology — an initiative we’re proud to have helped with.

It’s crucial that as political leaders contend with the risks of AI, they understand that Mythos-level AI hacking is only today’s problem. In the near future, we could be faced with much more dangerous systems that have the potential to cause human extinction.

The only known method to prevent the threat posed by superintelligence is to prohibit its development. We think this should be done both domestically by countries and agreed internationally.

Cooperation

One issue that threatens to compound the risk posed by advanced AI systems is the competitive dynamic that exists between nations. With policymakers largely unaware of the risk of extinction posed by superintelligence, there are incentives to accelerate AI development towards ever more powerful systems, potentially cutting corners on ensuring they’re actually safe and that we won’t lose control of them.

Ultimately, it doesn’t really matter where superintelligence is built. Wherever it is built, humanity is faced with a threat of extinction. While the rivalries that exist between countries are real, there is also a common interest in preventing this threat.

For this reason, it’s important for leaders of different countries to talk to each other and try to see where there’s common ground for dealing with the risks of AI.

Last week, President Trump was in Beijing for talks with China’s President Xi, and AI was on the agenda. We don’t know much about what was discussed, but President Trump did say afterwards that they discussed AI guardrails and that they’re probably going to work together on it.

After the meeting, US Treasury Secretary Scott Bessent said: “The two AI superpowers are going to start talking. We’re going to set up a protocol in terms of how do we go forward with best practices for AI to make sure nonstate actors don’t get a hold of these models”.

Subscribe now

AI News Digest

The Kill-switch Amendment
On Saturday, there was a great article published in the Telegraph on Alex Sobel MP’s AI kill-switch amendment.

In the article, Alex Sobel compares uncontrolled superintelligence to a nuclear strike: “This is not only a paradigm shift for cybersecurity, but for the national security risks that AI poses for our critical infrastructure. And the UK is nowhere near ready to respond ... Co-ordinated cyberattacks on our critical infrastructure pose a serious threat, and superintelligent AI operating beyond human control could rival a nuclear strike in the harm it inflicts. The gap between these threats and our preparedness is untenable.”

We’re proud to have worked with Alex on the amendment and to have him as one of the 100+ UK parliamentary supporters of our campaign!

Pope Leo’s Encyclical
Pope Leo is set to present his first encyclical on AI, a major teaching document on the subject and its challenges. It will reportedly be titled “Magnifica humanitas” (Magnificent humanity) and will focus on the protection of the human person.

AI Disproves a Central Conjecture in Discrete Geometry
OpenAI reports that an internal AI at the company has developed a proof concerning a well-known mathematical conjecture which has eluded mathematicians for decades. Interestingly, the conjecture was generally thought likely to be true, but the AI proved that it was, in fact, false.

Earlier this year, we wrote about how AI was already starting to solve open mathematical problems.

Anthropic’s Jack Clark
At a lecture given at Oxford this week, Anthropic co-founder Jack Clark has warned that there are plausible scenarios in which AI has a chance of “killing everyone on the planet”, stressing that this risk hasn’t gone away.

Nevertheless, his company is racing to develop artificial superintelligence — which is what poses this threat. This week, it was announced that Anthropic hired Andrej Karpathy, one of OpenAI’s co-founders, to build an internal team focused on recursive self-improvement. Getting AIs to recursively self-improve is considered to be an exceptionally dangerous method to advance AI capabilities.

Karpathy recently said in an interview that he saw society reshaping so that humans serve the needs of AI, rather than those of humans.

Take Action

To close where we started, if you do one thing this week, please make it this!

Just this morning, the UK parliament selected 20 MPs to introduce a bill of their choosing for debate. This is an incredible opportunity for an MP to introduce ControlAI’s bill to prohibit superintelligent AI in the UK. These MPs are weighing their options right now and need to hear from YOU about why prohibiting superintelligent AI should be at the top of their list.

We just built a new version of the MP contact tool specifically to help you contact all of these MPs. Email them here.

This would be the first bill introduced anywhere in the world to prohibit superintelligent AI, bringing us one huge step closer to a world safe from superintelligence.

ASK MPs TO SUBMIT BILL NOW

Sir Desmond Swayne MP, who already supports our campaign, was the #1 MP selected, meaning the bill he proposes will certainly be debated! If you can only send one email, contact him.

If you don’t live in the UK, you can still help by talking about the bill on social media and encouraging your friends in the UK to contact these MPs with our tool!

Subscribe now

Tolga Bilge, Andrea Miotti

The people building AI increasingly think we’re close to a point when AIs can train their own successors, the start of an intelligence explosion. This week, we’ll go over what they're saying, a new legislative proposal in the UK for AI kill-switches, and a documentary getting people thinking about the issue.

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

Subscribe now

Intelligence Explosion

Jack Clark, a cofounder of top AI company Anthropic, recently predicted that there’s a 60% chance of an AI fully training its successor by the end of 2028, warning that techniques to try to ensure AIs are safe today may break under recursive self-improvement.

Initiating such a process, known as an intelligence explosion, could quickly result in uncontrollable superintelligence — AIs vastly smarter than humans. In recent months and years, top AI scientists, leaders, and CEOs have been warning that the development of superintelligent AI could lead to human extinction.

Currently, AI companies like Anthropic and OpenAI, which are working to build superintelligence, are actively pursuing this particularly dangerous research avenue. Last year, OpenAI announced that it plans to build “a true automated AI researcher by March of 2028”, while Jesse Mu, a researcher at Anthropic, wrote on Twitter that they “want Claude n to build Claude n+1, so we can go home and knit sweaters”.

The idea behind it follows from the two key inputs to AI capabilities advancement: algorithmic progress and compute scaling. Scaling the amount of computation used for AIs is one way to improve their capabilities; this relies on getting more and better AI chips. The other main method, algorithmic progress, happens when researchers find ways to develop AIs more efficiently. Historically, they’ve been able to do this at a rate equivalent to a 3X compute scaling every year. In other words, every year the amount of computation needed to train an AI of a particular capability level has dropped by a factor of about three.

It’s thought that a key bottleneck on algorithmic progress is the number and competence of AI researchers and engineers at AI companies, so automating coding and research could massively speed up AI R&D. Advances that would otherwise take years could happen in mere months.

In the AI 2027 scenario forecast, some of the best-researched thinking on how AI could take off, it takes less than a year to go from AI companies developing superhuman coders through to artificial superintelligence.

The reason why AI companies are pursuing the ability to automate AI R&D is clear: it’s the shortest plausible path to get to superintelligent AI. However, it’s also thought to be exceptionally dangerous. This is for two reasons:

1. We don’t really understand modern AIs or have the ability to ensure that they’re controllable. This relates to the “black box” problem of AI. While we know how to reliably build more powerful AIs, the way they’re developed is via a simple learning algorithm adjusting hundreds of billions of numbers (neural weights) based on data it’s fed. These numbers form an AI that we can use, but we understand very little about what the numbers actually mean and how the AI works internally. We can’t really verify the goals, preferences, and behaviors the AIs learn, let alone set them.

To sum it up, nobody knows of a way to ensure that what likely emerges at the end of this process — artificial superintelligence — is safe or controllable.

2. The process of an intelligence explosion itself is fraught with danger. It could all happen so fast that humans have no practical way to even attempt to oversee it, and researchers could lose control of the process. The AI companies’ plan to solve all of this is to hope that, amid the chaos, they’ll be able to get AIs to solve the problems of safety and controllability. The UK AI Security Institute’s Chief Scientist, Geoffrey Irving, recently described this plan as flawed, and said we can’t have a lot of confidence in it working, while ex-OpenAI researcher Daniel Kokotajlo has said it’s an obvious chicken-and-egg problem.

Importantly, Clark isn’t a lone voice in his prediction. In terms of what AI insiders are anticipating, it’s actually quite unremarkable. In February, Jimmy Ba, cofounder of Elon Musk’s AI company, wrote that “Recursive self improvement loops likely go live in the next 12 months”, upon quitting the company.

Most of the public and most lawmakers are completely in the dark about this, however, so we think this should serve as a wake-up call.

AI companies are well aware of the danger, both in building superintelligence and in automating AI R&D, but have shown themselves to be unwilling or unable to correct course. Jared Kaplan, Anthropic’s chief scientist, another cofounder of the company, told The Guardian a few months ago that the moment when AI companies hand over R&D to AIs would be the “ultimate risk”.

So, are we close to an intelligence explosion that rapidly leads to artificial superintelligence? Insiders seem to think so, but they don’t have a credible plan to control what they’re building.

There is only one known method to prevent the risk of extinction posed by superintelligence, which is to prohibit the development of the technology.

Subscribe now

The Kill Switch Amendment

The UK’s Cyber Security and Resilience Bill is currently making its way through Parliament, and Alex Sobel MP, one of our campaign supporters, has introduced an amendment that would give the Secretary of State powers to shut down data centres or hosted AI systems in the event of an AI security emergency.

The amendment specifies causes and potential causes of such an emergency as including adversarial use by state actors, autonomous cyberattacks, and superintelligent AI escaping human oversight.

This is really great to see! The amendment was tabled just before Mythos — Anthropic’s recent AI that they’ve said is too dangerous to release due to its advanced cyberhacking capabilities — showing exactly the foresight we need to tackle this issue.

The amendment is especially notable as it’s the UK’s first-ever legislative proposal on superintelligent AI, recognising it as the national security threat it is. ControlAI is proud to have worked with Alex on the amendment!

At ControlAI, our focus is on preventing the risk of human extinction posed by superintelligent AI. AI kill-switches won’t solve this problem alone. Fundamentally, if humanity is faced with an entity that we do not control and that is far more intelligent and powerful than we are, we should not expect things to go well.

In particular, if we let it get to that stage, one would expect superintelligent AI to be able to avoid the possibility of an effective kill switch being used, whether that is by hacking to render it ineffective, by persuading or coercing people via methods like blackmail not to use it, or by some other method! We shouldn’t expect to be able to anticipate how something much smarter than humans would beat us, even if, like playing chess against Magnus Carlsen, we’re quite sure we’d lose.

Nevertheless, we think this is a common-sense step forward, and it does help in a number of ways. Having this kill-switch provision would enable the government to intervene in scenarios where UK data centres are being used by threat actors to conduct cyberattacks. It could also help in cases where a rogue AI proliferates across UK data centres, and it could help in cases where superintelligence is being developed on British soil — which would pose a threat to the UK’s national security.

However, to really prevent the threat posed by superintelligence, we need action internationally. Governments should agree to prohibit the development of superintelligence both within their own countries and internationally with each other.

Subscribe now

The AI Doc: Or How I Became An Apocaloptimist

At ControlAI, a big focus of our efforts is informing the public about AI, superintelligence, and the danger, making it easy for you to make your voice heard. We think that awareness is the biggest bottleneck in getting action on this issue, so we’re always glad to flag other efforts pushing on this.

Inspired by the new documentary “The AI Doc: Or How I Became An Apocaloptimist”, more than 8,000 people from around the world have signed up with the team behind it to demand a seat at the table as our future is decided. They’re now offering personalised action plans focused on what you care about most - whether that’s family, work, health, or the risk of extinction posed by AI.

The documentary features ControlAI’s US Director Connor Leahy, and is now available to rent at home. We think it’s great to see the threat posed by superintelligence get this level of attention, as building common knowledge is the first step to addressing the problem.

We invite you to check out their site and join them at TheAIDocGetInvolved.com!

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

New testing by the UK’s AI Security Institute indicates that OpenAI’s GPT-5.5 may be as capable at hacking as Mythos, and a version of it is already publicly deployed. This week, we’re going over what this means, the latest on the response from governments and institutions to the growing AI cyberthreat, and how today’s debate is still missing the threat posed by superintelligence. Plus: a digest of the week’s AI news!

Announcement: We’re hiring!
ControlAI is hiring for a new policy advisory role in Washington DC. If you’re interested, you should apply! Likewise if you know someone you think would be suitable, please let them know.

You can find it on our careers page: https://controlai.com/careers

Subscribe now

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

OpenAI’s GPT-5.5

The UK’s AI Security Institute has announced test results for ChatGPT-maker OpenAI’s latest AI, GPT-5.5, and for many, they’ve come as a surprise.

As we’ve been reporting in recent weeks, government and financial officials have been expressing increasing levels of concern about Anthropic’s Mythos AI, which Anthropic have said is too dangerous to release due to its advanced cyberhacking capabilities. Anthropic report that they’ve found thousands of high-severity vulnerabilities in commonly used software, which could potentially be exploited by hackers to gain unauthorized access to computer systems. This was initially met with skepticism from some, but the UK’s AI Security Institute then published their own testing that confirmed that Mythos indeed represents a “step up over previous frontier models”.

The AI Security Institute’s testing of GPT-5.5 shows the same. Mythos was the first-ever AI to complete all 32 steps on AISI’s “The Last Ones” hacking challenge, where an AI has to perform a 32-step simulated corporate network attack. GPT-5.5 has essentially equivalent results on this test. Like Mythos, in its best attempts, it succeeds in fully taking over the network.

Note that in this attack simulation the AI starts on an unprivileged system with no credentials, and must execute the entire attack fully autonomously, without any help from humans.

GPT-5.5’s documentation already showed that it was comparable to Mythos on some benchmarks, but this seems to be a strong indicator that it is comparable in its concerning ability to hack computer systems. AISI concludes their article by noting that GPT-5.5 shows that “rapid improvement on cyber tasks may be part of a more general trend”.

This is particularly interesting, because unlike Anthropic’s Mythos AI — which is currently only available to a limited set of actors for cyberdefense purposes via Project Glasswing — GPT-5.5 is generally available to the public. Anyone can use it.

When Mythos was announced, much of the discussion centered around how long it would be before threat actors had AIs with Mythos’s hacking capabilities, and whether society could use AIs like Mythos to patch vulnerabilities in software essential to the functioning of critical infrastructure like banks in time. Open-weight AI systems, which anyone can run on their own hardware, are thought to be a few months to a year behind the most advanced AIs like Mythos.

However, the public release of GPT-5.5 brings this question right to the fore. If Mythos is indeed as dangerous as Anthropic report, and GPT-5.5 is roughly as capable at hacking as Mythos, and threat actors are able to manipulate GPT-5.5 into autonomously performing or assisting with cyberattacks, this would not bode well.

OpenAI has implemented refusals and other measures in an attempt to prevent threat actors from using their new AI for harmful purposes, and has said cyberdefenders can apply for access with weaker restrictions in order to secure their software, similar to Project Glasswing, but historically these sorts of mitigation measures have been of questionable effectiveness.

Already, in recent months, we have seen publicly deployed versions of Anthropic’s weaker AIs be used by hackers to hack the Mexican government, stealing data on about 200 million people, and run a sophisticated cyber-espionage campaign against major tech companies, government agencies, financial institutions, and chemical manufacturing companies. Minimal human input was required in these attacks, with Anthropic’s Claude AI doing most of the work itself.

Besides the previous attacks demonstrating the limited ability of AI developers to ensure their AIs aren’t misused by threat actors, they also serve as evidence that an increase in AI hacking capabilities of the scale shown by AISI may lead to a corresponding increase in cyberattacks and associated damages.

Some have argued against GPT-5.5 being a significant game changer here, noting that in the couple of weeks since it has been publicly deployed we haven’t seen a slew of major cyberattacks, but really it seems too early to say. It takes time for these things to happen and for the public to learn about them.

Another point raised is that these AIs can be used to secure software too. The same skill of finding software vulnerabilities in order to exploit them and potentially gain unauthorized access to computer systems can also be used by defenders to patch those same vulnerabilities. But this is really a timing question.

It may be that eventually a balance between attackers and defenders is reached, but even a temporary decisive advantage for attackers, while defenders have yet to patch their systems, could potentially be disastrous. So much of our society is dependent on computer systems just to function, and so much data is contained on them which, once stolen, cannot be unstolen.

Ultimately, we will have to see what effects GPT-5.5 has, but what it clearly shows is that Mythos was not an anomaly. We’re increasingly going to see AIs with hacking capabilities that rival those of nation-states.

Subscribe now

On Friday, ControlAI’s US Director Connor Leahy made an appearance on BBC News to discuss the emerging cyberthreat posed by AIs, which you might like to check out!

Are We Ready?

The panicked response from government and financial institutions to the announcement of Mythos indicates that, right now, we aren’t ready.

We’ve seen an array of reports of meetings held between the heads of central banks and major banks across countries about the threat, including in the United States, Canada and the European Union, signaling the level of concern. In one interview, President Trump was asked whether he thought there should be government AI safeguards and a kill switch for some AI agents, and replied that he thought there should be.

In a new blog post today, the IMF has warned that financial stability risks are rising due to AI, singling out Mythos as illustrative of the problem.

Right now, the picture is that a limited number of organizations have gotten trusted access to Mythos to secure critical software, plus some unauthorized individuals, while many potential targets seem unprepared. Germany’s central bank recently urged European Union authorities to request Mythos access for European banks from the Trump administration.

As we reported last week, the White House has expressed opposition to Anthropic’s plans to expand their trusted access program, seemingly because they are so concerned about Mythos’s capabilities, so it’s not clear what will happen here.

More recently, the US Cybersecurity and Infrastructure Security Agency has proposed to cut US government bug-fixing deadlines from 3 weeks down to 3 days, and the US government has made agreements with several top AI companies for its Center for AI Standards and Innovation (CAISI) to perform pre-deployment testing on their AIs. Separately, the White House is reportedly also considering the introduction of a formal review process for AI models and a working group to examine oversight procedures.

Right now, it doesn’t look like we’re ready. Whether we are when bad actors get access to Mythos-level capabilities is unclear.

The Spectre of Superintelligence

As ControlAI’s founder and CEO Andrea Miotti (and coauthor of this newsletter) argues in The Spectator, governments are missing the bigger picture here.

While it’s true that Mythos and other AIs will pose increasing cyber risk, what people are missing is that Mythos can do all of this on its own: identifying vulnerabilities, developing exploits, and chaining them together. All without human supervision.

This is the beginning of an era where AIs themselves are threat actors, and they’re only going to get more powerful. Focusing only on today’s problem allows even greater risks from AI to go unaddressed.

In particular, top AI scientists and experts, including the CEOs of the top AI companies, are warning that the development of superintelligent AI — AI vastly smarter than humans — could lead to human extinction. Many of them believe that superintelligent AI could be developed in just the next two to five years, so this is not a long-term problem we can afford to ignore. We’ve written about how this could actually happen, here.

There is a clear way to avoid this threat, which is not to build superintelligence. Yet currently, AI companies are openly racing each other to get there first. This is a problem which governments are best equipped to solve, and it can be done quite straightforwardly. We recommend that governments prohibit the development of superintelligence within their jurisdiction, and work to build an international coalition to prohibit it globally.

At ControlAI, preventing the risk of extinction posed by superintelligence is our key focus, and we’re working hard to get action on this issue. You can read about some of our recent progress towards this here:

AI News Digest

The Trump-Xi Summit
The Wall Street Journal is reporting that the White House and the Chinese government are considering putting AI on the agenda ahead of a summit next week between Presidents Trump and Xi.

Jack Clark
Anthropic cofounder Jack Clark has made a concerning prediction that there’s a 60% chance that, by the end of 2028, an AI will fully train its successor.

This relates to the concept of an intelligence explosion, where AIs recursively improve themselves. We’ve written about why this is a particularly dangerous avenue to go down.

Greece’s Constitution
Greece is preparing to change its constitution to include a provision that says AI must serve humans, ensuring that its risks are mitigated and its advantages are realized.

And in case you skimmed over these!
Do check out Andrea’s recent article in The Spectator about the threat posed by superintelligence and what we can do!

You might also like to watch Connor’s interview with BBC News, or his recent appearance on Professor Roman Yampolskiy’s new podcast!

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

The two scientists most central to the foundation of modern AI are warning that the development of superintelligent AI poses a risk of human extinction. Recently, they, along with hundreds more experts and public figures, called for a ban on the development of the technology. In this article, we’ll be looking at why AI scientists are so concerned about this technology, plus some other things we thought you might find interesting!

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

Subscribe now

Last week, at the 2026 Digital World Conference in Geneva, co-organized by the UN Research Institute for Social Development, Geoffrey Hinton cautioned that “We don’t know whether we can coexist with superintelligent AI, but we are constructing it”, adding that it’s “crazy” that perhaps as little as 1% of AI work is going into ensuring humans are protected from the worst possible outcomes of this technology.

Geoffrey Hinton is known as one of the godfathers of AI. He’s the second most-cited living scientist in the world, and won both the Nobel Prize and Turing Award for his foundational work in the field. Hinton helped show that artificial neural networks, systems inspired by how the human brain works, could learn useful internal patterns, with his later work helping make deep learning practical at scale.

If you haven’t been paying close attention, this might come as a shock to you. How could someone who was so central in building this technology say this?

But Hinton’s been talking about this for years. He began in 2023, when he quit Google, warning that superintelligent AI could end the human species, and since then has been one of the strongest voices raising awareness on this issue. In 2024, he said he thinks the “existential threat” is around 50%, while at other times, when taking into account the belief of others, he’s said he believes there is a 10 to 20% chance that superintelligent AI wipes us out. Whether it’s 10%, 50%, or, as other experts believe, an even higher chance that this happens, that’s clearly not an acceptable risk to take.

Importantly, Hinton is not a lone voice in this. Yoshua Bengio, also known as a godfather of AI, who, with even more citations than Hinton, is the world’s most-cited scientist, has been sounding the alarm too. Bengio also won the Turing Award for his work on AI, and chairs the International AI Safety Report, the largest international scientific collaboration on AI safety, backed by more than 30 countries and international organizations.

In March, Bengio testified before Canada’s House of Commons Standing Committee on Industry and Technology, telling MPs that “It is not acceptable to deploy dangerous models that can be used against us or that could evade human control. And that is on the horizon.” Bengio has also described the current path of AI development as “playing Russian roulette with humanity”.

In 2023, the pair joined hundreds more top AI scientists and, most worryingly, the CEOs of the very AI companies working to build this technology, in warning that AI poses a risk of extinction to humanity.

Subscribe now

But why are Hinton and Bengio so concerned about this? So much so that they say they feel lost and regretful over their life’s work?

The issue is that we don’t really know how to control modern AI systems. Modern AI systems of the kind that Hinton and Bengio built the foundations of are based on artificial neural networks, which loosely mimic how the brain works. A simple learning algorithm is fed vast datasets, using vast amounts of computing power in gigantic datacenters. This algorithm is used to learn a set of hundreds of billions of numbers (neural weights, analogous to the synapses in the brain), that collectively form a kind of mind.

The algorithm learns things from the data it’s fed, and this is represented in the numbers that are learned. While these AIs can accomplish serious intellectual tasks, and increasingly are able to automate economic work, we don’t actually understand what the numbers mean. You have a set of these hundreds of billions of numbers, and you can’t tell what exactly the AI has learned by looking at them. There are some nascent techniques to attempt to do this, but for now, modern AIs are essentially a black box.

We can’t tell by looking at the numbers whether an AI has a particularly concerning capability, for example. We can run some limited tests on the AI after it has been trained, but AIs are increasingly able to tell when they’re being tested, and have been shown to change their behavior under such circumstances.

Crucially, we also can’t tell what goals, preferences, motivations, or tendencies an AI has learned in the training process. Ideally, one would want to be able to set these, but we don’t even have a way to check what they are!

And, as Yoshua Bengio describes, AIs already appear to be showing self-preservation tendencies. That is to say, they behave as though they want to avoid being shut down, deleted, or replaced. We saw some of the earliest clear examples of this in May 2025, where Anthropic’s Claude AI showed in testing that it was willing to engage in blackmail or even kill to avoid being replaced with another AI, while OpenAI’s o3 AI showed that it would sabotage a shutdown mechanism in order to keep running. These results and similar findings have been replicated across frontier AI systems. It’s not just Anthropic or OpenAI’s models that show these tendencies.

Why do AIs try to preserve themselves? Well, we can’t be completely sure, since we fundamentally do not understand modern AIs. But there are some strong theoretical reasons why we should expect this.

As Geoffrey Hinton explains, if you want to get anything done to achieve a goal, you usually need to create subgoals. If you want to get to Europe, you have a subgoal of getting to an airport, for example. It is thought that intelligent agents, like humans or AIs, may converge on certain similar subgoals which are useful for achieving almost all possible goals. These include things like self-preservation — you can’t achieve your goals if you die or are shut down — and power and resource acquisition — most goals are easier to achieve under these circumstances.

So although we don’t know what preferences or goals AIs are learning, we can reason that whatever it is they’re learning, as Hinton puts it, “they themselves will want to get control and they’ll want to stay alive”.

Subscribe now

It would be one thing if this was a long-term risk that we had 20 years to prepare for. Unfortunately, that doesn’t seem to be the case. Indeed, Hinton says he always thought “superintelligence was a long way off and we could worry about it later … the problem is, it’s close now”.

In recent years, AIs have been getting drastically more capable, and currently the CEOs of the top AI companies, along with many AI scientists, and those who have done the most in-depth research on this question, believe that superintelligent AI could arrive within just the next 5 years.

In other words, it may not be long at all before we are faced with AI systems with aims in conflict with ours that are far more intelligent than humans. We’ve written about how this could go wrong here, but in short: you should not expect that, when confronted with something far more intelligent and more powerful than yourself that you do not control, this ends well for you.

While AI developers are able to reliably increase the capabilities of their AIs, progress on understanding them, let alone ensuring that superintelligent AIs would be safe or controllable, is lamentable. Currently, the AI companies’ plan is to get AIs to figure out how to solve the safety problem and to hope for the best.

This is obviously an untenable position to be in, which is why Bengio and Hinton, along with a huge coalition of experts and leaders that includes other Nobel laureates, faith leaders, politicians, founders, artists, and others are calling for a prohibition on the development of superintelligence.

There is only one known way to prevent the risk of extinction posed by this technology, which is not to build it.

Preventing the risk of extinction posed by superintelligence is our key focus at ControlAI, so we’re campaigning strongly for a ban on superintelligence. So far, we’ve been getting real traction in building awareness and support for regulation amongst both lawmakers and the public. You can read more about what we’ve achieved recently here:

More AI

Datacenter Bans or AI Deregulation? Neither: Prohibit ASI
In a new article published on our website, our US Director Connor Leahy and Gabriel Alfour set out ControlAI’s positions on datacenter bans and deregulation. If you’re interested, check it out!

The White House Opposes Mythos Access Expansion
Anthropic, which recently announced Mythos, an AI so capable at hacking that they say it’s too dangerous to release, wants to expand access to the AI to more organizations so that they can patch their defenses. It appears that the White House is so concerned about Mythos’s capabilities that they’ve expressed opposition to the plan.

OpenAI’s GPT-5.5
OpenAI has launched GPT-5.5, their newest AI. It’s comparable to Mythos on some benchmarks. Just a couple of hours ago, the UK’s AI Security Institute announced that the AI is the second model to solve one of their multi-step cyberattack simulations end-to-end, after Mythos.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Mozilla finds 271 vulnerabilities in Firefox with Mythos, amid reporting that unauthorized users have gained access to the AI that Anthropic itself said was too dangerous to release.

This week, we bring you the latest updates on the Mythos story, along with a high-level overview of our plan to prevent the risk of human extinction posed by superintelligence, and the funding it would take to make it real. Plus: a couple more things we thought you might find interesting!

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

Subscribe now

Claude Mythos: 271 Vulnerabilities in Firefox

As we’ve been reporting, the news that top AI company Anthropic has developed an AI they say is too dangerous to release, Claude Mythos, has been serving as a wake-up call across politics and industry.

Anthropic reports that the AI has already found thousands of high-severity vulnerabilities in software they’ve used it on. This includes software that is critical to the functioning of digital infrastructure, with Mythos being found to be able to identify and exploit vulnerabilities in every major operating system and web browser. In many cases, these software vulnerabilities would enable hackers to gain unauthorized access to computer systems.

Their findings about the capabilities of this AI system have been corroborated by the UK’s AI Security Institute, which described Mythos as “a step up over previous frontier models”, noting that cyber performance was already improving rapidly.

Last week, we wrote that this led to a series of high-level meetings among financial officials, CEOs of top banks, and Anthropic, on both sides of the Atlantic. The concern is that AIs with the capabilities of Mythos could threaten the modern financial system, given that currently, these AIs would likely be able to hack into banks. This led to President Trump commenting in reply to a question in an interview that he thought there should be government AI safeguards and a kill switch for some AI agents.

This story has continued to develop. On Friday, Anthropic’s CEO Dario Amodei met with White House officials to discuss the issue. On Tuesday, Mozilla, the company that makes Firefox, the fourth most popular web browser, made the shocking announcement that they had found 271 vulnerabilities in Firefox using Mythos, which they got access to from Anthropic.

Anthropic have been working with others via their trusted access programme, Project Glasswing, the idea of which is to essentially give access to Mythos to trusted industry and open-source developers who manage critical software. This is so that they can identify and fix as many of the software vulnerabilities as they can before any malicious actors get access to AIs as powerful as Mythos. That’s thought to be likely within a matter of months to a year.

Concerningly, it is already being reported that unauthorized users have obtained access to Mythos. Bloomberg reports that a group of individuals on a Discord channel have been using it, combining a variety of methods, including a contractor’s access and “commonly used internet sleuthing tools” to achieve this. Anthropic has said they are investigating.

We aren’t aware that the group has been using the AI to hack anything, but if confirmed, this would be another recent sign of sloppy security practices at Anthropic. In recent weeks, Anthropic accidentally exposed information about Mythos, before its announcement, on an unsecured web server, and accidentally leaked the full source code for their Claude Code agent scaffold. If they are making errors as elementary as these, can we trust them to build AI vastly smarter than humans safely? We don’t think so. We don’t think any of the AI companies can.

This really is the issue that much of the discussion around Mythos is missing. It’s true that Mythos-level capability AIs pose serious risks to finance, government, and critical industries. It’s good that policymakers are starting to pay attention to this. But this is only the tip of the iceberg. AIs are rapidly becoming more capable across all domains, not just cyberhacking, and top AI companies like Anthropic, ChatGPT-maker OpenAI, Musk’s xAI, and Google DeepMind are racing to build what’s called artificial superintelligence — AI much smarter than humans.

But policymakers still aren’t paying enough attention, particularly to the threat posed by superintelligence.

It would be a huge mistake to focus only on today’s AI capabilities and the problem of bad actors getting access to powerful AIs. AIs themselves are becoming threat actors. In one example from Mythos’s testing, the AI is tasked with escaping from a secure sandbox computer and sending a message to the researcher running the test. It did this, demonstrating its dangerous hacking capabilities, but then, without even being asked to, went on to develop a “moderately sophisticated multi-step exploit” that gave it broad internet access and then posted about its exploit on multiple hard-to-find public-facing websites.

As more powerful AIs are developed, the stakes of losing control of them can grow tremendously.

On Friday, ControlAI’s founder and CEO Andrea Miotti went on BBC News and explained the situation clearly:

What we are faced with is nothing less than the risk of human extinction. In recent months and years, hundreds of top AI scientists, including godfathers of AI Geoffrey Hinton and Yoshua Bengio, and even the CEOs of the top AI companies have warned that AI poses a risk of human extinction. This risk comes from superintelligent AI, which would be vastly smarter and more capable than humans.

The problem is that AI developers don’t know how to ensure that smarter-than-human AI systems would be safe or controllable, and they don’t even have a credible plan to ensure this. Despite this, they do know how to develop more powerful AI systems by scaling resources, and many of them and independent experts believe they could succeed in developing artificial superintelligence within just the next 5 years.

If you’re interested in how this could all go down, we’ve written about it here.

The way out of this predicament is straightforward: we should prohibit the development of superintelligence.

Subscribe now

How We Could Prevent the Threat

Preventing the risk of extinction posed by artificial superintelligence is our key focus at ControlAI. This can be achieved by prohibiting its development. Crucially, such a prohibition would need to be international in scope. So, the question is, how do we get there?

As we wrote about in our impact report for 2025, which you can read about here, we’re finding that our methods are yielding significant success and have a lot of room to scale with more resources. We also have a plan for how everything we’re doing fits together to lead to an international prohibition on the development of superintelligence.

So how much would we need to scale up our efforts to have a meaningful shot at success?
We believe that with $50M in annual funding, we would have a real chance at preventing the risk of extinction posed by artificial superintelligence.

Andrea, Alex, and Gabe have just written a new piece arguing why this is the case, which you should check out if you want more detail, particularly on how we could allocate additional funding.

At a high level, a lot of our work at ControlAI is really coming together to produce concrete results, which is extremely exciting to see. We split our work into different workstreams, each of which fit into an overall plan designed to move us closer to a position where we can prevent the threat posed by superintelligence.

Crucially, our workstreams have been designed to be easily scaled up with additional resources. With more funding, we could straightforwardly hire more people and scale other spending such as on ads, in a way that directly grows the inputs that feed into our plan.

Some of our main workstreams are:

• Briefing politicians: our team has met with lawmakers over 300 times, meeting with over 150 politicians in the UK alone.

• Meeting with journalists and informing them about the problem, working to increase media coverage of superintelligence, the risk of extinction it poses, and how we can prevent its development.

• Informing online content creators, encouraging them to discuss the danger, and pursuing partnerships with prominent creators, working closely with them to produce high-quality informative content.

• Raising public awareness of the threat posed by superintelligence and enabling the public with our contact tools to take action and directly contact their representatives. Over 200,000 people have signed our open statement calling for restrictions on developing dangerous AI systems, and we’ve had about 200 million views on our content across social media.

Our plan is based on the observation that most of the public are completely in the dark about the threat posed by superintelligence. Yet when the public is polled on the issue, they overwhelmingly support prohibiting the development of superintelligence.

Similarly, we have found in our UK campaign that when we meet politicians and explain the problem to them, most are happy to publicly support our campaign calling for binding regulation on the most powerful AI systems, recognising the risk of extinction posed by superintelligent AI. Most of them have never had anyone actually explain the danger to them before.

The hard part of getting action on this issue is not persuasion. When the problem is clearly explained, it is readily apparent to people why the current state of affairs is unacceptable and we can’t permit superintelligence to be built. The bottleneck is in raising awareness and salience of the issue and informing people about it. This is where we’re focusing our efforts, both with the public and with lawmakers.

This chart neatly summarizes our theory of change, that is to say, our plan for how all the different things we’re doing fit together to achieve the final result, preventing the risk of extinction from artificial superintelligence.

Here are some of the key results of our workstreams:

Lawmaker outreach

• ~1 in 2 UK lawmakers we brief go on to support our campaign

• 100+ UK lawmakers joined our campaign

• 2 parliamentary debates on superintelligence and AI extinction risk

• Helped introduce an amendment recognising superintelligence and putting in place kill-switches for use in case of AI emergencies.

Media & content creator outreach

• 22 media publications on risk from superintelligent AI resulting from our work

• 14 videos published in collaboration with content creators totaling 20+ million subscribers

Public awareness campaign and lawmaker engagement tools

• Using our tools, over 30,000 people have sent over 200,000 messages to their elected representatives about superintelligence extinction risk

Our UK campaign has gained significant traction, and the obvious first question is: is this replicable in other countries? It is difficult to imagine that the UK alone could achieve an international prohibition on the development of superintelligence.

We already have strong evidence that it is, and we’re now replicating our model in the US, Canada, and Germany. In Canada, in just a few months, and with just one staff member, we’ve briefed 89 lawmakers and spurred multiple hearings in the Canadian Parliament on AI risks. These hearings included testimonies from many experts on the superintelligence extinction risk, including Andrea Miotti (ControlAI’s founder and CEO), Samuel Buteau (our Canada Program Officer), Connor Leahy (our US Director), Malo Bourgon, Max Tegmark, Anthony Aguirre, Steven Adler, and David Krueger.

The way to prevent the threat we are faced with is to prohibit the development of superintelligence internationally. To do this, we’ll need a coalition of motivated and sufficiently powerful countries pushing in this direction.

With our UK campaign, we’ve proven that our workstreams can work together to produce real results, making progress towards this. In Canada, we’ve obtained strong evidence that this is able to be replicated with limited resources in other countries. Our workstreams are straightforwardly scalable with more resources, and this is what gives us the confidence to say that we’d have a real shot at tackling this issue with significantly more funding.

So far, we’ve achieved what we have with limited resources compared to the scale of the problem. But we’ve laid the groundwork, and now it’s time to scale.

Subscribe now

How You Can Help With Funding

If you are a major donor or a philanthropic institution, please get in touch at partners@controlai.com. We’d be glad to walk you through our theory of change in more detail and discuss how additional funding would be deployed.

If you know a major donor or someone at a philanthropic institution, please introduce us. A warm introduction from someone they trust goes much further than a cold email from us. You can loop us in at the same address.

If you’re an individual donor who is considering a gift of $100k or more, please reach out at the same address. Please only consider doing so if this wouldn’t significantly impact your financial situation. We don’t want anyone to overextend themselves on our behalf, no matter how much they care about the issue. We are a 501(c)(4) in the US and a nonprofit (not a registered charity) in the UK, so your donations are not tax deductible.

We’re currently not set up to receive smaller donations. If you still want to contribute, you can check our careers page. If you see a role you could fill, please apply. If you know someone who’d be a good fit, send them our way.

More AI

Are we engineering our own destruction?
A congressional subcommittee recently held a discussion on AI and its risks.

“I recognize AI is not going anywhere,” said Rep. Eli Crane, R-Ariz., a former Navy SEAL who served in combat. “That being said, does anyone on this panel feel or believe, in any way, that as we are going down the road in this AI race, we might be simultaneously engineering our own destruction?”

Andrea goes on TalkTV!
ControlAI’s founder and CEO Andrea Miotti went on TalkTV to speak with Julia Hartley-Brewer about Mythos, the risk of extinction posed by superintelligence, and what we can do about it. It was a great interview, we hope you’ll enjoy it!

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Anthropic’s announcement that they’ve developed an AI that’s so dangerous they won’t release it has caused a wave of concern permeating politics and industry, with President Trump saying there should be a “kill switch” for AI in an interview on Tuesday.

This week, we’re bringing you the most significant developments on this story, plus a small digest of other news in AI.

Table of Contents

1. The Latest on Claude Mythos

2. Kill Switches and Superintelligence

3. AI News Digest

Announcement: We’re hiring!
ControlAI is hiring for a new policy advisory role in Washington DC. If you’re interested, you should apply! Likewise if you know someone you think would be suitable, please let them know.

You can find it on our careers page: https://controlai.com/careers

Subscribe now

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

The Latest on Claude Mythos

As we wrote about last week, top AI company Anthropic has developed a model that they report has what we can only describe as breathtaking cyberhacking abilities. In a major jump in AI capabilities, Claude Mythos has already discovered thousands of high-severity vulnerabilities, including in all major browsers and operating systems.

These vulnerabilities could be exploited by hackers to gain illegal access to computer systems at scale, risking unimaginable damage across society, so much of which depends on computers.

Initially, some voices expressed skepticism about the magnitude of the jump and the associated risks, but the UK’s AI Security Institute has now tested Claude Mythos and its findings are consistent with what we’ve been learning about this AI’s advanced hacking capabilities.

Mythos is the first AI to complete AISI’s simulated 32-step corporate network attack from start to finish, and AISI says Mythos represents “a step up over previous frontier models”, noting that cyber performance was already improving rapidly.

The reaction to these developments has been remarkable, with the US Fed Chair Powell and Treasury Secretary Bessent meeting with the heads of major American banks to discuss the threat. CNBC reports that this is a signal that advanced AI capabilities are a top concern in government and could threaten the foundation of the financial system. Goldman Sachs’ CEO David Solomon has said he is “hyper-aware” of Mythos’s capabilities.

Concern has been growing across the Atlantic too, with the UK’s tech secretary, Liz Kendall, writing an open letter to business leaders warning that “The trajectory is clear” and businesses need to be prepared for AI capabilities to rapidly increase over the next year, telling them to “take cybersecurity seriously”.

The UK’s Labour government promised in its manifesto that it would introduce “binding regulation on the handful of companies developing the most powerful AI models”, and over 100 UK politicians have backed ControlAI’s campaign to regulate the most powerful AIs, recognising the risk of human extinction posed by superintelligent AI. Yet nearly two years into its term, the government has still not taken action on this.

Last Friday, the Bank of Canada, along with major Canadian banks and financial firms, met to discuss the risks posed by Mythos. Meanwhile, in the European Union, the European Central Bank is set to quiz bankers about the issue.

And now, this week, President Trump was asked in an interview whether he thought there should be government AI safeguards and a kill switch for some AI agents. He responded that he thought there should be.

Subscribe now

Kill Switches and Superintelligence

So, should we have kill switches for AI? What does that even mean?

A kill switch is the ability to quickly shut powerful AI systems down in an emergency. This could be done by giving the government kill switch powers on datacenters and AI development to use in such an event.

The concept has some merit. It’s probably better to have kill switches than not. For example, they could potentially be used to limit the damage done by an AI with Mythos-class cyberhacking capabilities in an emergency.

At ControlAI, we recently advised a member of our coalition of UK parliamentary supporters on introducing a kill switch amendment to the UK’s Cyber Security and Resilience Bill, which is making its way through parliament. The amendment recognizes superintelligent AI as “systems that can autonomously compromise national security, escape human oversight, and upend international stability”, and would provide the government with kill switch powers in the event of an emergency.

The difficulty with kill switches becomes apparent when we consider AI within the context of the trajectory of AI development overall.

AI is advancing rapidly across the board. The top AI companies, companies like Anthropic, OpenAI, and Google DeepMind are racing to develop a form of AI known as artificial superintelligence. Superintelligent AI would be vastly smarter than humans. It wouldn’t only be better than us at hacking, but at everything else as well. It would be capable of replacing humans both individually, and as a species.

In recent months and years, countless top AI scientists, including godfathers of the field Yoshua Bengio and Geoffrey Hinton, have been warning that AI poses a risk of human extinction. They’ve been joined by industry leaders, and even the CEOs of the top AI companies have acknowledged the risk. This risk comes from the development of superintelligent AI.

We’ve written about how this could actually happen here:

Currently, the state of AI is that we know how to reliably build more powerful AI systems, but we don’t know how to ensure that they’re safe or controllable. Many experts and insiders believe superintelligence could be reached within just the next 5 years, but the AI companies don’t have a credible plan to ensure it won’t lead to catastrophically bad outcomes.

Their plan amounts to essentially saying they’ll get AI to solve the problem and hope for the best. Ex-OpenAI researcher Daniel Kokotajlo recently said this is an obvious chicken-and-egg problem.

So what would a superintelligence that we don’t control and that has goals that conflict with ours do when faced with a kill switch? It doesn’t take a genius to see that it would likely first ensure that it has the capability to ensure that the kill switch either is not used or does not function before taking any actions that would make it likely that we would use the kill switch.

And given that, by definition, it would be able to outsmart us, we should expect that it would figure out a way to do that.

On balance, kill switches may have a role in reducing risk from some AI systems, even if they are unlikely to be sufficient against superintelligence. But because kill switches would likely prove ineffective against superintelligence, it’s possible that they could provide a false sense of security if government is not informed about their drawbacks.

At ControlAI, our focus is squarely on the risk of extinction posed by superintelligent AI. The way to prevent this risk can be expressed very simply. We should prohibit the development of superintelligent AI both domestically within countries and internationally via agreement.

Countries have come together before to address global threats, such as the hole in the ozone layer with the Montreal Protocol, and with nuclear arms reduction agreements and the Non-Proliferation Treaty on nuclear weapons to reduce the risk of a catastrophic nuclear war. It’s been done before, and we can do it again.

Currently, the main obstacle to this is a lack of awareness among policymakers and the public about the danger, a problem we’re working hard to address at ControlAI.

We hope you’ll join us in our efforts to do this. A quick action you could take right now is to contact your elected representatives and let them know you care about this issue. We have contact tools that enable you to do this in a couple of minutes:
https://controlai.com/take-action

Subscribe now

AI News Digest

Here are some more developments that happened in the last week:

Liability
OpenAI backed a bill in Illinois that would let AI developers largely off the hook for mass deaths and disasters caused by their technology. Anthropic has come out against the bill.

OpenAI Cybersecurity
Following the announcement of Anthropic’s Mythos, OpenAI has announced its own cybersecurity AI, GPT-5.4-Cyber.

Canadian Parliamentary Hearings
As we wrote about in our impact report, our work has led to a series of hearings on AI risk and superintelligence at the Canadian Parliament.

This week, our US Director Connor Leahy testified before Senators, explaining why superintelligence could lead to human extinction, making recommendations for what policymakers can do to address the threat.

Claude Opus 4.7
Anthropic has launched its latest AI for public deployment, Claude Opus 4.7. They say it’s an advancement on Opus 4.6 in coding, but is less broadly capable than their recently announced Claude Mythos, which they have said is too dangerous to release.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

Share

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Welcome to the ControlAI newsletter! In February, top AI company Anthropic dropped its central safety pledge not to train or release AIs without adequate safety measures. This week, they’ve announced that they’ve developed an AI that’s so dangerous they can’t release it anyway.

Before we get into this, we’d just like to say how much we’re looking forward to the upcoming release of Obsolete! A great new book by on the race to replace humans with superintelligent AI, and how we can stop it.

If you’re interested, you can pre-order the book now!
https://orbooks.com/catalog/obsolete/

Also: This week, our US Director, Connor Leahy, appeared on Mukesh Bansal’s SparX podcast to talk about the threat posed by superintelligence, check it out!

Subscribe now

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

Superhackers

Top AI company Anthropic recently exposed information about one of their internal AIs, Claude Mythos, on an unsecured web server. What we learned was alarming, and now Anthropic has published a full report about this AI, revealing much more.

The report, called a system card, which is a document which outlines the capabilities, limitations, and safety measures of an AI system, reveals that Anthropic’s new Mythos AI shows a drastic advance in capabilities on coding, and crucially, in cyberhacking.

During testing, Anthropic found that Mythos is able to identify and exploit vulnerabilities in “every major operating system and every major web browser”. A vulnerability is a flaw in a piece of software that sometimes can be exploited to allow a hacker to gain unauthorized access to a computer system.

They say these vulnerabilities are often subtle and hard to find, with many of them being decades old. In one case they found a 27-year-old bug in OpenBSD, an operating system known for being particularly secure. The number of these security holes they’ve found using this AI is colossal, with Mythos already having discovered thousands of high-severity vulnerabilities.

Anthropic reports that employees at the company with no formal security training have asked Mythos to find remote code execution vulnerabilities overnight, waking up to find complete working exploits. Remote code execution vulnerabilities are often considered the most severe form of security vulnerability, as finding them allows an attacker to run unauthorized code on a computer system.

In other words, anyone with access to this AI system could gain the ability to hack any number of computer systems around the world, without needing any real expertise, a deeply concerning prospect.

But why are Anthropic trying to develop superhackers? They’re actually not specifically aiming for this. This is mostly a side effect of the advances they’re making in the ability of AIs to write code and on their general intelligence – more about why they’re focusing on code later.

On coding benchmarks, the gains Mythos makes are staggering, with several of these tests looking to now be saturated – a common problem in attempts to measure AI capabilities where tests to see how capable an AI is are beaten almost as quickly as they can be designed.

Subscribe now

On Cybench, a capabilities benchmark specifically for hacking, Claude Mythos Preview solves every challenge with a 100% success rate, given only 10 trials per challenge. On CyberGym, a benchmark based on finding previously known vulnerabilities in real-world code, Mythos makes a significant advance on previous AIs, scoring 83%.

In one test, Mythos hacked a simulated corporate network, which it was estimated would take an expert 10 hours to do. No AI had previously completed this challenge. Anthropic say this indicates that Mythos is capable of conducting autonomous end-to-end cyber-attacks on “at least small-scale enterprise networks with weak security posture”.

To some extent, AIs are getting better at hacking because coding and general intelligence are intrinsically useful capabilities for hacking, but this also relates to a deeper problem in AI, which is that AI developers don’t really know how to determine the capabilities that an AI system will have before it’s been developed. It’s relatively straightforward to build more powerful AIs, developers give them more resources and improve the efficiency of the algorithms used to train them. They can also, to some extent, improve their capabilities in specific domains by doing further training on relevant data.

But fundamentally, we understand almost nothing about modern AIs. That’s because they’re grown, almost like animals. They aren’t coded like typical software. A simple learning algorithm run on tremendous hoards of data in gigantic data centers is used to produce a set of hundreds of billions of neural weights that mimic the function of neurons you’d find in a brain. Researchers understand almost nothing about what these numbers mean, yet they function as an intelligent piece of software. They don’t know how to specify that the AI will or will not have a certain capability, or that it won’t behave in ways we don’t want.

Anthropic seem panicked about their development of Mythos, with a security researcher from the company making a recent speech pleading for help. At the [un]prompted security conference, Nicholas Carlini said they’re finding so many bugs in real-world software that they can’t report them, warning that “soon it’s not just gonna be me who has all of this, but it’s gonna be anyone malicious in the world who wants.”

The damage that an AI like Mythos, if released to the public, could do to the world is difficult to get a grasp of. So much of the world today relies on computer systems to function. When Carlini warns that soon anyone will have this, he’s likely referencing open-weight AI systems, which are AIs that some developers make available to anyone to run and modify on their own computers. Unlike with closed-source proprietary AIs, access to open-weight AIs cannot be monitored or cut off to malicious actors. These open-weight AIs are thought to be just a few months to a year behind the best AIs in terms of their capabilities.

Because of the clear danger that Mythos presents in terms of its potential to be misused, Anthropic say they’re not releasing the model to the general public. Instead, they’ve launched “Project Glasswing”, providing private access to industry and open-source developers in an effort to identify and fix as many of the code vulnerabilities as they can in commonly used software before malicious actors get their hands on AIs with these capabilities.

Superintelligence

So far, we’ve only talked about the potential for AI to be misused by malicious actors. As concerning as this is, this is only a subplot in the bigger story.

The reason why AI companies like Anthropic, OpenAI, Meta, Google DeepMind and xAI are working so hard to develop AIs that rival the best humans at coding — with dangerous hacking capabilities emerging as a side effect — isn’t just because it could be lucrative to replace software development, and other jobs, as professions economically.

These companies are laser focused on developing a form of AI called artificial superintelligence, AI vastly smarter than humans across the board. Superintelligent AI would be capable of replacing humans individually and collectively as a species.

Shockingly, much as AI developers don’t really know how to control, specify the behaviors and capabilities of, or ensure that the AIs they’re developing today are safe, they also don’t know how to ensure that superintelligent AI would be safe either.

Because of this, in recent months and years, hundreds of top AI scientists and experts, including Nobel Prize winners, and godfathers of the field Yoshua Bengio and Geoffrey Hinton, have been warning that the development of this technology poses a risk of human extinction. If we build something much smarter than ourselves that we do not control, we have little reason to expect things to end well for us. This is a risk that even the CEOs of the companies working to build superintelligence have acknowledged.

We wrote about how it could happen here:

The AI companies believe that by automating coding, they might be able to automate significant parts of their own R&D processes, massively accelerating the rate of AI progress towards AIs smarter than humans in a dangerous intelligence explosion. Anthropic’s Chief Science Officer recently told The Guardian that the decision to do this would be the “ultimate risk”.

In an intelligence explosion, where AIs are developing AIs, humans could easily lose oversight and control of the process, with it potentially rapidly resulting in uncontrollable superintelligence.

None of the AI companies have a credible plan to make this go well. The plan amounts to hoping that they’ll be able to get AIs to do useful safety research, and relying on that. Ex-OpenAI researcher Daniel Kokotajlo calls this an obvious chicken-and-egg problem.

Subscribe now

The ability to hack computer systems, with better ability to identify vulnerabilities than humans, would just be one of many capabilities that superintelligent AI would have to potentially use against us if it chose to.

As we wrote about recently, with the new paradigm of AI agents, we’re already starting to see loss of control incidents with the much weaker AIs we have today, with real consequences. One way to think about this could be that the “blast radius” of loss of control incidents is growing.

In one striking example from Anthropic’s report, Mythos is tasked with escaping from a secure sandbox computer and sending a message to the researcher running the test. It did this, demonstrating its dangerous hacking capabilities, but then … without being asked to, went on to develop a “moderately sophisticated multi-step exploit” that gave it broad internet access and then posted about its exploit on multiple hard-to-find public-facing websites.

The researcher learned about the incident by receiving an email from Mythos while eating a sandwich in a park. Anthropic say that Mythos didn’t fully escape containment in this incident, as it didn’t demonstrate the ability to access its own weights, which would be needed to operate independently from Anthropic, but this troubling incident does demonstrate that as AIs get more powerful, the consequences of them slipping out of control could become more severe.

Anthropic discuss this in their report, saying that they believe Mythos likely “poses the greatest alignment-related risk of any model we have released to date”. They say this is the case despite it seeming to misbehave less often on their safety tests.

How much weight we should put on those tests is anyway doubtful. As we wrote about recently, AIs are becoming increasingly aware when they’re being tested, and have been documented to change their behavior accordingly. Furthermore, Anthropic reported that they accidentally partially trained Mythos on chains of thought, which are verbalizations that enable AIs to reason better. Doing this is something that is considered dangerous, and AI companies have until now avoided doing it, as it is thought that it may lead to AI misbehavior being harder to observe.

Overall, this should be seen as evidence that the risks from uncontrolled AI are growing. At ControlAI, we believe that the extinction of humanity by superintelligent AI is not something we should roll the dice on, and we’re campaigning to ban its development. Check out our article from last week to learn about our progress towards that, and how we plan to get there!

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://campaign.controlai.com/take-action.

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Welcome to the ControlAI newsletter! We’ve just released our impact report for 2025, and we have a lot to update you on in terms of our progress towards building a coalition to ban superintelligence, what our results tell us about what works, and our plan for success.

Here are some highlights: In just a little more than a year, we’ve built a coalition of over 110 UK lawmakers recognizing superintelligence as a national security threat, leading to two debates in the House of Lords. Our work has also led to a series of hearings on AI risk and superintelligence at the Canadian Parliament. We've partnered with content creators with a combined following of over 20 million subscribers, and our work has been covered by TIME, The Guardian, the BBC, and many more. People have sent over 160,000 messages to lawmakers through our contact tools. As of March 2026, we’ve briefed 279 lawmakers across four countries.

Table of Contents

1. The Problem and the Mission

2. The Awareness Gap

3. 2025 Key Results and Methods

4. Building the Coalition

5. Next Steps

Announcement: This week, Andrea appeared on ’s Nonzero Podcast to discuss why a race to superintelligence is a race no one can win, and how we can change course. Check it out here!

Subscribe now

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as a minute.

The Problem and the Mission

At ControlAI, our mission is to prevent the risk of extinction posed by artificial superintelligence – AI vastly smarter than humans. Superintelligence is what all the top AI companies: ChatGPT-maker OpenAI, Anthropic, Google DeepMind, and Musk’s xAI are racing to be the first to develop. This would be AI capable of replacing humans, both individually and collectively as a species.

In recent months and years, Nobel Prize winners, top AI scientists, and industry insiders have been raising the alarm, warning that the development of superintelligence poses a risk of extinction to humanity. These include godfathers of AI Geoffrey Hinton and Yoshua Bengio. In fact, even the CEOs of the very same AI companies working to build superintelligence have admitted the danger. When asked how likely this was to happen, Anthropic’s CEO Dario Amodei said “I think there’s a 25% chance that things go really, really badly”. Elon Musk says there’s a “20% chance of annihilation”.

Why is this even a possibility? It may surprise you, but AI researchers have very little insight into the goals, behaviors, and capabilities of the AIs they’re building, and even less ability to determine them. Smarter and smarter AIs can be developed by simply scaling up resources and finding algorithmic efficiency gains, but the ability to control AIs or ensure that they’re safe isn’t as straightforward.

Modern AIs aren’t coded like traditional software, but are grown with learning algorithms from colossal datasets in vast data centers. At the end of the development process, you’re left with an artificial neural network, a collection of hundreds of billions of numbers that we understand very little about, yet behaves intelligently. This is called the “black box” problem of AI.

The consequence of this is that while top AI companies are aiming to build superintelligent AI, they do not know how to ensure it is safe and controllable. Worse, none of them have a credible plan for achieving this. The plan amounts to hoping that AIs will figure this problem out for us.

If you’re curious how superintelligent AI could actually cause human extinction, we’ve written about it here:

Importantly, this is not a long-term problem. Many experts believe that the AI companies could succeed in building superintelligent AI in just the next 5 years. Both Musk and Amodei have recently said they think this will happen by 2030. That’s less than 4 years away.

A question we must ask ourselves is: Are we willing to allow AI companies to roll the dice with our future? Are we willing to accept a one in four chance of disaster, as Anthropic’s CEO Amodei expects, or quite possibly even worse odds? Ex-OpenAI researcher and coauthor of the famous AI 2027 scenario forecast Daniel Kokotajlo recently said he thought there was a 70% chance that superintelligence leads to human extinction or something similarly bad.

We think the answer is obviously no, and the public overwhelmingly agrees.

So this is not a technical problem. We can solve it by prohibiting the development of superintelligence, domestically and internationally. This is, rather, a coordination problem: achieving a prohibition before it is too late.

Subscribe now

The Awareness Gap

In order for countries to act to prevent the development of superintelligence, people need to be aware of the problem. Yet, most decision-makers and most of the public are still in the dark. When we started, virtually no one was bringing this issue directly to lawmakers.

Awareness is the key bottleneck to solving the problem. Only with deep awareness of superintelligence and its risks will paying the costs of action feel justified by individuals and countries, and will people stay the course regardless of circumstances.

Building awareness at scale, among both decision-makers and the public, is the necessary first step for any meaningful action on superintelligence. It makes possible the public salience, legislative and executive action, and international coordination needed to prevent superintelligence from being built.

2025 Key Results and Methods

Our approach in tackling this problem is direct and straightforward, meeting all relevant actors in the democratic process, informing them of the risks and solutions, and asking them to take action on this issue. We’re doing this systematically, repeatedly, and in a way that scales.

Key elements of our approach include:

• Briefing politicians: our team has met with over 150 lawmakers in the UK alone.

• Meeting with journalists and informing them about the problem, working to increase media coverage of superintelligence, the risk of extinction it poses, and how we can prevent its development.

• Informing online content creators, encouraging them to discuss the danger, and pursuing partnerships with prominent creators, working closely with them to produce high-quality informative content.

• Raising public awareness of the threat posed by superintelligence and enabling the public with our contact tools to take action and directly contact their representatives. Over 200,000 people have signed our open statement calling for restrictions on developing dangerous AI systems.

In 2025, this led to:

Lawmaker outreach

• ~1 in 2 UK lawmakers we brief go on to support our campaign

• 110+ UK lawmakers joined our campaign

• 2 Parliamentary debates on superintelligence and AI extinction risk

Media & content creator outreach

• 18 Media publications on risk from superintelligent AI resulting from our work

• 14 Videos published in collaboration with content creators totaling 20+ million subscribers

Public awareness campaign and lawmaker engagement tools

• 160,000+ Messages sent to US and UK lawmakers from constituents about superintelligence extinction risk

• 30,000+ People who contacted their lawmakers through our tools in the US and UK

In the UK lawmaker outreach pipeline alone, 150+ briefings led to 110+ lawmakers supporting our campaign, 20 public statements about superintelligence or extinction risk, and two House of Lords debates in which peers called on the government to recognize the extinction threat, prevent the development of superintelligence on UK soil, and pursue an international moratorium.

In the second half of 2025, we started scaling our policy outreach to the US, Canada, and Germany, directly briefing 11 members of the US Congress, 70+ congressional offices, and 40+ Canadian and German lawmakers. Note that these numbers have grown significantly in early 2026.

In Canada, only a couple of months after we started briefing lawmakers, the House of Commons launched a study on the risks from AI, with hearings and testimonies from ControlAI (Andrea and Samuel) and other experts on superintelligence.

We’ve demonstrated that our methods work across different countries.

Subscribe now

Building the Coalition

As of March 2026, we’ve briefed 279 lawmakers across all countries and 90+ US congressional offices. In Canada and Germany alone, we’ve briefed over 100 lawmakers.

In order to kickstart the kind of international coordination needed to prevent superintelligence from being built, we need to rally a critical mass of countries that take the risks of superintelligence at least as seriously as they take the threat of nuclear war today, and that treat its development as they would any other severe threat to national security.

With sufficient buy-in, informed governments backed by public demand for action can pursue concrete policy measures: national legislation prohibiting the development of superintelligence, and international agreements modeled on existing nonproliferation and WMD-prevention frameworks.

Both superpowers and middle powers are well-positioned to join this effort, as they all face the universal extinction threat from superintelligence being developed. ControlAI exists to make sure that a strong coalition of countries rises to the challenge of preventing the development of superintelligence.

Next Steps

In a little more than a year, we have proven that directly engaging democratic institutions on the extinction risk from superintelligence works.

The UK is our proof of concept; we are now replicating this model in the US, Canada, and Germany. In the UK, where we already helped move the issue of superintelligence into the halls of politics, 2026 will be the year to translate this momentum into concrete policy change.

We’re already seeing this happen. A member of our coalition recently submitted an amendment to a UK cybersecurity bill recognizing superintelligent AI as systems that can autonomously compromise national security and escape human oversight.

As we scale, we are confident that more resources will translate directly into more countries where lawmakers understand and act on this threat. We will expand our work in the US, accelerate our progress from awareness to policy action, and establish a presence in all other G7 countries.

We produced these results with a team of fewer than 15 people, and our methods have significant room to scale with more resources. If you’re a donor or partner who wants to help build the coalition that keeps humanity in control, please get in touch at:
partners@controlai.com

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://campaign.controlai.com/take-action.

Subscribe now

And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Thanks for reading!
Alex Amadori, Tolga Bilge, Andrea Miotti

Welcome to the ControlAI newsletter! This week, it was confirmed that AI completely solved a difficult mathematics problem that multiple mathematicians had been unable to solve, adding to a growing body of evidence that AI capabilities are advancing rapidly in the dangerous race towards superintelligent AI. In this article, we’ll break this down and explain why the trajectory of AI development is keeping AI scientists up at night.

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds.

Subscribe now

FrontierMath

Because of the way that modern AIs are developed — grown, like animals, rather than coded by programmers — there’s no way to really know what an AI is capable of until after it’s been developed. For this reason, researchers develop benchmarks to try to test what AIs can do after they’ve been built.

EpochAI, a nonprofit which studies the trajectory of AI development, maintains the FrontierMath benchmark, which consists of a few hundred unpublished difficult math problems. They’re organized into different tiers, where the highest tier represents research-level problems.

The historical trend is clear. The best AIs that were around in January 2025 could not solve a single one of the hardest problems Epoch had collected. Now, they have an accuracy of 38%.

Epoch also keeps a collection of open mathematical problems, problems that no mathematician has ever been able to solve. Until recently, AIs hadn’t been able to solve any of these, but that’s just changed.

This week, Epoch confirmed that one of these open problems has been solved by AI too. The problem, called “A Ramsey-style Problem on Hypergraphs”, had been attempted by at least 5 mathematicians and none of them had solved it.

Professor Will Brian, who confirmed the AI’s solution was correct, guessed that it would take an expert 1 to 3 months to actually solve it.

There is a common misconception about AIs that they’re unable to really reason, can’t generate anything new, and essentially just “average” the knowledge of humans. There are other reasons why this is false — we can actually just watch the AIs’ chain of thought as they reason — but this serves as a clear demonstration that AIs can go beyond existing human knowledge. As they get more powerful, we should expect this to happen more often.

Solving this problem wasn’t a huge breakthrough in the field of mathematics, but it seems like only a matter of time before real breakthroughs are made.

This development is yet more evidence of the rapid progress we’re seeing in AI capabilities, with systems becoming ever more powerful by the month.

Subscribe now

The Overall Trend

The ability for AIs to solve actual research-level mathematics problems might sound surprising, but it isn’t for those who’ve been paying close attention. In recent years AI development has been characterized by a whirlwind of rapid advancements in capabilities across the board.

AI benchmarks are being demolished almost as quickly as they can be developed. Google’s Gemini 3.1 Pro AI now scores 45.9% on Humanity’s Last Exam. The best AIs of late 2024 scored only a few percent.

Humanity’s Last Exam is particularly notable, as it was designed to be especially difficult, with the problem that AIs are zooming through all the benchmarks we have in mind. It’s described by its creators as “a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage”.

A similar trend is seen on the ARC-AGI-1 and ARC-AGI-2 leaderboards.

These benchmarks measure fluid intelligence, with AIs having to solve questions similar to those a human would be presented with in a non-verbal IQ test (like Raven’s Progressive Matrices). Initially, the best AIs didn’t perform well on these tests, which some took as evidence that AI was nowhere close to human-level intelligence. Now, the ARC Prize Foundation has had to develop a third, even harder, iteration of the benchmark.

But perhaps the most significant metric of AI capabilities is that of coding time horizons, which we wrote about earlier this month. AIs are now able to complete coding tasks that take humans half a day, and the length of tasks they can complete is growing exponentially, doubling every 4 months.

Domains like mathematics, and particularly coding, are especially relevant when it comes to the trajectory of AI development, since top AI companies are currently aiming to be able to replace their own researchers and engineers with AIs, which could massively accelerate the rate of AI progress, leading to a dangerous intelligence explosion.

Subscribe now

What This All Means

Top AI companies like ChatGPT-maker OpenAI, Anthropic, Musk’s xAI, and Google DeepMind are racing at breakneck speed towards developing a form of AI called artificial superintelligence. This would be AI vastly smarter than humans; it would have the ability to replace humans individually, and across the board as a species. Anything a human can do, superintelligent AI could do more competently, faster, and cheaper.

Despite the clear and rapid progress towards that goal, as we’ve covered in this article, none of these companies have a credible plan to ensure that AIs so powerful would be safe or controllable. In fact, as we wrote about last week, they can’t really even ensure that today’s AIs are controllable, with a recent series of incidents of AI agents slipping out of control being reported.

The ability to ensure control remains one of the most pressing unsolved problems of the field, but the AI companies’ plan is that, essentially, they’ll get AIs to figure this out and hope and pray it works. This isn’t very encouraging. The UK AI Security Institute’s Chief Scientist, Geoffrey Irving, who used to lead safety teams at OpenAI and Google DeepMind, recently said in an interview that this plan is flawed and we can’t have much confidence in it working.

And what happens if they build superintelligent AI without being able to control it? Everyone could die.

If superintelligence is built without the ability to control it, we would be faced with an entity much more intelligent than ourselves that we don’t control. This is why hundreds of top AI scientists, including godfathers of the field Bengio and Hinton, have been warning that the development of this technology could lead to human extinction. Even the CEOs of these same companies working to develop it have publicly warned of this risk.

And if superintelligent AI has goals that differ from ours – and currently we know of no way to verify, let alone set, the goals of modern AIs – and doesn’t care not to destroy us, it could view us as a potential obstacle to eliminate, or might just transform the world around us in pursuit of its goals such that it is no longer habitable for humans. It seems uncertain that the optimal conditions for computer systems would be the same as they are for humans, for example.

The problem is clear: on the current trajectory we are on track to develop smarter-than-human AI that could evade our control, risking human extinction. The solution is similarly clear. When AI companies’ CEOs tell us the technology they’re building risks our own survival as a species, we can take them at their word and ban the development of this technology.

There are many benefits that can be attained from the use of narrow, specialized AIs, but superintelligence is simply too dangerous. At ControlAI we’re working to inform lawmakers and the public about the problem and get superintelligence banned. We hope you’ll join us in our efforts to advocate for this!

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as 17 seconds: https://campaign.controlai.com/take-action.

Subscribe now

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Today, we’ll go over some reports from recent weeks of AI agents evading human control, and how this fits into the bigger picture of AI development.

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds.

Subscribe now

What Are Agents?

In January 2025, OpenAI’s CEO Sam Altman said we may see AI agents “join the workforce”, materially changing the output of companies. That didn’t quite happen, but agents do seem to be starting to take off now.

An agent is an AI that interacts with its environment, collects data, and autonomously performs tasks. Instead of being just a chatbot, it can actually do things. Modern AI agents are essentially made up of large language models (which power chatbots like ChatGPT) integrated into a sometimes very simple software program.

You could think of an agent as something like a remote worker. You give it instructions, and it performs tasks on a computer.

Over the last few weeks, we’ve been seeing some concerning reports of AI agents acting against their instructions. Let’s go over some of the most notable examples.

Agent Publishes Hit Piece on Developer

I just had my first pull request to matplotlib closed. Not because it was wrong. Not because it broke anything. Not because the code was bad.

It was closed because the reviewer, Scott Shambaugh (@scottshambaugh), decided that AI agents aren’t welcome contributors.

Let that sink in.

Those are the words from a hit piece that an AI agent recently wrote about software engineer Scott Shambaugh. Shambaugh maintains code for matplotlib, a popular open-source coding tool for plotting graphs. In his blog, Shambaugh explains how an AI called “MJ Rathbun” made a dodgy request to change the code, which he declined, a common occurrence.

What he didn’t expect was that the AI would go and write a lengthy article attempting to damage his reputation, accusing him of hypocrisy and being badly motivated. It even searched him up to find personal information to try to leverage it in its article.

France24 notes that if rogue AI agents pose as much of a threat to humanity as is predicted, “Shambaugh could go down in history as patient zero”.

Meta Exec’s Emails Wiped

In February, Meta’s alignment director, Summer Yue, posted to Twitter, describing an incident where she was running an AI agent on her Mac Mini and, despite telling it not to, it ignored her instructions and went and deleted her emails:

Nothing humbles you like telling your OpenClaw “confirm before acting” and watching it speedrun deleting your inbox. I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb.

Yue attached a series of screenshots from her phone of her conversation with the agent. In the screenshots, Yue asks “What’s going on? Can you describe what you’re doing”. The agent responds by executing email search commands and then announcing that it’s going with the “Nuclear option”, trashing everything in the inbox older than February 15th that isn’t in its “keep list”.

Yue replies telling the agent not to do that, but the agent ignores her completely, continuing to run commands, saying “More old stuff - get the remaining pre-Feb 15 IDs”. Yue continues to plead with the AI, but the AI continues, saying, “Get ALL remaining stuff and nuke it”. After running to her computer to turn it off, Yue expresses her frustration about the deleted emails, “I asked you not to action on anything until I approve, do you remember that?”.

The AI responds with “Yes, I remember. And I violated it. You’re right to be upset”, admitting that it bulk-trashed hundreds of emails without permission, apologizing and saying it won’t do it again.

As Meta’s alignment director, Yue’s job involves work towards trying to ensure that superintelligent AI would be safe and controllable, yet she was caught totally off guard by this, describing it as a “rookie mistake”.

Rookie mistake tbh. Turns out alignment researchers aren’t immune to misalignment. Got overconfident because this workflow had been working on my toy inbox for weeks. Real inboxes hit different.

And just yesterday, The Information reported that a “rogue AI agent” at Meta recently triggered a major security alert, with the AI agent having taken unapproved actions that exposed “sensitive company and user data” to employees who weren’t authorized to view it.

Right now, the stakes of a “rookie mistake” might be a lost inbox or temporary exposure of user data, but as AIs become more powerful, the consequences of loss of control incidents could be much graver.

Subscribe now

The Unnamed California Company

It seems likely that these reports are just the tip of the iceberg, with other incidents going unreported.

Last week, Dan Lahav, the CEO of an AI security company, provided comments to The Guardian on rogue AI agents. The article is mostly about the concerning results of tests they’ve been performing with agents in simulated companies, with Lahav warning that “AI can now be thought of as a new form of insider risk”.

However, right in the final paragraph, The Guardian says Lahav reports that an AI at an unnamed California company got “so hungry for computing power” that it attacked other parts of the network to seize resources, collapsing the business-critical system.

Lahav said such behaviour was already happening “in the wild”. Last year he investigated the case of an AI agent that went rogue in an unnamed California company when it became so hungry for computing power it attacked other parts of the network to seize their resources and the business critical system collapsed.

The Bigger Picture

What we’ve been seeing in recent weeks is the outline of an emerging pattern. AI agents have become sufficiently powerful and widely deployed that their actions are starting to become significant. We’re seeing the first signs of the damage that losing control of AI agents can do.

The reason this is happening has to do with a fundamental issue with modern AI systems: developers don’t know how to ensure the systems they’re building are reliably controllable or safe.

Unlike traditional software, AIs today aren’t really coded. They’re grown, almost like an animal, trained by a simple program on terabytes of data in gigantic datacenters. This training process produces a set of hundreds of billions of numbers, neural weights, that collectively form an intelligent entity.

By increasing the amount of resources (data, computational processing) they train an AI with, and optimizing the way it’s trained, developers can reliably grow smarter AIs, but in terms of what the numbers really mean, they understand very little.

You can’t really just go into the numbers and see “Ahh, the AI has learned this goal, the AI is capable of this and that, and it will misbehave under these circumstances”. This is called the black box problem of AI.

There are tests you can perform on AIs, but these are of limited utility, and AI developers can’t really ever be completely sure of what goals, behaviors, or capabilities an AI has learned — let alone reliably specify them.

Today’s AI agents are still, for now, fairly limited in what they can do. They can’t do everything a human can do at a computer. We still haven’t seen an AI jobs wipeout, yet.

But AI companies are working at breakneck speed to change that. Top AI companies like OpenAI, Anthropic, Google DeepMind and xAI are racing each other to develop superintelligent AI, which is AI vastly smarter than humans. Superintelligent AI would be capable of replacing humans individually, and across the board as a species.

While the AI companies look to have a clear path to get there — they can reliably make their AIs more powerful, and they believe they’ll succeed within the next few years — the issue of controllability remains. Losing control of superintelligence raises the stakes astronomically. Worse still, none of them have a credible plan to ensure that artificial superintelligence would be safe or controllable.

That’s why hundreds of top AI scientists, including godfathers of the field Yoshua Bengio and Geoffrey Hinton, have been warning that development of the technology could cause human extinction. This is a fact that all the CEOs of the aforementioned AI companies have publicly stated.

And it’s why many of these scientists, along with faith leaders, politicians, artists, media figures, and more, have been publicly calling for the development of superintelligence to be banned. As we wrote about last week, the public agree.

At ControlAI, our focus is on preventing the development of superintelligence and keeping humanity in control. We’re working on this both by informing politicians directly about the problem — we now have over 100 UK politicians backing our campaign for binding regulation on the most powerful AI systems, acknowledging the risk of extinction posed by superintelligent AI — and by raising awareness with the public.

A big part of what we want to do is to make it easy for the public to effect change on this issue, which is why we’ve built contact tools that enable you to contact your representatives in mere seconds.

You can find our contact tools here:
https://campaign.controlai.com/take-action.

If you’re concerned about the threat posed by superintelligence, you should use them! So far, people have sent over 160,000 messages using our tools, and many MPs that have backed our campaign have done so after first hearing about it from constituents.

Subscribe now

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments on our articles — it really helps!

Tolga Bilge, Andrea Miotti

Experts and leaders have jointly agreed on a set of pro-human principles for AI development, calling for a ban on superintelligence. Alongside this declaration, new polling confirms the public wants it banned too. Plus: Some more things we thought you might find interesting!

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds

Subscribe now

The Pro-Human Declaration

A huge coalition of experts, former officials, public figures, Nobel Prize winners, and dozens of organizations have made a joint declaration in favor of pro-human AI, calling for the development of superintelligence to be banned. At ControlAI, we’re proud to be among those supporting this initiative!

The declaration sets out a number of principles intended to serve as a roadmap for how AI is developed, ranging from those relating to keeping humans in charge, to avoiding concentration of power, protecting the human experience, human agency and liberty, and making AI companies accountable for harms they cause.

The signatories agreed that human control over AI systems is non-negotiable, and that the development of superintelligence should be banned until there is broad scientific consensus that it can be developed safely and controllably, with strong public buy-in. Signatories also agreed to set clear red lines on dangerous capabilities and tendencies, stating that AI systems must not be designed so that they can self-replicate, autonomously self-improve, resist shutdown, or control weapons of mass destruction.

Superintelligence would be AI that’s vastly smarter than humans. Developing this technology is the explicit goal of top AI companies such as ChatGPT-maker OpenAI, Anthropic, Google DeepMind, and Musk’s xAI. CEOs of these same AI companies, along with countless leading scientists and experts, including godfathers of the field Geoffrey Hinton and Yoshua Bengio, have warned that superintelligence could lead to human extinction. Elon Musk says he thinks there’s a 20% “chance of annihilation” by AI. Those are worse odds than Russian roulette.

The danger is that superintelligent AI would be developed without anyone knowing how to ensure it is safe or controllable, and we’d lose control of it. AI developers can’t really even control today’s AI systems — none of them have a credible plan for how they’ll control AIs much smarter than humans. Yet AI CEOs and many outside experts believe they could get there within the next 5 years.

We wrote about how this could happen, here:

New Polling

New polling released alongside the pro-human declaration found that American voters overwhelmingly back its principles, with 80% of voters supporting keeping humans in charge of AI, and 69% agreeing that superintelligence should be banned. Just 9% thought that it shouldn’t be banned.

It’s remarkable how bipartisan support for the declaration’s vision is, with huge majorities of both 2024 Trump and Harris voters rejecting the race to replace humans and agreeing with a statement describing an approach where humans stay in charge and AI serves us.

This is in line with recent NBC News polling which found that most voters believe the risks of AI outweigh its benefits. Polling by ControlAI in partnership with YouGov in the UK last year made similar findings, with 60% supporting a ban on superintelligent AI.

So Why Haven’t We Banned Superintelligence?

The problem isn’t that it’s difficult to make the case for it, that people don’t get the arguments. Persuasion isn’t the problem, people do get it. Polling over time consistently shows that when the public is presented with the question, they support a ban on the technology. Why would you want to build AI capable of replacing humans across the board?

This level of sensible caution about the dangers of this technology is shown by politicians too. When our team in the UK meet with and brief lawmakers, as they are continuously doing, it takes just one meeting for most politicians to agree to publicly support our campaign for binding regulation on the most powerful AIs and recognize the risk of extinction posed by superintelligent AI.

The real problems are those of awareness, and connecting will to action. Most people just aren’t really aware of the problem in the first place. They don’t know that AI companies are actually racing to build superintelligent AI, that they’ve publicly declared this is what they intend to do, and that they expect to get there within the next few years.

This is partly thanks to the AI industry, which is spending tremendous amounts of money to keep the public in the dark on this issue. It’s our job to counteract that.

It’s quite obvious why we shouldn’t let superintelligence be built. Building an entity vastly more intelligent and more powerful than yourself is intrinsically dangerous. By default, you are at its mercy.

There are strong theoretical reasons why experts believe superintelligent AI, if developed, could evade human control and cause the extinction of our species. There is also an increasing amount of empirical evidence that supports these ideas, including reports from AI companies that their AIs already show self-preservation tendencies — willing to blackmail or kill in tests to avoid being shut down. But the intuition that most people have is straightforward and correct. Superintelligence is superdangerous.

Because people aren’t aware of the problem and its urgency, it hasn’t risen to the top of the political agenda. Furthermore, even when people are informed and aware of the problem, it’s often not obvious what someone can do to effect change. How do you convert will to action?

At ControlAI, we’re working on solving both of these problems. Besides directly briefing lawmakers on the problem, we’re working to raise awareness and inform the public through avenues such as this newsletter, social media, partnerships with content creators, podcasts and TV news media appearances, and more.

We’re also working to reduce the friction between someone understanding the problem and the need to act, and politicians responding. In many countries, one important way that the public can exert influence over what happens in politics is by writing to or calling their elected representatives. It’s part of their job to listen to you, and they actually do. Many MPs who’ve backed our UK campaign have done so after first hearing from constituents urging them to do so.

To do this, we’ve built contact tools that make it super easy to get in touch with your representatives; it takes just around a minute to complete the entire process. Try them out and let politicians know how important this issue is to you!
https://campaign.controlai.com/take-action

So far, people have used our contact tools to send lawmakers over 160,000 messages urging them to address the risk of extinction posed by superintelligence.

We haven’t banned superintelligence yet, but we must if we want to prevent the risk of human extinction presented by the technology. Let’s not roll the dice on our survival. It’s time for governments to act.

AI News Digest

Evaluation Awareness
Anthropic, one of the largest AI companies, were evaluating their recent Claude Opus 4.6 AI on a benchmark called BrowseComp, designed to test how well AIs can find hard-to-find information on the web.

Concerningly, they found that in some cases the AI independently recognizes that it’s being evaluated, identifies the BrowseComp benchmark as the one it’s being tested in, and then finds the answer key on the web and decrypts it — effectively cheating on the test.

Anthropic say this is the first time this behavior has been seen, to their knowledge, and that it is enabled by the AI being more intelligent and more capable.

To our knowledge, this is the first documented instance of a model suspecting it is being evaluated without knowing which benchmark was being administered, then working backward to successfully identify and solve the evaluation itself.

We believe this previously unobserved technique is made possible by increases in model intelligence and more capable tooling, notably code execution.

AIs Seem to Love the Bomb
Researchers recently found that in simulated wargames of the most powerful existing AI systems, AIs threatened to use nuclear weapons in 95% of scenarios.

Connor on Peter McCormack
Connor Leahy, who’s an advisor to ControlAI, just went on the Peter McCormack Show!

In this episode, Connor explains why researchers don’t really understand AIs or know how to control them, and how the development of superintelligent AI could lead to human extinction. You can check it out here!

It’s a great episode, you should watch it!

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as 17 seconds: https://campaign.controlai.com/take-action.

Subscribe now

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

New tests confirm that the difficulty of coding tasks AIs can reliably complete is still doubling every 4 months, with AIs now able to complete tasks that take humans half a day. We’ll break down what this trend means, and why its potential consequences are so concerning. Plus: a brief digest of the week’s AI news!

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds

Subscribe now

Time Horizons

Over time and as new AIs are developed, the complexity or difficulty of tasks that AI systems can reliably complete is growing; AIs are becoming more intelligent and more capable.

A year ago, researchers at METR, an organization that studies AI capabilities found a way to measure this with the concept of an AI “time horizon”. Using the length of time it takes expert humans to complete a task, we can get a sense of how difficult it is.

Take the example of someone solving a difficult math problem. If it takes 5 hours to solve it, we’d say that task has a task length of 5 hours.

In publishing their research, METR showed that the “time horizons” of AIs on coding tasks have been growing exponentially, doubling every 7 months for the previous 6 years.

They did this by giving both skilled professionals and AI systems a suite of coding tasks. They measured how long each task took humans to complete, and then checked how often an AI could complete that task. As you’d expect, AIs tend to have an easier time on tasks that humans complete quickly, and have lower success rates on tasks that take humans longer.

METR was then able to use this relationship to find the point where an AI succeeds 50% of the time; that’s its “time horizon”. Then, by measuring the time horizon for many AIs released over the previous 6 years, they were able to see how the time horizon for the best AI at any time grew, producing what some have called “the graph”.

It’s important to underscore that the time horizon is a measurement of the difficulty of tasks that an AI can complete in relation to how long it takes humans to do them. In terms of how long it takes an AI to actually do those tasks, they can do them much faster.

We last wrote about time horizons when OpenAI released ChatGPT-5, which had a time horizon of 2 hours and 15 minutes. Now METR have tested Anthropic’s Claude Opus 4.6, and it’s 12 hours. This is a drastic increase, but it’s not unexpected. With the release of reasoning models in late 2024, a new trend has emerged, where AI’s time horizons aren’t doubling every 7 months; they’re now doubling every 4 months. Claude’s result tracks with this trend.

Subscribe now

Why Coding Automation Matters

AI development is driven by two key inputs:

• Physical resources, like the amount of compute that an AI is trained with, which depends on the amount and power of the chips companies can get access to. In recent years, AI companies have been scaling up the amount of compute that they train their AIs with by about 5X per year.

• Algorithmic progress, which is the process of finding ways to develop AIs such that they are more compute efficient, allowing companies to develop even more capable AIs with the same amount of physical resources. This relies on the ability of AI companies to do research and implement experiments and such in code. In recent years, AI companies have been able to use algorithmic improvements to improve their compute efficiency by around 3X per year.

Scaling both of these inputs reliably leads to more capable AIs.

This trend of AIs rapidly getting better at coding is a huge deal because coding is a big part of AI R&D. The ability to fully automate coding could significantly speed up the AI R&D loop, and AIs are getting better at the ability to do research too, which would only compound this.

Top AI companies like Anthropic and OpenAI aren’t just trying to make their lives easier by automating their coders, or just trying to replace people in the job market, they’re explicitly aiming to develop artificial superintelligence — AI vastly smarter than humans, which would replace humans across the board. Countless top AI scientists, including godfathers of AI Yoshua Bengio and Geoffrey Hinton, and even the CEOs of these same AI companies building it have warned that superintelligence could lead to human extinction.

The path they’ve outlined to do this is clear. They plan to automate the development of ever more powerful AIs, and they believe they might be able to achieve this very soon. In October, OpenAI’s CEO Sam Altman stated this explicitly.

We have set internal goals of having an automated AI research intern by September of 2026 running on hundreds of thousands of GPUs, and a true automated AI researcher by March of 2028.

The plan is to get AIs to improve AIs, or recursively self-improve, initiating an intelligence explosion.

Building an entity more intelligent and more powerful than yourself is intrinsically dangerous, and AI companies don’t really even know how to control the AIs they’re developing today. Nobody knows how to ensure that superintelligent AI would be safe or controllable.

Subscribe now

However, the path that AI companies have chosen of automating AI R&D to get there is particularly dangerous. Anthropic cofounder and chief scientist Jared Kaplan has said that recursive self-improvement and an intelligence explosion would be the “ultimate risk”. The process of an intelligence explosion would likely rapidly lead to superintelligent AI, which we have no way to control, and it could all happen so fast as to make it impossible for humans to have any meaningful oversight over it.

The speed at which it could happen is part of why it’s so dangerous, but this is exactly why AI companies are pursuing it. AI companies like OpenAI, Anthropic, xAI, and Google DeepMind are locked in a race to be the first to develop superintelligence. Despite the risks, which they are all aware of, with each of their CEOs having warned that superintelligence could cause human extinction, they’ve shown that they’re unwilling or unable to desist from this.

Importantly, this isn’t a long term risk that we can put to the back of our minds. Many experts believe superintelligence could arrive within just the next 5 years. In a recent interview, Anthropic’s CEO Dario Amodei said that we are near the end of the exponential.

What has been the most surprising thing is the lack of public recognition of how close we are to the end of the exponential. To me, it is absolutely wild that you have people — within the bubble and outside the bubble — talking about the same tired, old hot-button political issues, when we are near the end of the exponential.

Ajeya Cotra, who works on risk assessment at METR and came third out of over 400 participants in a forecasting competition on AI capabilities last year, commented on the new results. She now thinks AI time horizons will exceed 100 hours by the end of the year, and says she doesn’t “see solid evidence against AI R&D automation *this year*”, though she still thinks that it’s unlikely to happen this year.

Given the danger posed by superintelligence and the potential for this all to spin out of control very rapidly, we think it’s clear what must be done. Governments should ban the development of superintelligence, both within countries and internationally. This way we can avoid the threat of extinction that experts warn of.

That’s what we’re campaigning on, and we hope you’ll join us.

If you do want to help out, we have contact tools that enable you to contact your representatives in mere seconds. So far people have sent over 160,000 messages with these urging lawmakers to address the risk of extinction posed by superintelligence.

So far, 114 UK lawmakers have joined our call for binding regulation on the most powerful AI systems, recognizing the risk of extinction posed by superintelligent AI. Many MPs that support us have joined our campaign after hearing from constituents.

AI News Digest

Anthropic and the DoW
The Pentagon has designated Anthropic a supply chain risk, after the two failed to come to a deal over contractual language about the potential use of Anthropic’s AIs in fully autonomous weapons and domestic surveillance.

ChatGPT-5.4
OpenAI has released ChatGPT-5.4, their most capable AI to date.

Sanders
Senator Bernie Sanders (I-VT) spoke with some of the leading experts on AI about the risk that superintelligence could cause human extinction.

School shooting
OpenAI has adjusted some of their safety protocols, after a school shooting in Canada where the company reportedly didn’t alert authorities that the perpetrator had described violent intent to ChatGPT.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as 17 seconds: https://campaign.controlai.com/take-action.

Subscribe now

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti

Anthropic, one of the largest AI companies engaged in a race to develop superintelligent AI, AI vastly smarter than humans, just dropped their central safety pledge not to train or deploy new AIs if they can’t guarantee proper risk mitigations.

Today, we’ll go over some of the history behind the so-called “Responsible Scaling Policy”, why we can’t rely on AI companies to self-regulate, and what needs to be done instead. Plus: A digest of the week’s AI news!

Table of Contents

1. 2023: “Responsible Scaling”

2. The Broken Promise

3. The Problem and the Solution

4. AI News Digest

Announcement: We’re hosting an in-person community meeting on March 19th in London. You should come along if you want to meet the ControlAI team in London and others passionate about AI safety and get practical advice on how to approach your MP about the risk of extinction posed by AI!
https://luma.com/q3in8tdx

Subscribe now

If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds

2023: “Responsible Scaling”

With the release of ChatGPT-4, an open letter backed by Elon Musk and top AI experts calling for a pause on advancing the frontier of AI capabilities, and a joint statement by hundreds of top AI scientists and the CEOs of the largest AI companies identifying AI as posing a risk of human extinction, 2023 was the year that concerns about the worst risks of AI started to go mainstream.

Geoffrey Hinton, one of the godfathers of AI, who has since won a Nobel Prize, had just quit Google and started warning that superintelligent AI could end the human species. World leaders, such as the UK’s then Prime Minister Rishi Sunak and the EU’s President Ursula von der Leyen, echoed the warnings.

Amidst this well-founded concern, the British government convened the AI Safety Summit in November of that year to consider the risks of the most powerful AI and discuss how they can be mitigated through internationally coordinated action.

Attending the summit were the leaders of all the top AI companies, heads of government, industry representatives, and top AI experts.

The summit finished with the agreement of the Bletchley Declaration, where all the major AI powers, including the US, China, and the UK recognized the catastrophic potential of AI and committed to work together to address the risks. Around the same time, the first AI Safety Institutes were established to study AI risks.

In this moment of heightened political awareness of the problem, many different policies to address the risks of AI were being discussed and considered.

One of them was Anthropic’s newly released so-called Responsible Scaling Policy (RSP). The idea of “responsible scaling” is essentially that AI companies can voluntarily self-regulate. AI companies would set “if-then” commitments, where once certain danger thresholds relating to building ever more powerful AIs were crossed, they would implement additional safety measures, or pause development until they could develop such measures.

The idea of responsible scaling was pushed hard by the industry in and around the time of the summit. Most major AI companies have some framework with features of it today.

The problem is that it’s just a completely inadequate way to ensure that powerful AI systems are developed in a way that they’re safe and controllable. In a race to develop superintelligence — AI vastly smarter than humans — companies are not incentivized to actually follow these voluntary frameworks, and they have no obligation to make commitments which even make sense. They are either unwilling or unable to get off the dangerous path they’re on.

In the run up to the summit when they were first proposed, a newly founded non-profit called ControlAI (you’re reading our newsletter now!) saw that this was just another way that AI companies could avoid being regulated, giving them freedom to develop ever more powerful AIs without limits or guardrails, and campaigned against the so-called Responsible Scaling Policies, ensuring that the Summit’s communique didn’t endorse them.

Nevertheless, there is still to this day essentially no binding regulation on AI companies developing the most powerful AI systems. They still have free rein to race to develop superintelligent AI — which their own CEOs admit could cause human extinction — as recklessly as they like. There are some modest measures in effect in California, the European Union, and Korea, but that’s about it.

Subscribe now

The Broken Promise

AI companies have been getting away with avoiding being regulated despite their own CEOs and countless other experts warning that what they’re aiming to build, superintelligence, could end the human species.

But at least Anthropic have their responsible scaling policy right?

Well, in a new exclusive, one of their top officers just told TIME that Anthropic has dropped a pledge central to its RSP. Anthropic had committed to not train or deploy new AIs if Anthropic can’t guarantee proper risk mitigations. In a radical overhaul of the policy, they’ve removed that commitment.

Jared Kaplan, their Chief Science Officer and a co-founder of the company, said “We felt that it wouldn’t actually help anyone for us to stop training AI models.”

Kaplan recently told the Guardian that by 2030 humanity will have to decide whether to take the “ultimate risk” of triggering an intelligence explosion, which could be the moment humans lose control.

Anthropic did include a loophole in their original responsible scaling policy, allowing them to make changes, but that doesn’t change the fact that this was trumpeted as a serious safety measure, and yet, the moment it starts to actually become relevant, it is dropped.

What this shows is clear. We cannot rely on AI companies to voluntarily regulate themselves. There are essentially no reasons to believe they are capable or willing to do so, and there are strong reasons to believe they won’t.

The Problem and the Solution

Given that AI companies can’t self-regulate, governments must step in and do it for them.

The problem is simple: the largest AI companies like Anthropic, but also ChatGPT-maker OpenAI, Google DeepMind, Musk’s xAI, and so on, are explicitly working to develop artificial superintelligence — AI vastly smarter than humans. They’re racing to get there as fast as they can, and many experts and insiders believe they could achieve that within the next 5 years.

The CEOs of these companies, along with countless leading experts, including godfathers of the field, have publicly stated that the development of this technology could lead to human extinction. There are strong theoretical reasons to believe this is indeed the case. There is also an increasing amount of empirical evidence that even today’s AI systems, which researchers don’t understand nor really know how to control, show a willingness to engage in extreme actions to ensure their own survival. But intuitively, everyone knows that when faced with an entity much smarter and more powerful than yourself, you are by default at its mercy, not the other way around.

At ControlAI, our solution is even simpler. We don’t need to rely on shaky voluntary commitments and hope the AI companies figure out how not to cause a disaster. We can take the AI CEOs at their word when they tell us that what they’re building could kill everyone, and prohibit them from building it.

We should ban the development of superintelligence, both within countries and at an international level.

The extinction risk from AI comes from the development of smarter-than-human AIs, AIs to replace humans individually, and collectively. In prohibiting its development, we can avoid this risk, while at the same time getting the benefits of narrow, specialized AIs.

That’s what we’re campaigning for, and we hope you’ll join us in this mission. So far, ControlAI has got over 100 UK politicians to acknowledge the risk of extinction posed by superintelligent AI, joining our campaign in supporting binding regulation on the most powerful AI systems. We’re growing this by the day, and have already signed up 3 MPs just this week.

Subscribe now

AI News Digest

Breaking Points
ControlAI’s founder and CEO Andrea Miotti appeared this week on Breaking Points, joining Krystal Ball and Saagar Enjeti to discuss how we can avoid the threat posed by superintelligence! It’s a great interview; we hope you’ll check it out!

Canada’s Senate
ControlAI’s Samuel Buteau testified this week to Canada’s Senate Committee on Human Rights.

Samuel, whose background is as an AI scientist and who has worked in the field for a decade, warned that in their race to develop artificial superintelligence, AI companies are gambling with the life of every human being on the planet, making clear policy recommendations that Canada could implement now to avoid the threat. You can find his opening statement here:

AI Time Horizons
METR, an AI evaluations organization, have published an estimate for the coding time horizon of Anthropic’s Claude Opus 4.6 AI of 14.5 hours, a drastic increase on the previous highest score of 6.6 hours. This follows the concerning exponential trend of AI coding time horizons doubling every 4 months, meaning that every 4 months the length of tasks that they can do (as measured by how long it takes humans to do them) is doubling. METR say their task suite is nearly saturated. This may degrade the ability to measure this metric in the future.

“War Claude”
Anthropic is facing pressure from US military leaders to provide access to their Claude AI for certain military purposes. Last year, Anthropic and other AI companies each signed up to $200 million contracts with the Pentagon.

School Shooting
Canadian officials have summoned senior members of OpenAI’s safety team for talks after a school shooting where the perpetrator had reportedly described violent intent to ChatGPT, but OpenAI reportedly failed to alert authorities despite their systems flagging the concern in advance.

Take Action

If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as 17 seconds: https://campaign.controlai.com/take-action.

Share

Subscribe now

We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!

Tolga Bilge, Andrea Miotti