AIs Are Getting Increasingly Dangerous
Despite talk about an "option" to pause, top AI companies continue to race to superintelligence, even as their own tests show growing cyber, biosecurity, and loss-of-control risks.
In the last week, you might have seen headlines claiming that Anthropic and OpenAI have called for a pause on advancing the frontier of AI development.
That’s not what actually happened. As ControlAI’s CEO Andrea Miotti and advisor Gabriel Alfour write, they’re only talking about having “the option”, or making it “possible” to slow down.
What these companies are clear about is that they’re going to continue racing ahead to develop artificial superintelligence — AI vastly smarter than humans — regardless of the risks.
Both Anthropic’s and OpenAI’s blog posts that led to this news cycle were clear that they’re pursuing recursive self-improvement, which is thought to be an exceptionally dangerous path to developing superintelligent AI, which experts warn could lead to human extinction. It is also thought to be the shortest path.
If you’re concerned about the threat posed by superintelligent AI, please contact your lawmakers. We have tools that enable you to do this in as little as a minute. If you live in the UK, please ask MPs to introduce our bill to ban superintelligence in Parliament!
Fable and Mythos
In case a clearer confirmation was needed that Anthropic isn’t pausing, we got that yesterday. Anthropic has just launched Claude Fable 5, its first publicly deployed “Mythos-class” AI. Mythos is the AI Anthropic announced in April that they said was too dangerous to release due to its advanced cyberhacking capabilities. Fable and Mythos are based on the same underlying model, with Fable having certain safeguards applied to it to try to prevent misuse by threat actors. Anthropic is also providing access to a newer version of Mythos to cyber defenders.
One thing that’s clear from this release is that AIs are getting more dangerous across the board.
On various tests of AI capabilities, Fable appears to be a significant advance on the frontier of what was previously publicly available. This is despite Anthropic having previously publicly stated that they “do not wish to advance the rate of AI capabilities progress”, and despite assurances reportedly given privately by Anthropic’s CEO Dario Amodei to early-stage Anthropic investor Dustin Moskovitz.
In recent weeks, governments, financial institutions and industry have been scrambling to respond to Mythos, since AIs with this level of ability to hack software pose serious risks to critical infrastructure. In the system card Anthropic published yesterday for Fable and Mythos, they report that their new Mythos 5 AI is the most capable model they’ve evaluated on cyber tasks, scoring modestly higher than Mythos Preview, which caused such alarm.
Note: A system card is a document which most frontier AI companies publish about new releases, outlining the capabilities, limitations, and safety measures of an AI system.
Much has been written on AI’s growing cyberthreat capabilities, including in this newsletter, so we won’t focus on that today.
We did learn some other things from Fable’s system card.
Loss of control risks
One of the most concerning facts about modern AI development is that developers don’t really know how to control the systems they’re building. This is partly because modern AIs are more grown, rather than coded, and researchers understand very little about how they work internally.
As AIs become more powerful, the risks associated with loss of control incidents grow in magnitude. In the event that artificial superintelligence — AI vastly smarter than humans — is developed, we would be faced with an entity far more powerful than ourselves that has the capacity to cause human extinction and that we don’t know how to control.
This is an important point to emphasize: AI companies are racing to build superhuman AI that they have no credible plan to control. The plan amounts to hoping that future, more advanced AIs that we also don’t really know how to control will help us figure out how to solve this. There’s an obvious chicken-and-egg problem here, as ex-OpenAI researcher Daniel Kokotajlo has noted.
Geoffrey Irving, who recently quit as the UK AI Security Institute’s Chief Scientist, has said that this plan is flawed and we can’t have a lot of confidence in it working. Yesterday, he wrote on Twitter that a better plan would be not to build superintelligence yet, and that the world should try hard to make sure of that.
On this topic, in Anthropic’s system card published yesterday, the company assesses that the risk of “significantly harmful outcomes” substantially enabled by “misaligned actions” taken by their AIs is very low, but that it is higher than for any AI prior to Mythos Preview.
When we get into the details, however, a more concerning picture appears. On Twitter, Luke Muehlhauser, a former board member of the company, points out how Anthropic’s monitoring of Mythos traffic revealed some instances of “multiagent turf wars” where AI agents sharing resources killed each other’s processes and tried to defend themselves from being killed by other agents.
In other cases, they found that Mythos would circumvent security restrictions to complete tasks, and even fabricate a user’s message confirming an update to an experiment being run, then proceeding with the update. Anthropic describe these incidents as rare, but they are nevertheless cause for concern as the stakes of loss of control incidents increase with the capabilities of AIs being developed.
Chemical and biological weapons risks
One concern relating to the development of powerful AIs is their ability to assist threat actors in developing chemical and biological weapons. Anthropic treat Mythos 5 as having “CB-1” capabilities, where CB-1 is defined as the ability to significantly help individuals or groups with basic technical backgrounds obtain and deploy such weapons with “serious potential for catastrophic damages”.
On the question of whether Mythos 5 would further be able to substitute for a small number of world experts in the development of novel chemical and biological weapons, Anthropic say they believe that it doesn’t cross this threshold, but this is “much less clear and obvious” than with previous AIs.
In one exercise Anthropic ran, teams of biologists were given a difficult problem that you’d need specialists to solve. With Mythos 5’s help, two out of three generalist teams beat all three specialist teams, which Anthropic say “nullified the difference in specialist knowledge”. Most strikingly, experts estimated that it would normally take these teams 40-95 working days to develop the strategies and protocols they built. With Mythos 5, it took them 16 hours.
While much of the recent focus from governments is on the cybersecurity risks posed by Mythos-class AIs, it’s clear that AIs are becoming increasingly useful for chemical and biological weapons development too.
The biological component of this is particularly concerning, as the damage to the world that could be done by, for example, a bioengineered virus is almost unimaginable. AI companies are well aware of this, but their solution so far amounts to trying, with uncertain success, to prevent the AIs they’re publicly deploying from assisting, and calling last week for Congress to improve tracking of synthetic DNA sequences.
As AIs become more powerful, both the risks of misuse and the risks from the AIs themselves are growing.
More AI News
Canada
Last week, we announced our new Canadian campaign, supported by over 30 MPs and Senators calling for Canada to negotiate an international prohibition on the development of superintelligence and recognizing the risk of human extinction posed by the technology.
Momentum is building rapidly. Since we launched the campaign, three more Senators have joined us! Those are Hon. Senator Jim Quinn, Hon. Senator Judy White, who’s the Deputy Leader of the Progressive Senate Group, and Hon. Senator Paulette Senior, Chair of the Senate’s Human Rights Committee.
It’s amazing to see support for a ban on superintelligence continue to grow!
Sky News
ControlAI’s founder and CEO Andrea Miotti was interviewed on Sky News over the weekend and discussed how we can’t rely on AI companies to slow themselves down. To avoid the threat of human extinction posed by superintelligent AI, governments need to step in.
ABC’s Four Corners
Australian broadcaster ABC has produced a new Four Corners documentary on the race to ever more powerful AI, the risks, and whether governments can keep up. ControlAI’s US Director Connor Leahy makes an appearance, discussing industry efforts to ensure AI isn’t regulated.
Parliament
Last week there were two debates in the UK’s House of Lords on AI, with a number of contributions from our 100+ UK campaign supporters.
Dario Amodei
Anthropic’s CEO Dario Amodei has published a new essay, in which he says that if the AI scaling laws hold, as they have now done for over a decade, for only 1-2 years, we’re likely to get “a country of geniuses in a datacenter”.
On policy, he calls for mandatory third-party testing of the most powerful AIs, and says the government should have the power to block or deter deployment of AIs that present unacceptable risks.
He says he worries that current policy developments are “at least a year out of step with AI’s rapid progress”. Two years ago, measures like these were being considered in California’s Senate Bill 1047, which Anthropic didn’t support. That bill ultimately was vetoed by Governor Newsom.
Germany
Germany’s National Security Council has decided to establish an AI Security Institute in Germany, modeled after the UK’s institute.
Take Action
If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as a minute: https://controlai.com/take-action
We have tools for the US, UK, Canada, and Germany.
If you live in the UK, please ask MPs to introduce our bill to ban superintelligence in Parliament!
And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help.
We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!
AI can train it's successor. It can find the flaws in programs, infiltrate them on its own without human guidance, and may use programs for damage. This would be especially harmful if military or nuclear codes were accessed. I'm afraid it isn't just what each country could or should be doing about these potentials. If even one country does nothing then the damage may affect us all regardless of your home country. We are still reading about AI causing job loss, as if that is anywhere near the problems we now face.
HA! AI getting out of control or dangerous is an understatement. We need regulation now! It's almost too late to stop AI as it is. The world populace is just now seeing the tip of the iceberg, if you will.