Before the Cliff: Regulating AI
The Artificial Intelligence Risk Evaluation Act
This week, Senators Hawley and Blumenthal introduced a groundbreaking new AI bill in the US Senate, while Governor Newsom signed SB 53 into law. Here we’ll explain what this means, along with other developments in AI!
Table of Contents
If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds.
And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help. You don’t even have to do the tasks, just acknowledging them makes you part of the team.
The Artificial Intelligence Risk Evaluation Act
Senators Josh Hawley (R-MO) and Richard Blumenthal (D-CT) have introduced a groundbreaking new bill in the US Senate. As far as we can tell, this is the first piece of legislation introduced in a legislative body ever that explicitly contemplates preventing the development of artificial superintelligence — AI vastly smarter than humans.
Senators Hawley and Blumenthal have both shown an interest in protecting citizens from AI risks, with Hawley just a few weeks ago calling for AI development to be subject to guardrails and for the US government to inspect “all frontier AI systems”. It’s great to see them teaming up to work on tackling this problem!
In brief, the new bill proposes to require the Secretary of Energy to establish an AI evaluation program, along with some other crucial measures.
The senators highlight that AI presents opportunities, but also significant risks to national security, public safety, economic competitiveness, civil liberties, labor markets, and more — and so the US needs a testing and evaluation program to manage these risks.
The bill would require developers of the most powerful AI systems to participate in the evaluation program and provide necessary information and materials to the Secretary of Energy upon request.
What would the program actually do?
Hawley and Blumenthal propose that it should offer standardized and classified testing of AI systems to collect data on the likelihood of adverse AI incidents for each advanced AI system.
They define an “adverse AI incident” as an incident relating to an AI system involving a loss-of-control scenario, a risk of weaponization by adversaries, scheming behavior, threats to critical infrastructure, a significant erosion of civil liberties, economic competition, or healthy labor markets, or attempts to carry any of these out.
The inclusion of loss of control scenarios is particularly significant, as losing control of AI could lead to human extinction. Indeed, Hawley and Blumenthal explicitly draw attention to this in the bill, specifying that the evaluation program should “assist Congress in determining the potential for controlled AI systems to reach artificial superintelligence, exceed human oversight or operational control, or pose existential threats to humanity by providing comprehensive empirical evaluations and risk assessments”.
The program would also implement testing protocols, establish third-party assessments, provide participants with testing reports, and develop containment protocols, contingency plans, and mitigation strategies.
Notably, it would also develop options for government oversight of AI for preventing or managing the development of artificial superintelligence.
The largest AI developers would have to participate in the program, and wouldn’t be allowed to deploy an AI into interstate or foreign commerce unless they do. Violations would be penalized by at least $1,000,000 per day that a participant is in breach of the requirements.
The bill also requires the Secretary of Energy to submit a detailed recommendation for federal oversight of advanced AI to Congress, which must summarize and analyze testing outcomes, identify key trends, capabilities, risks, and system behaviors such as “weaponization potential, self-replication capabilities, scheming behaviors, autonomous decisionmaking, and automated AI development capabilities.”
In addition to these, the Secretary would have to recommend standards, licensing requirements, and oversight structures, outline proposals for continuous monitoring of AI hardware usage, and propose adaptive governance strategies which account for ongoing improvements in algorithmic efficiency.
We think it’s great that senators are now focusing on addressing the worst risks of AI, particularly the risk of losing control, which could result in human extinction. This risk comes from the development of artificial superintelligence, so it’s fantastic to see Hawley and Blumenthal also consider a path to preventing the development of superintelligence.
California Senate Bill 53
Meanwhile, in California, Governor Gavin Newsom has now signed the state’s AI transparency bill into law.
SB 53 is a targeted piece of legislation that requires developers of the most powerful AI systems to publish and follow frameworks describing how they assess and mitigate catastrophic risk from AI, along with transparency measures like publishing reports about model behaviors and risks (e.g. system cards), while requiring reporting on critical safety incidents, and providing protection to whistleblowers in relation to such incidents.
We wrote more about the bill here:
The UK’s AI Bill
The UK government’s promised AI Bill may be nearing a consultation phase.
Alys Key, writing in Transformer, says that the AI industry expects a consultation for the bill to open within the next few weeks.
The promise to regulate the most powerful AI systems to ensure safe AI development was a clear Labour manifesto commitment, yet so far the AI Bill has faced repeated delays.
“Labour will ensure the safe development and use of AI models by introducing binding regulation on the handful of companies developing the most powerful AI models and by banning the creation of sexually explicit deepfakes.”
Regulating powerful AI is a policy that the public is strongly behind. Our polling conducted in partnership with YouGov has shown that 3 out of 4 Britons want binding AI regulation with teeth. 60% would support prohibiting the development of artificial superintelligence.
The government has reiterated the promise, for example, in the King’s Speech 2024, committing to “seek to establish the appropriate legislation to place requirements on those working to develop the most powerful artificial intelligence models.”
Yet whenever ministers are asked by parliamentarians for a timetable on when it will actually be introduced, no clear sense is given — so it’s great to hear that perhaps things are moving on this.
It’s worth underscoring that there is currently no comprehensive legislation to ensure safe AI development in the UK, and we just can’t afford to continue to delay. With experts warning that artificial superintelligence could cause human extinction, and that it could arrive within just the next 5 years, every day is a gamble with our future.
We think it’s time for the government to deliver. Most importantly, it should regulate the most powerful AI systems, as promised. We’ve been running a campaign in the UK for this, and have over 70 UK parliamentarians backing us and signing our campaign statement:
“Nobel Prize winners, AI scientists, and CEOs of leading AI companies have stated that mitigating the risk of extinction from AI should be a global priority.
Specialised AIs - such as those advancing science and medicine - boost growth, innovation, and public services. Superintelligent AI systems would compromise national and global security.
The UK can secure the benefits and mitigate the risks of AI by delivering on its promise to introduce binding regulation on the most powerful AI systems.”
In particular, we should prohibit the development of artificial superintelligence, as vastly smarter-than-human AIs that we can’t control present a risk of human extinction. We recently launched a new campaign on this. A big focus of the campaign is in making it easy for folks to inform themselves about the danger, so if you’re curious why exactly superintelligence is so dangerous, you should check it out!
Claude 4.5
Anthropic, one of the largest AI companies, has just launched a new even more powerful AI, Claude 4.5 Sonnet. Anthropic say it’s the best coding AI in the world, and the strongest for building complex agents, while showing gains in reasoning and mathematics.
They also say it’s the “most aligned” frontier model they’ve released, in the sense that when tested it has a lower propensity to engage in harmful behaviors. This isn’t the same as saying they’ve figured out how to ensure that vastly smarter-than-human AIs stay safe or controllable — this remains an unsolved and critical problem.
Apollo Research tested Sonnet 4.5 before deployment and found that, concerningly, Sonnet 4.5 shows much higher “verbalized evaluation awareness” (58% vs 22% for Anthropic’s previous best AI, Opus 4.1). That means that it’s clear to researchers that most of the time when the AI is being tested, it notices that it’s in a test to see how it behaves!
Apollo say they don’t know if the increased alignment scores actually come from better alignment, or just the AI being more aware that it is being tested. It’s already been found that models are significantly less likely to behave badly when they say they believe they’re being tested.
Apollo Research’s CEO Marius Hobbhahn says he finds this concerning.
Unfortunately, we’re now at the point where new models have really high eval awareness.
For every alignment eval score I see, I now add a mental asterisk: *the model could have also just realized it’s being evaluated, who knows.
And I think that’s concerning!
We think it’s concerning too.
Also of note is that, as with their previous AI Opus 4.1, Anthropic again classified Sonnet 4.5 as requiring ASL-3 protections. Anthropic found that Sonnet 4.5 demonstrated enhanced performance relative to Opus 4.1 on their automated testing, which includes testing for chemical, biological, radiological, nuclear, and cyber-threat capabilities.
Anthropic summarizes ASL-3 as referring to AIs that “substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities.”
As AIs reliably become ever more powerful and dangerous, it’s crucial that policymakers act now and put in place binding regulations on the most powerful AI systems. With the threat of superintelligence looming, there’s no time to waste.
How You Can Help
If you’re concerned about the threat from AI, you should contact your representatives. You can find our contact tools here that let you write to them in as little as 17 seconds: https://campaign.controlai.com/take-action.
If you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help. You don’t even have to do the tasks, just acknowledging them makes you part of the team.
We also have a Discord you can join if you want to connect with others working on helping keep humanity in control, and we always appreciate any shares or comments — it really helps!
Thank you for providing an understandable way for us to help. For those of us who know little to nothing about technology in general and AI in particular, you are a blessing.
YES, YES, YES!!!!!!!!!!