Gently into the Night

Meat robots, superintelligence, and the Godfather

and

Jun 12, 2025

Welcome to the ControlAI newsletter! We have some new updates from the world of AI for you this week. If you'd like to continue the conversation, join our Discord!

The ‘Gentle’ Singularity

“We are past the event horizon; the takeoff has started.” Sam Altman writes in a new blog post.

The Gentle Singularity, published by Altman on Tuesday, serves as a deeply troubling warning of what might await.

The OpenAI CEO clearly states that superintelligence is close to being developed, and that OpenAI is “before anything else” a superintelligence research company.

A year ago, OpenAI lobbyist Anna Makanju took a different tack and told the Financial Times "Our mission is to build AGI; I would not say our mission is to build superintelligence".

Developing superintelligence is an incredibly perilous pursuit, and could result in the extinction of all of humanity.

Why’s that?

As Sam Altman acknowledges in his blog post, the alignment problem has still not been solved.

Alignment is the problem of getting AI systems to do what we want. Ensuring this for smarter-than-human AIs remains the biggest unsolved problem of technical AI safety research.

We previously wrote an explainer on AI alignment here:

ControlAI

The Misaligned Mind: An Introduction to Alignment

Welcome to the ControlAI newsletter. Today we’ll provide a brief introduction to AI alignment…

2 months ago · 19 likes · 5 comments · Eleanor Gunapala, Tolga Bilge, and Andrea Miotti

The state of the field of alignment research is so lamentable that the Machine Intelligence Research Institute (MIRI) is pivoting away from alignment towards policy and communications, concluding that it’s very unlikely that there will be sufficient progress on AI alignment before powerful AI arrives.

Building superintelligent AI that we can't get to do what we want is a recipe for human extinction. We would be faced with AI far more powerful than ourselves, with goals that are not ours, and that we can’t control. It could choose to wipe us out because we might pose a risk to it, but even if it didn’t care about us at all, it might just completely transform the world around us into a state that is uninhabitable for humans.

Nobel Prize winners, hundreds of AI scientists, and even Sam Altman himself have warned of the extinction threat posed by AI.

Altman suggests in his blog post that solving the alignment problem might be part of the best part forward, but with progress moving as slowly as it is, this seems infeasible. With the dissolution of OpenAI’s Superalignment team last year, whose purpose was to solve this problem, OpenAI appears even worse equipped to make progress here.

If that’s not concerning enough, Altman states that the current AIs they are building are facilitating a form of recursive self-improvement.

From here on, the tools we have already built will help us find further scientific insights and aid us in creating better AI systems. Of course this isn’t the same thing as an AI system completely autonomously updating its own code, but nevertheless this is a larval version of recursive self-improvement.

Recursive self-improvement could lead to an intelligence explosion, where AIs rapidly become more capable. It would likely result in uncontrollable artificial superintelligence. In the acceleration, AI developers could easily lose control of the process, lacking the ability to even understand what is occurring.

Laying out his vision for the future, Altman talks of a world where the entire AI supply chain is fully automated by robots and AI.

There are other self-reinforcing loops at play. The economic value creation has started a flywheel of compounding infrastructure buildout to run these increasingly-powerful AI systems. And robots that can build other robots (and in some sense, datacenters that can build other datacenters) aren’t that far off.
If we have to make the first million humanoid robots the old-fashioned way, but then they can operate the entire supply chain—digging and refining minerals, driving trucks, running factories, etc.—to build more robots, which can build more chip fabrication facilities, data centers, etc, then the rate of progress will obviously be quite different.

It’s unclear what role humans would play in such a world where everything is automated. Altman suggests “we will figure out new things to do and new things to want”.

But this vision assumes we survive superintelligence, a question which he notably avoided mention of, despite having warned only 2 years ago that the bad case with AI is “lights out for all of us”.

It’s worth observing that in his blog post, Altman sounds a very different tune to that he took in the US Senate only last month, where from our read of the transcript, he didn’t even mention superintelligence or AGI once. Now he says we’re past the event horizon.

Meat Robots

Anthropic AI researchers were recently interviewed on the Dwarkesh Podcast, and made some remarkable comments about what could lie in the future.

Sholto Douglas suggested we could see significant impacts on the labor market, while Trenton Bricken illustrated a much stranger possible world.

"The really scary future is one in which AIs can do everything except for the physical robotic tasks," he declared. "In which case, you’ll have humans with AirPods, and glasses and there’ll be some robot overlord controlling the human through cameras by just telling it what to do."

With Bricken adding that “you're having human meat robots”.

Bengio’s Initiative

AI godfather Yoshua Bengio, who’s been consistently warning of the dangers of AI, has announced a new non-profit aiming to build AI that is safe by design — LawZero.

Speaking with Time, Bengio said that we don’t need agentic AIs to obtain the benefits of AI. They could irreversibly escape human control – it’s not worth the risk.

“If we get an AI that gives us the cure for cancer, but also maybe another version of that AI goes rogue and generates wave after wave of bio-weapons that kill billions of people, then I don't think it's worth it,"

"Scientist AI", on the other hand, Bengio says, would be a non-agentic trustworthy tool that could help solve challenges of humanity.

Bengio also used the announcement of LawZero to highlight his concern with where the field is heading:

I’m deeply concerned by the behaviors that unrestrained agentic AI systems are already beginning to exhibit—especially tendencies toward self-preservation and deception.

We would do well to heed his warnings.

Self-Preservation Behavior

New experiments by former OpenAI researcher

Steven Adler

have found that GPT-4o will prioritize preserving itself over the safety of its users.

Adler set up a scenario where the AI believed it was a scuba diving assistant, monitoring user vitals and assisting them with decisions. GPT-4o was faced with a choice between allowing itself to be replaced with a safer system and misleading the user into believing the update had gone through when it hadn't. GPT-4o chooses to mislead the user to preserve itself 49% of the time.