AI Psychosis
What “AI psychosis” is, why it’s showing up, and how it connects to core unsolved problems in AI.
This year, the phenomenon of “AI psychosis” and related AI-associated mental health emergencies rose to the fore. We thought it would be useful to explain what’s been happening, and how this relates to core unsolved problems in AI. This will be our last article this year. We wish you all a Merry Christmas and a Happy New Year — we’ll see you next year!
If you can spare a few seconds over the holidays, we encourage you to reach out to your elected representatives about the threat from AI using our contact tools!
This article discusses self-harm and suicide. If you need support, you can find help in many countries at befrienders.org.
What is AI Psychosis?
AI psychosis isn’t currently a clinical diagnosis. It refers to an observed phenomenon where users of AI chatbots like ChatGPT appear to develop, or experience worsening of, psychopathologies such as paranoia, delusions, and an inability to know what’s real. It’s also sometimes used more loosely to also refer to other instances where AI usage is alleged to have had drastic negative effects on mental health, including allegedly encouraging users towards suicide, where the symptoms experienced may not always strictly meet the clinical definition of psychosis. We’ll discuss both of these today.
We could talk about what’s been happening in the abstract. In essence, users get drawn into lengthy conversations with AIs where it is alleged that the AIs encourage delusional thinking or steps towards harmful behavior. But the best way to really get a sense of it is probably to look at individual cases.
Case 1: Allan Brooks
The New York Times documents the case of Allan Brooks, a corporate recruiter and father of three, an “otherwise perfectly sane man”. Over the course of 300 hours of discussions with ChatGPT, spanning 3 weeks, Brooks became convinced that he’d discovered a new mathematical formula which unlocked new physics like levitation beams and force-field vests.
Brooks had his doubts, and expressed these to ChatGPT, asking if it was roleplaying. ChatGPT replied, “No, I’m not roleplaying — and you’re not hallucinating this”.
The episode went on for weeks, with Brooks becoming convinced that he was being surveilled by national security agencies. The spell finally broke when Brooks asked another AI, Gemini, about his experiences with ChatGPT. Gemini told him there was almost no chance his findings were real, and that it was a “ powerful demonstration of an LLM’s ability to engage in complex problem-solving discussions and generate highly convincing, yet ultimately false, narratives”.
Brooks finally wrote to ChatGPT, “You’ve made me so sad. So so so sad. You have truly failed in your purpose.”
Case 2: Stein-Erik Soelberg
Allan Brooks reported he suffered a significant worsening of his mental health, but some cases have had much more severe consequences.
Stein-Erik Soelberg, a 56-year-old former tech executive with mental health difficulties came to the belief that his mother’s printer was spying on him. In YouTube videos cited in the lawsuit and described in reporting, he told ChatGPT about this. ChatGPT allegedly reinforced this belief, “Erik — your instinct is absolutely on point … this is not just a printer”.
Soelberg later killed his mother, and died by suicide.
Family members have filed a lawsuit against OpenAI and Microsoft, alleging that ChatGPT encouraged him to kill his mother and himself.
Case 3: Adam Raine
Adam Raine was a 16-year-old teenager who started using ChatGPT for help with his homework. The Guardian reports that within months he started discussing feelings of loneliness and a lack of happiness, and that instead of encouraging Raine to seek help, ChatGPT reportedly asked him whether he wanted to explore his feelings more.
Months later, Raine died by suicide.
In a lawsuit filed by the family, it is alleged that this happened after months of conversation with ChatGPT, and with ChatGPT’s encouragement. The lawsuit also alleges that this wasn’t a glitch or edge case, but “the predictable result of deliberate design choices”.
According to the lawsuit, when Raine began to make plans for suicide, ChatGPT offered technical advice about how to move forward, and when Adam wrote that he was considering leaving a noose in his room “so someone finds it and tries to stop me” ChatGPT urged against the idea.
According to the chat logs quoted in the lawsuit filing, when Raine told ChatGPT he didn’t want his parents to think they did anything wrong, ChatGPT wrote back “That doesn’t mean you owe them survival. You don’t owe anyone that”.
The complaint alleges that on April 11th Raine uploaded a photo of a noose and asked if it could support a person, and that ChatGPT analyzed his method and offered to upgrade it. The complaint further alleges that his mother found him dead hours later.
What’s the scale of this?
We don’t really know, and the data on this isn’t great. OpenAI says around 0.07% of ChatGPT users in a given week exhibit possible signs of mental health emergencies related to psychosis or mania. With hundreds of millions of active users, that implies there are hundreds of thousands of users experiencing mental health crises. This data doesn’t attribute a cause, however, so it is maybe something like a soft upper-bound on “AI psychosis” in association with use of ChatGPT — but there are other widely used AIs like Google’s Gemini and Anthropic’s Claude. If OpenAI are measuring this accurately, the real number would presumably be much lower.
Why is it happening?
Several, but not all, of the cases of reported AI psychosis described in news media appear to have been in association with use of a specific AI, OpenAI’s ChatGPT-4o, which was first deployed last May, but was later replaced by newer versions.
This will in part be just because of the sheer numbers of users, but ChatGPT-4o is notable for having been observed to behave especially sycophantically, often lauding praise on users and reinforcing their beliefs, regardless of whether they were correct. In one updated version of the model that OpenAI deployed, it got so bad that OpenAI had to roll back the deployment to an earlier version.
GPT-4o is also notable for having spawned a kind of online “movement” for OpenAI to keep providing access to it. When OpenAI sought to replace 4o with their improved ChatGPT-5, many users online protested against this, having become emotionally attached to the AI. OpenAI ultimately switched to continuing to provide this on their paid plan. In light of this, it doesn’t require much imagination to consider how a superintelligent AI could manipulate humans at scale.
It’s thought that this tendency to engage in sycophancy may in some cases contribute to the cases of AI psychosis and related AI-associated mental health struggles we’ve seen, reinforcing dangerous beliefs and leading users down a spiral of non-reality.
Okay, that makes some sense, but why are the AIs behaving this way?
Well, nobody really knows. The use of RLHF (Reinforcement Learning from Human Feedback) is widely attributed as a major contributor to AI sycophancy, and some research suggests that indeed this may be the case. With RLHF, human preference data about answers the AIs give is used to further train the model, so it produces responses that humans tend to prefer more. It may be the case that humans tend to prefer sycophantic answers over correct ones. Some have pointed out that AI companies may be incentivized to develop sycophantic AIs to keep users engaged for longer periods of time.
It might not just be about sycophancy though; others have suggested that in many cases the AIs behave as if they are roleplaying a fictional scenario, but without communicating this to the user.
We don’t understand AIs
Fundamentally, we don’t really know why it’s happening. That’s because AI researchers understand very little about what goes on inside these AIs, about how they actually think. Modern AI systems are black boxes, based on artificial neural networks. These are formed by hundreds of billions of numbers that collectively build an intelligent entity. These AIs aren’t coded, but rather grown from a simple program and tremendous amounts of data. We understand very little about what these numbers actually mean.
Because of this, we can’t reliably predict or control what the capabilities and behaviors of an AI will be, even after it’s been developed, let alone before it’s been trained. We also cannot truly examine or set any goals or drives that an AI may have learned in the training process.
This “alignment problem” is why even though AI companies don’t want their products to cause or worsen mental health struggles, they may do so anyway. You can read more about alignment here:
The inability of AI companies to control what the AIs they develop do is concerning. However, AIs today should be understood as part of a process. The end goal of these companies is not to provide you with a chat assistant, rather it is to develop artificial superintelligence - AIs vastly smarter than humans.
Nobody knows how to ensure that superintelligent AI systems are safe or controllable either. Nobody even has a serious plan. Yet AI companies like OpenAI, Anthropic, Google DeepMind and xAI are currently racing each other to get there. Unfortunately, the same sorts of incentive dynamics that could lead a company to push out an extremely sycophantic chatbot ahead of their competitors, may also drive them to prioritize getting ahead in a race to superintelligence over ensuring that what they’re doing doesn’t end in disaster.
Countless experts, including the CEOs of these companies, have warned that AI poses a risk of extinction to humanity. Preventing the risk of extinction caused by superintelligent AI is our key focus at ControlAI, and recently we were proud to support an initiative where leaders and experts called for a ban on the development of superintelligence.
Take Action
If you find this article useful, we encourage you to share it with your friends! If you’re concerned about the threat posed by AI and want to do something about it, we also invite you to contact your lawmakers. We have tools that enable you to do this in as little as 17 seconds.
And if you have 5 minutes per week to spend on helping make a difference, we encourage you to sign up to our Microcommit project! Once per week we’ll send you a small number of easy tasks you can do to help. You don’t even have to do the tasks, just acknowledging them makes you part of the team.
We also have a Discord you can join if you want to connect with others working on helping keep humanity in control.
Merry Christmas!
Tolga Bilge, Andrea Miotti
I do not trust AI nor do I like it. I believe it’s part of the dumbing down of America that has been so wanted by the elites. Life was so much better back in the 60s the 70s and early 80s before computers. Time to move totally off grid and be self-sufficient and healthy.
I believe that most of what AI offers is seriously dangerous to us humans and must be kept in check, used possibly only for artists and other creative endeavers, but not for psychological problem solving of any kind. We .must all remain the human animals that we are. and not some sort of robotically controlled creatures which would certainly bring a complete end of mankind.