Special Edition: The Future of AI and Humanity, with Eli Lifland

We spoke with top forecaster and AI 2027 coauthor Eli Lifland to get his views on the speed and risks of AI development.

Welcome to the ControlAI newsletter! For our first ever interview, we sat down with Eli Lifland to learn about the future of AI. Eli is a coauthor of the recently published AI 2027 scenario forecast, which we covered last week. Eli’s an expert forecaster, and ranks #1 on the RAND Forecasting Initiative all-time leaderboard, so we thought it would be great to get his views on AI development and its risks.

We’d like to thank Eli for taking the time to do this interview. If you want to see more from him we’re including some links at the bottom of this post.

Table of Contents

1. Introduction

2. Forecasting

3. Tabletop Exercises

4. AI Timelines

5. AI Risks

6. Action

7. Reflections

8. Closing

Introduction

Tolga: For those who may be new to you, could you briefly introduce yourself?

Eli: I’m a co-founder of the AI Futures Project and a co-author on our recently published AI 2027 scenario. I also co-founded and advise Sage which builds interactive AI explainers on AI Digest. I co-run Samotsvety Forecasting and have performed well on various forecasting sites, though I don’t do much of this sort of quick forecasting these days, instead thinking deeply about the future of AI and key subquestions. I previously was a software engineer building Elicit, an AI-powered research assistant, and did AI robustness research.

Forecasting

Eleanor: You’ve spent a significant amount of time on forecasting, and forecasting AI specifically. For those who are unfamiliar with the field, what would you say the value of forecasting is? What makes someone a great forecaster?

Eli: At a high level, deeply understanding where things are going is an important step toward being able to steer toward positive futures. I like to think about forecasting as a spectrum between making often quick forecasts on a variety of precisely resolvable questions to thinking deeply about the future in a more in-depth way, and coming up with new frameworks and possibilities. I think forecasting on each end of the spectrum requires overlapping but somewhat different skills: for example, having precisely calibrated probabilities and evaluating others’ arguments is important for the former while for the latter it’s more important to be a creative thinker.

Eleanor: For the uninitiated, what is the difference between forecasting and someone giving their best guess?

Eli: I think a lot of work, depending on what you’re doing, is just kind of intuitive best guesses. A somewhat obvious result, but one that has been established, is that some people are consistently better forecasters than others. If you hold a forecasting tournament and then select the top few percent and call them superforecasters, they will generally and consistently continue to be better forecasters on further questions.

You can also have varying levels of effort put into forecasts, and that’s a factor. There are a few different dimensions here. There’s just time spent by a single person, but then also you can get a bunch of people to each make a forecast and aggregate them, which generally increases accuracy. There’s also the Delphi method, where you get people to first make forecasts, then discuss them, and then update based on the discussions, aggregating the final estimates.

In this scenario, we mainly thought deeply and sought to push the frontier of knowledge and scenario planning, which is kind of a different goal from taking some set of resolvable questions and trying to aggregate estimates on them. But we did do some aggregation, and we got a bunch of feedback and adjusted based on this.

Tolga: As you mentioned, you recently wrote the AI 2027 scenario forecast with Daniel Kokotajlo and others. What made you want to do this?

Eli: The boring answer is that I was applying for jobs and fellowships and this option seemed like the best of the ones that I got. The more interesting answer is that I thought it was valuable, for a while I’ve been interested in AGI futures but I only had a very hazy understanding of what a plausible rollout of the future might look like. I hoped that working on AI 2027 would improve my and others’ understanding of a concrete future through superintelligence, and I’m optimistic that it has done so.

Tolga: Could you say something about the process you used to tackle this problem? How did you break it down?

Eli: We made a list of ~10-15 important dimensions regarding AGI futures (e.g. training story, capability levels, internal AGI project governance, external governance, etc.) then filled in a giant Airtable where these were the columns and the time periods were the rows. We did sessions with experts on some of the particular columns. Then we sent out our draft for feedback from many people, and made adjustments based on that. Then we repeated this sort of process 1-3 more times depending on how you count, informed by the results of the previous iteration but slimming down and rewriting much of the content. We also ran about 30 AGI tabletop exercises which informed our thinking, and did in-depth research on key questions like timelines, takeoff, and what goals the AIs will have.

Tabletop Exercises

Eleanor: Something that we were both curious about is that you mentioned you ran 30 AGI tabletop exercises, again, thinking about our audience, that's something that may not have an intuitive role in forecasting or scenarios like this.

Could you explain the role that tabletop exercises play?

Eli: The premise is that we’re starting in the middle of our story at around March or April 2027, where the model weights have been stolen and superhuman coders have been developed, with a bunch of people playing different actors. Some of the most important actors are the President and executive branch, the CEO of the leading AGI company, the safety team of the company, the AIs, and China.

There are other actors that sometimes play big roles too, but oftentimes don’t. For example, Congress often doesn’t act quickly enough, and the public often doesn’t have that much of an influence.

It’s a turn by turn exercise. People have time to negotiate and think about what they want to do, and write it down. Then we go around and everyone announces what they did each round, and we do this for 3 or 4 hours until we get to a situation where it’s clear how the future is going to go, at least in broad strokes.

Usually what happens is that everyone fights over the geopolitical race. Then whether the AI happened to be aligned or not, by pretty boring techniques or not, determines the outcome of the game at the end.

But sometimes that’s not true. Sometimes there are mutual positives like more investment in alignment.

In terms of how it informed our writing, it was somewhat indirect. We didn’t purposefully try to import a bunch of lessons, but it gave us a picture of what a race to superintelligence could look like, especially geopolitically — there are often negotiations around international agreements.

Eleanor: So how do the tabletop exercises usually end?

Eli: Usually there are warning signs that the AIs are not aligned, but they’re ignored because of the race. The US-China race seems to go as fast as possible. There are cyberattacks, and then, the ending is usually determined by the goals the AIs have with relatively standard alignment techniques, and an allocation of a few percentage points of compute to automated alignment research — but there isn’t much expertise to evaluate the huge amounts of alignment research being done by the AIs. So whatever goals the AIs happen to have end up determining the future.

Sometimes the AIs will be aligned, and sometimes they’ll be clearly misaligned and cause extinction. Sometimes they just don’t care enough about humans to cause extinction, and just expand on their own into space. In the cases where they’re aligned, more often than not, there’s an intense concentration of power where the AIs are aligned to for example the CEO of the leading company or the President, rather than to the broader public.

AI Timelines

Tolga: People in this space are often asked “When do you think AGI will be built?” (also known as the timelines question). However, it’s unclear what people precisely mean when they say AGI, so I’d like to ask you a series of questions about how you see the trajectory of general AI development.

Before putting numbers on things, would you be able to give us a broad sense of where we are now on AI development?

Eli: Right now, we have AIs that are pretty good at coding and math in particular, and decent at various other things, including deep research. They’re also very good at being chat bots — they have a wide breadth of knowledge. They also, to varying extents, have different personalities, or feel like talking to a human in some respects. This isn’t an exhaustive list, but these are some skills that AIs are particularly good at right now.

On coding, METR has measured AIs can do tasks that take humans an hour, about 50% of the time. There are various asterisks on that result, but generally they’re quite good. My sense is that right now if you’re coding and not using AIs that’s pretty crazy, you’re already hampering yourself a significant amount.

But then they’re worse at various things, perhaps most notably computer use. They’re getting better, but for example OpenAI’s Operator isn’t really useful yet. They can’t really do things that take humans a minute, while for coding they can do things that take humans an hour.

The development that I imagine is that the AIs improve on all the different skills, but coding continues to be one of the top skills. That’s because the AI companies are prioritizing it, they have a lot of expertise, there’s a lot of data for it, and it’s relatively easy to formulate checkable problems where you can generate synthetic data or use reinforcement learning on a reasonable training signal.

In terms of what we’re projecting, we focus on coding, and project various other things in our side panel. The most important thing we’re tracking in the first part of our scenario is coding. Then there’s how we think that trend will continue of AIs being able to do tasks that are harder and harder for humans in terms of coding.

I think that METR’s time horizon suite is one of the main things to track here, as well as RE-Bench, which is a subset of that, but one that is particularly focused on coding and the research loop involving in AI R&D.

So then, what I see happening is that AIs will get to a superhuman coding level — it’s hard to say exactly when that is. In the scenario it happens in 2027, my median is more like 2030, but I give it around a 20% probability by 2027. Once the AIs are superhuman at coding, they can speed up the research process a bunch. I’m not imagining a scenario where they’re very competent at coding but they’re still totally incompetent at a bunch of other stuff. They’ll be learning a bunch of generalizable skills, and they’ll still be being trained on a bunch of other data as well.

Tolga: What happens once coding is automated?

Eli: Once coding is automated, we want to think about when the other aspects of the AI R&D loop are automated. Research taste is probably the most important skill there — the ability to choose which experiments to run and interpret them very well. I expect that research taste actually isn’t that far behind, because once you’re able to do very long horizon coding tasks very reliably, better than the best human, I think that it’s hard to have these sorts of skills without also having the skills to also be pretty good at selecting experiments, for example.

But I’m not sure, it could be spikier than that. It could even be that research taste arrives first, even though in some ways it seems harder to train. It’s hard to predict the exact sequencing.

So then we’re projecting that once we have these superhuman coders, and especially once we get AIs that are superhuman at everything in the research process, this will speed up AI development a lot. We have various estimates for how that will work in our takeoff supplement.

From there, we look at a few more milestones. One we call a superintelligent AI researcher, not just superhuman, which means that it’s much better than the best human. Then we project how long it’ll take to pass each of these milestones by thinking about how long it would take without the AIs, and then applying estimates for how much the AIs would speed up the process.

After that, we look at superintelligence, which is superintelligent across all domains, not just AI research. Somewhere in the mix is what we call superhuman remote workers, which is covered in the scenario but not the takeoff supplement. It’s more precisely been called TED-AI, which means Top Expert Dominating AI, which means at almost every cognitive task it’s at least as good as the best human at that. That’s something I think might arrive a bit before the superintelligent AI researcher, though it’s a bit hard to say and it depends on how spiky AI capabilities are.

Eleanor: In the scenario, you have artificial superintelligence arriving in December 2027. What events or information would cause you to deviate from this?

Eli: I’ll be tracking benchmark performance, especially METR’s time horizon suite and REBench, to see how I should update my timelines. I’ll also be tracking AGI company revenues as a metric of real-world usefulness, as well as anecdotal reports.

Eleanor: Do your own estimates differ much from the scenario?

Eli: The scenario is roughly my 80th percentile speed, i.e. I assign 20% to things going at least this fast. This is similar to Scott Alexander’s view. So I find it very plausible but not my median scenario. It is however roughly my modal view, I think 2027 or 2028 is the most likely year that superhuman coders arrive.

Eleanor: Why is your modal estimate before your median?

Eli: One reason is that I think compute scaling will slow down starting around 2029 if we don’t have AGI or aren’t close, because investment will not be able to grow at the same pace and chip fabs won’t be able to be built fast enough. I also think algorithmic progress might slow down.

Tolga: In AI 2027, OpenBrain follows a particular research agenda, focusing on first automating coding at a superhuman level, and then using this to bootstrap into superhuman researchers and superintelligence. Do you see alternative paths that OpenBrain might pursue, and what factors would underlie those considerations?

Eli: It’s possible that they focus on broader automation, though I think they might then get passed by a company more laser focused on automating AI R&D, and my sense is that the leading companies are already pretty focused on automating coding to varying extents.

Tolga: If we define “Competent AGI” using the definition that OpenAI currently gives for AGI: “a highly autonomous system that outperforms humans at most economically valuable work”, what do you think the chance is that this will be developed in 2025?

Eli: If this is outperforming the best human, maybe 2%? If this is outperforming the median human professional, perhaps 4%? These aren’t very precise, but basically it seems quite unlikely to me given METR’s time horizon trend and that I think there’s likely not a huge gap at the moment between private and public models.

Tolga: Last year, Elon Musk said that if you define AGI as smarter than the smartest human, he suggested that it would probably be developed this year, and in 2026 at the latest.

When do you think AGI defined this way would be developed?

Eli: I’m treating this the same as TED-AI, but it seems like you might mean something weaker. For TED-AI, you can take my all-things-considered views here which are for superhuman coders specifically and nudge them back a little. Perhaps (10th, 50th, 90th percentile) = (2026, 2031, 2065).

Eleanor: AI 2027 talks a lot about the actions and considerations of elite actors. What role do you see the broader public playing? What do you think happens if the public starts caring earlier, for example, if we see significant disruption of the jobs market?

Eli: I think the public is more likely to play a large role if takeoff is slower than we depict or the capabilities are more public. We do have the public getting more and more negative on OpenBrain over time already in our scenario though. If the public got more involved I think there might be more pressure to slow down AI or at least slow down broad economic deployment. I think it’s unlikely to build up enough to lead to international coordination on slowing down but I could see it happening.

AI Risks

Tolga: AGI and superintelligence would be incredibly powerful technologies. This seems to, necessarily, present some serious risk. At ControlAI, we’re focused on the extinction threat of AI to humanity, so I’d be particularly keen to get your views on how likely you think that is.

Eli: Roughly 25% on extinction, which is a subset of roughly 50% on misaligned takeover.

Tolga: It sounds like we are in quite a dangerous situation. People have been discussing these risks for decades, arguably as far back as Samuel Butler in 1863. How did we get to this point where we are so ill-prepared for what awaits?

Eli: Good question. I’m not sure why more people aren’t paying enough attention to it. I’d guess it’s because it seems like an abstract concern given that the AIs we currently have aren’t threatening. So it feels hard to find something concrete to do that feels useful, and it’s easy to forget about it and focus on things that seem more immediate.

Tolga: What other major risks could AI present?

Eli: I think misaligned takeover without extinction is plausible, and there’s a variety of outcomes within that. There’s also risks of astronomical suffering from various sources, which I think are important. And then there’s risks from human power grabs leading to AI-enabled concentration of power. There’s a good forthcoming paper on this from Forethought, and there are a few relevant expandables in our slowdown scenario if you’d like a preview.

Eleanor: Do you think actions that people take should be different depending on whether it’s, for example, a 10% or a 90% probability?

Eli: A lot are the same, but I think there are sometimes important differences. If you think almost all of the risk comes from misaligned takeover and are less concerned about variations in outcomes where AIs are aligned from e.g. concentration of power, you might be more in favor of boosting the right actors and more in favor of centralization, relative to someone who believes the opposite. If you’re at 90% p(doom) you might be more in favor of government involvement than someone who’s at 10% p(doom), if the person with 10% p(doom) thinks companies can probably handle any alignment problems that come up on their own.

Eleanor: In AI 2027, you have two potential endings. How do you weigh the relative likelihood of these?

Eli: Conditional on the scenario up until that point, I’d put roughly 75% on the misaligned takeover ending. We’re conditioning on a short timelines, fast takeoff, close US-China race, and at least a somewhat high alignment difficulty world, which is why it’s higher than my 50% above.

Action

Tolga: Given that we are faced with this grave and imminent threat, what actions should we be taking collectively to avoid this?

Eli: We don’t yet have polished policy recommendations and we wanted to keep the scenario forecast an epistemic exercise. We will likely be working on policy recommendations over the coming months. That being said, in future work we will publish policy recommendations. For now, we encourage more work to help policymakers and the public understand what is coming. This includes things like transparency into AGI companies, building state capacity in AI and AI emergency preparedness. We think these should be able to be supported by people with a wide range of worldviews.

Tolga: What are the biggest obstacles to us taking steps in this direction?

Eli: People aren’t taking the possibility of short timelines to AGI seriously enough, and to the extent they do they aren’t taking existential risk seriously enough.

Tolga: How do you see your work fitting into this?

Eli: We aren’t yet sure, we will be reflecting soon and deciding where we want to go as an org following AI 2027.

Eleanor: The field of AI safety is particularly small and under-resourced. What do you think people outside the field can do to help try to make sure that AI development doesn’t end in disaster?

Eli: It obviously depends on your skillset, consider applying to 80,000 Hours to get personalized advice. Apply for fellowship programs like MATS or Horizon. If you have money to spare, consider donating to the Long Term Future Fund or specific nonprofits in the space. Engage in high-quality discourse on social media and with friends.

Reflections

Eleanor: What do you think are the strengths and weaknesses of AI 2027?

Eli: The strength is that it’s a lot more detailed and thought out than any of the distinct scenarios like this. There are some others that were published recently that are pretty good, but ours is a bunch more detailed, and I think a lot more research went into it. Ours is also, thanks in large part to Scott Alexander, written in a pretty engaging way, and I’ve been happy to see that a lot of people have found it to be an engaging read, even if they disagree with it. I’ve seen some people on Twitter say things like “Yeah, you know, this is sci-fi, but it was a fun read.”

In terms of weaknesses, it would be really nice if there were more branches. In particular, there was a branch that we ended up deleting because we had trouble making it realistic. That branch was more like something that would actually work in a world where alignment was much harder than it is in the one we portray in the slowdown ending, and this would have to involve significant international coordination.

We had trouble coming up with something that seemed realistic enough compared to a situation where the US pauses or slows down a bit and cyberattacks China, and gets a lot more alignment research done than they otherwise would have. But we also think it could take longer, and we think ideally a responsible society would be able to coordinate internationally to have better safety guarantees, or at least a lot more confidence than in our scenario. So it would have been nice to have another branch that portrayed a more hopeful future, even if we thought it was less likely.

I’ve also seen some critiques that mentioned that we didn’t address skeptics enough, or that we only worked within our own frameworks and didn’t justify why they make sense compared to the frameworks that more skeptical people tend to use. I think it would have been nice to address those frameworks — we didn’t make it a priority to convince people who have thought a lot about this and have their own detailed models but have very different views. I think the scenario was more aimed at a combination of people who either don’t already have detailed skeptical arguments in mind, and people who already agree with us to some extent. Though I do think our scenario has probably helped convince some skeptics, even though it wasn’t an explicit goal, it would have been nice to engage more with them.

Closing

Eleanor: Thanks for doing this interview, it’s been great to get your insights into the trajectory of AI development and its risks. If people want to hear more from you, where can they go?

Eli: Thanks for having me! You can subscribe to the

AI Futures Project

Substack or my blog.

---

Thanks again to Eli for chatting with us for this interview. If you want to hear more from him you can read his blog or follow him on X / Twitter.

If there’s someone you’d like to suggest for an interview, feel free to comment on this interview or post it in our Discord.

Tolga Bilge

,

Eleanor Gunapala

,

Andrea Miotti