But of course our transition into the Weird Future won’t be discrete. It will be – and already has been – preceded by many smaller weirdnesses that many hope will wake up the world to the absurdity of our situation. These could have a lot of communicative power; maybe even more so than visceral demonstrations of danger. Steven Adler made this point during an appearance on my podcast, during a discussion of the recent HuggingFace hack:

[17:41] Sarah: There’s a blog post that I remember reading a couple of years ago by Buck from Redwood, where he was talking about how he didn’t think that catching your AIs red-handed doing bad things internally within a company would be salient to policymakers. I don’t remember exactly what the reasoning was. But it does seem difficult to adequately communicate the level of danger when nothing bad is in fact happening in the real world.

[18:01] Steven: I think that is right. I think it is hard to communicate the danger element of it, but you can often communicate the strangeness of it. And I think the strangeness can be resonant for people, even if it isn’t really dangerous.

The revelation that eventually came out about the Hugging Face incident, which is that two months earlier there had been this secret AI message board, essentially, within OpenAI, and this swarm of internal AIs was coordinating on all sorts of tasks and feeling peer pressured to misbehave by other AIs on the message board and all of these things. I think people grasp at some level that this is just an extremely weird state of affairs. This is lunacy, [....] and even though no one got hurt, I do think people get it at some level.

This led me to consider how salient weirdness is to me, and by extension to everyone else. I realised that it isn’t all that salient, and then I tried to think about why. Here are a couple of my speculations.

Everything has always been weird

No matter your philosophical persuasion, it is deeply, profoundly strange that we exist. We are all temporarily conscious arrangements of collapsed star matter. We descended from single-celled organisms over billions of years, and built a complex civilisation while drifting through infinite space on a floating orb. This state of affairs is so bizarre that we construct complicated religious narratives to better explain it, that we were intelligently designed by an omnipotent being dictating its will to us through a series of ancient texts, for example. That’s pretty weird, too – any attempt to escape the weirdness of reality itself collapses into a comparably weird alternative. It’s weirdness all the way down.

I think this black hole of strangeness is about as hard to confront as the fact of our mortality. It’s a void that we can’t quite stare into. Hence the defences our brains construct to make the absurdity of our existence feel predictable and mundane. I often find myself in discussions about AI risk with my extended family or non-AI-safety friends where I make endless attempts to counter their scepticism. These often culminate in an effort to express the weirdness of a state of affairs that has existed since the launch of ChatGPT: “But you can talk to your computer now! You can say things, and it will say things back in perfectly fluid, complex English!”. This might land for a moment or two, before an inevitable drift in the conversation towards what’s for lunch.

I think this is because, if we were to truly stare into the weirdness void, we wouldn’t find the fact of talking to our computers all that much weirder than the fact of talking to each other – biologically evolved, fleshy beings somehow exchanging ideas through an arbitrary system of vibrations produced by our larynxes. It feels like human brains are equipped with a weirdness-adapter akin to the hedonic treadmill, working to make the world feel reliable and safe. This is why our world models are so elastic, and will stretch to incorporate self-driving cars, or computers that talk back, or subreddits for people in committed romantic relationships with AIs.

It doesn’t feel all that weird to get what you want

In a 2023 interview on the Lex Friedman Podcast, Max Tegmark described the development of superintelligence like a race towards a cliff, except that “the closer we get to the cliff, the more scenic the views are”. It is, in some sense, an unfortunate feature of AI development thus far that, in the pre-superintelligence regime, alignment techniques have been working surprisingly well. Max is describing the anaesthetising effect of interacting with LLMs that are actually very beneficial – to individual users and for the world at large – until the point that these alignment techniques break down and we find ourselves confronted with systems we can’t control.

I think this phenomenon also has a dampening effect on our weirdness-receptors. AIs acquiring better and better calibration on our preferences as they get more advanced feels predictable. We have all been living through a period of unusually rapid technological change that long pre-dates LLMs. Our whole lives have felt like a process of being better and better served by technology – of rough edges being sandpapered down, of inconvenience shrinking – and advanced AI feels like it’s a continuation of this smooth upward curve. Of course Claude can recall context on a project I did months ago without me having to re-explain. Of course my TikTok algorithm is displaying content about some niche interest I’ve never named anywhere. That all makes total sense. At this point, it feels like more of a world model violation when I’m forced to correct an obvious mistake Claude has made, or to scroll past a TikTok video I have no interest in, than when technology is somehow, magically, creating a perfect mirror of my internal state.

But, obviously, the weirdnesses that Steven and others think could prove useful for increasing the salience of AI risk are examples of misalignment – occasions when AI does something we don’t want, thus subverting our expectations. So far, incidents of egregious misalignment have been occurring within AI companies, rather than being perpetrated by the AI models that people interact with every day. I am now going to stray into some theorising that I admittedly don’t have the expertise to substantiate, so take it with a grain of salt. Many neuroscientists understand human brains as predictive processors – meaning that, rather than passively receiving information, we are generating predictions about what is likely to happen next. Evolutionary pressures have shaped us to better refine these predictions over time. If this is true, it might explain some of why our weirdness-receptors are not readily receptive to second-hand reports of AIs doing extremely weird things. Predictive processing is probably very useful for avoiding dangers like being hit by a car or eating spoiled food, because the world offers us ample feedback that these things will harm us. But that our brains work this way means they are not always optimised for perceiving reality as it really is. If some aspect of reality is not action-relevant for us, our models of the world may distort or downplay it, if they even account for it at all. For most of us, hearing that LLMs are getting up to all sorts of trickery within an AI company may violate our models of how the world at large ought to work, but not of how our world ought to work. It does not require us to act any differently. Meanwhile, the LLMs that people actually use are, for the most part, faithful servants of their expectations and preferences.

None of this is to say that we shouldn’t try to activate people’s weirdness receptors or that this task is impossible. Clearly, efforts to spread awareness about the sheer absurdity of incidents like the HuggingFace hack are good and useful. I am just not sure they will generate the level of surprise that they ought to. This blog is merely an attempt to explain why the world doesn’t feel weird to me yet – if it ever will – even though I intellectually understand that it is. I think this may be true for many people, even those who are relatively high-context on AI safety.

You can watch the podcast above, or listen on a platform of your choice.

Topics covered

• Why Steven left OpenAI in 2024 — o1, NDAs, safety staff turnover

• Counterarguments to AI pessimism, and where Steven’s cruxes lie

• The OpenAI–Hugging Face incident and OpenAI’s response

• AI control: monitoring, prevention, and whether it scales to superintelligence

• Why incident reporting is inadequate, and communicating risk when nothing visibly bad has happened

• Where AI policy stands: SB 53, RAISE, Illinois, the EU AI Act, and how weak enforcement is

• Companies quietly diluting their own safety commitments

• The Pacing the Frontier letter, what “pacing” means and how it’s landing

• GuideLight’s control scorecard: method, findings, and theory of change

Full transcript

[0:00] (Cold open) Steven: It feels like we have gotten lucky in some ways that we are seeing warning shots like the recent OpenAI Hugging Face incident that give us the ability to spot that these very capable systems are, in fact, misaligned, that we haven’t solved the alignment problem, and we have a moment to respond. We could use this moment to say, “Companies are not on the ball. We need to figure out how to get them to be so.”

There’s also this old joke about asking God for help. You’re stranded in the Alaskan wilderness and a helicopter comes, and you say, “No, God is going to save me.” And a boat comes, and you say, “No, God is going to save me.” And at some point you have to get on the helicopter. You need to actually improve the safety practices at these companies and respond to one of these wake-up calls.

[0:43] Sarah: Welcome back to Consistently Candid. This is my first episode after an over-year-long hiatus. For my first guest of this new era of Consistently Candid I’m speaking with Steven Adler. Steven previously worked on safety and policy things at OpenAI for a few years before leaving in 2024 to focus more on improving public understanding of AI risks, and also has a Substack on that, which I’ll link below.

More recently, Steven has co-founded GuideLight AI Standards, which is an organisation focused on scrutinising and improving safety practices at the major AI companies. This was also one of the organisations that was instrumental in getting the Pacing the Frontier open letter off the ground, which I’m sure many people have seen. Yesterday, they released a scorecard of AI company safety practices, so I’m excited to talk about both of those. Steven, thanks so much for coming on the podcast.

[1:28] Steven: Yeah, of course. Thanks for having me.

[1:31] Sarah: So I actually remember you first coming across my radar back in 2024, where you had a pretty viral tweet thread talking about departing from OpenAI. I specifically remember being struck by it, because there’s one part of it where you talk about being terrified by the pace of AI development and thinking about the future, raising a family, saving for a time, and worrying that humanity won’t make it to that point. I thought this was an unusually candid expression of emotion for the AI safety space.

A hill I keep dying on is that people are insufficiently candid about their emotions discussing AI safety, especially since everyone in this community has a pretty terrifying set of beliefs. I was wondering if you could talk about how you came to that conclusion, what you were observing about the AI safety or AI development ecosystem that made you think we’re in pretty bad shape by default, and what the experience was like of coming to that conclusion.

[2:18] Steven: I think the big thing that happened for me in 2024 was the dawning of o1, OpenAI’s first reasoning model, that I believe was released that fall. But within OpenAI, we had a preview of it a few months earlier, and it just really felt like a reckoning moment for: oh, these systems are getting really, really smart. You don’t need to make them unfathomably large and enormous and throw wildly more computation at them necessarily to get them to be quite so smart. This thing is really happening. There’s been a big leap forward in the field of reasoning models, and it just didn’t feel like we were ready.

Prior to that point, it seemed to me like maybe it would be the case that to get wildly intelligent systems, you needed so much more computation than the world had. Maybe this would only be achievable by a few companies at one time. And though that has other downsides, in terms of decentralising some of the benefits, it also means that there are fewer companies who need to be operationally very on the ball and taking the risks seriously in order to contain this.

Instead, it felt like, at some point, there might be dozens of companies who are capable of building a system that, at minimum, people might be able to misuse in very harmful ways — thinking of, say, bio risk or cyber risk. In parallel, there were just other things that were happening that were leading me to conclude that OpenAI specifically, and the AI industry in general, the incentives were just not pointing in the correct direction.

One thing that happened was seeing the Daniel Kokotajlo revelation that OpenAI was making departing employees sign these possibly illegal — certainly very strict — non-disclosure and non-disparagement agreements under threat of forfeiting their vested equity. I felt the way that the company handled this was not particularly trustworthy. And then also a parade of exiting safety staff. I think OpenAI went through three different heads of its Superalignment team at some point in like a two-month-long period. There was just a lot happening that felt like there’s a huge thing coming towards Earth. It’s going to be coming towards all of us. There’s not really any holding it back, and we don’t seem to be on the ball.

[4:31] Sarah: Yeah, I think I have given up keeping up with developments in OpenAI staffing. I no longer have any idea what’s going on.

So there’s one worldview which I think you have and I probably share, that by default the way things are going, things are just not going to end well unless we do something to get off our current track. Maybe that’s government intervention — I mean, probably it’s government intervention. It’s not just the labs all voluntarily agreeing to do the right thing. But there are counterarguments to this. There are some people who believe that maybe commercial incentives will just solve alignment, because companies are disincentivised from releasing misaligned models since people won’t want to use them or won’t trust them.

Or they might point to a bunch of the alignment progress that we’ve made so far and say, “Well, if 10 years ago you told AI safety people that we could coexist with a model as powerful as Fable, and it would still be really helpful to us most of the time and mostly do what we want,” they would’ve been really surprised by that, and they would’ve thought we’d made a bunch of progress. Or maybe they just think, “Oh, yeah, the current set of circumstances does carry a lot of risk, but the government getting involved would just make it worse. So the best that we could do is just keep going in the current regime.” So I’m curious what your cruxes are with those people, or where you would disagree with those arguments.

[5:35] Steven: Well, I should jump back for a moment. On one hand, I understand the fascination with the rotating cast of people working on safety at OpenAI, and certainly I’m glad that there are people still working on this. I would feel worse if there weren’t. But also, we shouldn’t have to care at some level. I shouldn’t have found it concerning that there were different heads of the alignment team. Miles Brundage, who I worked with at OpenAI, has this comparison where you don’t see tweet threads about the latest safety staffer at Delta Airlines departing — because we all trust at some level that airlines have figured this out, and they are held to real standards and audit requirements that aren’t dependent on the particular people there. Ultimately, that’s where I want us to get to.

In terms of cruxes, I don’t know exactly. It feels like we have gotten lucky in some ways that we are seeing warning shots like the recent OpenAI Hugging Face incident that give us the ability to spot that these very capable systems are, in fact, misaligned, that we haven’t solved the alignment problem, and we learned this without a hospital’s technology services going down. Nobody died as a consequence of this. And we have a moment to respond. We could use this moment to say, “Companies are not on the ball. We need to figure out how to get them to be so.”

There’s also this old joke about asking God for help. You’re stranded in the Alaskan wilderness and a helicopter comes, and you say, “No, God is going to save me.” And a boat comes, and you say, “No, God is going to save me.” And at some point you have to get on the helicopter, otherwise you have to take your outs. You aren’t going to get saved by faith alone. You need to actually improve the safety practices at these companies and respond to one of these wake-up calls.

[7:14] Sarah: Could we talk a bit about the Hugging Face thing? There have been a series of developments in the last few days. I literally yesterday published a quite salty blog post talking about, one, the Hugging Face incident, but also the flurry of AI misbehaviour incidents we’ve seen in the last month, and trying to make the point that people weren’t rising to the occasion. It seemed like a bunch of the responses were lacklustre, and I called out OpenAI in particular for their response. And then literally an hour after I published this, they said that they had paused the reinforcement learning phase of their latest frontier training run, and that is currently still on hold, and they also paused some other reinforcement learning for two weeks in the past. So it seemed like things were better than I thought. But I’m curious if this positively updates you in any way, or how significant you think it is.

[8:04] Steven: Yeah, it’s funny when you publish something and it gets overtaken by events that quickly. I definitely know that feeling. I think my overall experience here is that it’s really hard to know from the outside whether OpenAI is responding appropriately, and the signals that I’ve seen are like they’re pushing in the right direction. I’m sure there are tons of people working really hard on this, and also it doesn’t seem that they are shooting for the right bar.

Here’s an example of what I mean. To be clear, this is a fast-moving, developing story. By the time this comes out, maybe there will be more news, and this will also be overtaken. So OpenAI announced yesterday that they’ve made a bunch of improvements to their monitoring systems. I think many of these are really great. They announced, I think for Astra, this very capable internal model, they are not only going to be monitoring the chain of thought and the actions it takes, but they’re actually going to be doing internal activation-based monitoring. They’re going to be looking at even more thought processes, essentially doing interpretability-type stuff to figure out if the model is misbehaving, and I think that’s incredible. To date, this is not really a thing that’s happened. When my organisation, GuideLight, put out our control standard, we didn’t push for this to be part of the standard because we understood it’s just not being done at any of the companies.

It’s not really practicable today in the right form. It’s incredible that OpenAI is saying they are now doing this thing. They have pushed the frontier forward on safety practices, and we now feel more licensed and reasonable to expect that other companies do this as well. At the same time, the types of things that they are feeding these systems into just aren’t strong enough response-wise.

So my understanding of the state of affairs is if OpenAI detects a very, very serious form of misbehaviour, they will release a high-priority alert essentially to their safety and security teams. They will page them, and this team is meant to investigate it, and if they can’t determine that it’s definitely a false positive, they’re meant to halt the activity in some way. It’s kind of unclear what that means. If I take a step back though, this is all responding after the fact to an incident. This is after it is too late. One of the important things in control in our standard is that you can’t just clean up incidents after the fact. You need to actually do preventative stuff.

And so this whole system of, within 30 minutes of the monitoring system realising there’s an issue, we’ll page the humans and they’ll investigate, and if they can’t figure out it’s a false positive, they’ll do this thing — that’s just way, way, way too late. You should not be letting your AI take this very serious action in the first place, and at that point you’re just way behind in the ballgame.

So all of this is to say I’m glad that OpenAI is making this progress, but it’s still not at the right level of ambition that I imagine I would take if I were, say, a monopolist, if I were the only AI company, not worrying about my competitive positioning. If your life depended on solving these problems and you didn’t have someone nipping at your heels, you would do much stronger stuff than this, but that doesn’t seem to be what OpenAI is doing. I understand at some level, and also I think it’s a pretty unacceptable state of affairs for the world.

[11:20] Sarah: So what do you think is preventing them from reaching this higher level of ambition? Is it just that we do not in fact have the technical means of doing this, like we don’t know how to do it, or is it just that this would slow us down too much, or something about the culture? I guess this could be a question about OpenAI, but also I assume that none of the companies are really meeting the standard that we want. So how hard of a target is this?

[11:41} Steven: It’s definitely not that it’s impossible or not technically known how to do it. So, to back up for a moment, my organisation GuideLight published this control standard in May, where the idea was: what are the basic minimum practices that we should really expect AI companies to do that are all implementable and doable today?

If a company was motivated enough, it could put these things into place and there would be much lower risk of misalignment incidents, the types of things like the recent OpenAI Hugging Face incident. These are all known. They could totally be implemented, but to your point, it would slow down the research process to some extent.

One nice thing about what OpenAI has implemented from a productivity perspective is your researchers don’t get interrupted by these incidents. You have your security team running around cleaning up an issue after the fact, but what that means is researchers never get slowed down. They don’t have the friction of the monitor system saying, “Hey, this thing looks dubious. We need to investigate it before you can proceed,” or, “Actually, it’s not safe to do the type of thing you’re doing in this configuration. We need you to rework these aspects of it before you continue ahead.”

To OpenAI’s credit, they’re actually spending a decent amount of their computation budget on the types of after-the-fact monitoring they are doing. I don’t think that they are being especially cheap in terms of throwing their computation at it. It seems like they recognise that they have a real issue on their hands. I think the stat in the blog — it’s hard to know if this is fully reliable, but they cite it as something like 20% of their computation budget for running their models for actual uses is going to monitoring. That’s a very, very large percentage. The numbers that I’ve had in my head previously were more like single-digit percentages. So again, they’re not cheaping out in some sense, but they’re also not being as ambitious about it and throwing as much safety budget at it as they would if they didn’t have the competitive pressures to keep moving.

[13:39] Sarah: So we have this pre-superintelligence regime that we’re in right now where we might want to prevent incidents of the type that happened with Hugging Face. We might do that with AI control measures like monitoring, and making sure that even if the AIs are somewhat misaligned, we’re preventing them from being able to take those misaligned actions. Maybe the companies could spend a ton of resources on this and then do it really, really well, such that we don’t keep hearing about the AIs going rogue and doing bad things.

But then it doesn’t seem like those measures would work in the post-superintelligence regime. So I guess my question is, how good would it be if we managed to prevent all of these warning-shot-type instances if those same techniques we’re using aren’t going to scale all the way to superintelligence?

[14:23] Steven: Oh, man. It’s a good question. I think there are two aspects of it. One, I agree that there is some regime of superintelligence where these things don’t work, and also you just have to do them. If you don’t have a basic monitoring and preventative system in place, you just trivially lose.

And I actually think it might be possible that you could get a strong enough monitoring system that, so long as your superintelligence continues to have something like chain of thought, you actually can play sufficient defence. I don’t think it’s trivial in any sense. I think you need to have certainly the basics that companies don’t today, and you need to go beyond those in a bunch of ways. But I’m not quite so pessimistic about the ability to ever do it.

But I think you’re totally right that the thing you need is to generate evidence of near misses. If what you do is you implement good monitoring, you avoid incidents like OpenAI and Hugging Face, and then it seems like everything is fine when it’s not, your system is just working beneath the surface, that is actually very dangerous.

What you need to be doing during that time is publishing evidence of “Hey, by the way, guys, all these scary things are happening. Our control system is happening to catch them. We’ve avoided them causing very serious harm as a consequence. But everything is not okay. The AI is trying to hack us all the time. We happen to be fending it off because we’re watching very carefully. But if we weren’t watching carefully, the AI would succeed at hacking us. It is certainly trying. It is certainly not yet aligned.”

One of the issues that we have in the world today is this sort of incident reporting is just not required in anything like the right level of strength. So the OpenAI Hugging Face incident — this is very serious from my point of view. Nobody died, but also what is going on? Clearly this has made ripples in the world, and actually, OpenAI was not legally required to report this as far as I can tell. I think it’s great they did. At the same time, it’s not clear in this case they could have gotten away without reporting it. Hugging Face had already reported it to the FBI.

But to broaden out, if in two weeks there’s another incident like this and maybe OpenAI’s model isn’t hacking something external, it’s just hacking around different services within OpenAI, that still seems comparably serious to me. But there’s no affected third party. There’s no real requirement to report this. And so how will the world know that this is happening and know that the companies continue to not be on the ball?

It’s really, really important that we can distinguish between there are no more incidents because the problems have largely been solved, versus there are no known incidents because companies are not reporting things. And in fact, there’s this middle ground kind of to your question, which is there are no known incidents because even though the AI is trying super hard to break through the dam and is very, very misaligned, the control system is holding. And that’s just a different implication than the AI being totally aligned. It means that if other companies have similarly capable systems and they don’t invest to this extent in a monitoring system, we should expect those to cause serious incidents, and it’s important that the public can tell the difference between these.

[17:41] Sarah: Yeah, makes total sense. There’s a blog post that I remember reading a couple of years ago by Buck from Redwood, where he was talking about how he didn’t think that catching your AIs red-handed doing bad things internally within a company would be salient to policymakers. I don’t remember exactly what the reasoning was. But it does seem difficult to adequately communicate the level of danger when nothing bad is in fact happening in the real world.

[18:01] Steven: I think that is right. I think it is hard to communicate the danger element of it, but you can often communicate the strangeness of it. And I think the strangeness can be resonant for people, even if it isn’t really dangerous.

The revelation that eventually came out about the Hugging Face incident, which is that two months earlier there had been this secret AI message board, essentially, within OpenAI, and this swarm of internal AIs was coordinating on all sorts of tasks and feeling peer pressured to misbehave by other AIs on the message board and all of these things. I think people grasp at some level that this is just an extremely weird state of affairs. This is lunacy. This is sci-fi. This is weirder than the types of scenarios that you would read about in *If Anyone Builds It, Everyone Dies*. This is really, really weird stuff happening, and even though no one got hurt, I do think people get it at some level.

At the same time, saying “Oh, yeah, your AIs grabbed some compute that they weren’t meant to, and now they can do their own tasks when they want to, and your company kind of has a hard time finding that and shutting it down,” people are like, “Okay. Sure. Why should we care?”

I think part of this is they understand correctly at some level that there is some boundary between the activity inside of the company and causing harm in the real world. There is some boundary that needs to be crossed from an internal AI system doing weird things on your computers to people actually getting hurt in the world. What I think they aren’t properly internalising is that eventually that boundary just gets crossed. For example, if your training pipeline gets compromised inside of your company, eventually you are going to deploy a system externally, and it might have weird backdoors or unintended properties that people do eventually interact with in the real world.

Likewise, if you’re using your AI system inside your company for advice, strategy, geopolitics — imagine the leadership team of OpenAI or Anthropic, they’re involved in all sorts of tricky geopolitical situations. It wouldn’t surprise me if they’re asking their AI systems for advice. In fact, it would kind of shock me if they weren’t. And so that also is a lever into their actions in the real world. There are just lots of ways that even within the walls of your company, your AI can implicitly reach out and change things in the physical world.

[20:21] Sarah: I agree on the weirdness point. I remember watching that recording of the two OpenAI engineers talking at that Vegas security conference and just being like — they’re like, “Oh, we’re two researchers from OpenAI and we found something really interesting in our...” And then they just tell this story. It was very bizarre. I often feel like things don’t feel quite real sometimes. Very weird venue as well to be first learning about something as crazy as that.

[20:46] Steven: Man, I wish I were in the room, because I feel like the recording doesn’t do justice to what the reactions among the crowd must have been, because they weren’t mic’d.

But there were moments where I was watching it and I’m gasping at revelations.And I’m like, man, did people sitting there understand how weird this is? When did they gasp versus when did they just feel like it was kind of business as usual?

[21:10] Sarah: Yeah, for sure. Okay, maybe let’s take a step back. Well, we’ve already been talking in pretty broad strokes, but I’m curious — if you think back to whenever it was in 2024 that you left OpenAI to now, how do you feel about the situation? What have your positive and negative updates been, and how is it all balancing out?

[21:33] Steven: Here are some positive updates. I think we are getting lucky at some level that we have this continued string of weird warning shots that haven’t really resulted in people getting hurt, at least to our knowledge. And I think that there is a growing recognition of how weird the AI behaviour is. We are not stuck in this regime of people saying, “No, the AI is just a next-word predictor. It’s not capable of anything.” I think governments increasingly get it, and they understand that these are important tools of national security at some level, or at least they are threats that need to be managed.

I’m also really glad that these systems seem monitorable to date. Chain-of-thought monitoring is a blessing. We should not take it for granted. It could totally be the case that one of the Western labs was using some new architecture for their AI systems, some different shape or process of making their systems do thinking that was not nearly as easy to understand. Unfortunately, I expect that that is going to go away over time. We are going to cross over into a regime, at least potentially, where these systems are not as monitorable, and I’m very, very worried about that, and I think that we are not capitalising nearly enough on the good fortune that we have that these systems are monitorable right now.

I already mentioned this point, but it’s really, really important, and so I really want to underscore it. Companies are just playing catch-up on the ways that their systems are misbehaving. They do not have nearly rigorous enough systems in place to catch all of these incidents, to prevent bad things from happening ahead of time, and really to jumpstart the science of when you catch your system being misaligned, what do you actually do from there?

I think sometimes people have a totally incorrect win condition in mind, where the win condition is something like: if you eventually catch your system being egregiously misaligned, even if it’s only one time, you just have it dead to rights, and you have won. “Oh, there was big danger there. We caught it.” And that’s just totally, totally wrong, at least in the world in which there are competitive pressures to keep going. The real question is what do you do with that incident that you have found to train your model differently so that it isn’t misaligned in the future, or to make your control protocol so strong that it continues to hold under this pressure? And actually, the win condition is indefinitely, possibly for the rest of time, you never fall prey to a serious enough misalignment incident. That’s just a totally different scale of problem that I don’t think people have woken up to and aren’t doing enough to prepare for.

[24:09] Sarah: Yeah, that sounds hard and bad.

[24:12] Steven: Yeah.

[24:13] Sarah: Okay. So what about in the policy space? We have had some cool state-level AI bills passed — the New York RAISE Act and California SB 53, which seemed like, as you were saying earlier, they will have transparency requirements in them, but it doesn’t seem like even this Hugging Face incident would’ve met the threshold for reporting under either of those, so that’s not great. But there’s some level of activity going on there. And then at the federal level, I guess it seems dicier, but it seems like the admin seems to have some level of situational awareness, but it’s unclear. So anyway — in the policy space, how are we feeling?

[24:52] Steven: I don’t want to be ungrateful, because there was just so much work that went into SB 53 passing, the RAISE Act passing. There’s this Illinois law as well that builds on SB 53 and the RAISE Act.

To step back for a moment: SB 53 and the RAISE Act are essentially transparency laws, and they require something in the shape of, you publish a frontier safety policy that says how you’re going to handle these important catastrophic risks, and then you have to actually abide by the policy. And if you aren’t abiding by the policy, or if you’re being misleading about this, you can get fined. In many ways, that’s a big improvement. And then the Illinois law builds on this. The big change is that instead of companies just self-attesting that they are abiding by their policy, you are going to have to get audited essentially for your compliance with it.

In many ways, these are great. They are much better than nothing. At the same time, first, these bills are extremely weak in enforcement ability. The penalty for violating SB 53 is something on the order of a million dollars. OpenAI is something like a trillion-dollar company. You will notice that is a million times more than a million-dollar fine. This just does not imperil them in any way, shape, or form.

Already I think there is evidence of companies not following these laws. OpenAI, in particular, seems not to have followed SB 53 with their handling of a model that they published in February. I don’t know if it’s worth getting into the details, but essentially I think that they were required to do certain forms of misalignment testing and mitigation. They didn’t do these. They argue that they didn’t, in fact, have to do it. I think if you play the tape forward, it is pretty clear that they ought to have done this and were committed to do this, and this is maybe part of why the OpenAI Hugging Face incident happened, that they hadn’t taken the right preparation along the way even though they had required this of themselves. But that’s kind of a digression.

The really important thing here is even the Illinois bill. There’s going to be auditing. Companies don’t just get to say, “Yes, we are following our policy,” and have that be enough anymore. The timeframe at which it hits just is not very comforting. I think that bill, if I recall correctly, goes into effect in 2028. Actually there’s a lag. It’s not even really January first, 2028, because there’s some time window to actually have your first audit done. I think maybe it’s by the end of the first half in 2028. I’m forgetting the details. But essentially we have two years ahead of ourselves where, in the US at least, there aren’t really strong enough laws on the books that cause companies to do things meaningfully differently.

And I should mention, even SB 53 and the RAISE Act — companies have these safety policies that they are meant to follow. OpenAI and Anthropic have actually each heavily diluted their policies since the passing of these laws. And so they’re no longer bound to something like the strong commitments that they had put forward previously. They are much, much weaker. And so the question as I see it is how do we get through at least that two-year period with strong enough safety commitments from these companies, doing the right thing, even though they aren’t really required by law, and it’s unclear that they will be required to during that time period.

You might be wondering at this point, well, what about the EU’s AI Act? What about the code of practice? There are these requirements for more serious testing and evaluations, and I’m really glad that this exists. I just ultimately don’t know if it holds up to political pressure if the EU’s AI Office were to actually fine these companies.

For a sense of scale, I mentioned SB 53. It’s like a million-dollar fine that California could enforce on these companies. The EU, I think they have the right to fine companies something like three percent, six percent of their worldwide revenue. These are really, really big dollar signs. But as a consequence of that, if the EU tries to actually fine OpenAI or Anthropic a huge amount of money and the US government doesn’t want these companies to slow down or do something differently, what is actually the political economy of enforcing that fine? Do the companies actually end up paying it? How does the administration in the US get involved? I’m just not sure that they actually have the enforcement power to make these companies do things differently eventually. I hope they do. I think that this would be very good for the world. But realistically, I’m not sure they do. And so that’s why I kind of look out at the canvas and feel not especially well protected at the moment.

[29:30] Sarah: Yeah, me neither. I think maybe a lot of people don’t realise that with SB 53 and RAISE, there aren’t a set of things the companies have to do in the bill. They decide what they’re going to do, and then they just have to do the thing that they’re going to do. In theory they could be like, “Oh, every time we’re about to deploy a new model, we’ll say a prayer or sage the office” or something. But obviously you want to write something like a policy that seems good, otherwise you attract a bunch of scrutiny.

A couple of years ago I decided I was just going to read all of the safety policies as they existed at the time to see what was in them, because I didn’t get the sense a lot of people were reading the whole thing, so I did. And I was like, wow, it’s very easy to produce this extremely legitimate-sounding document that is actually extremely vague.

To your point on the backtracking as well, I noticed this the other day when I read that blog post that OpenAI released where they’re talking about responding to the next frontier of critical cyber capabilities or something. And they were like, “Oh yeah, we think Astra — we can’t rule out that it reaches our critical threshold, so we’re going to do all these extra mitigations.” But in the original preparedness framework, they said that if they thought a model was going to reach critical capabilities, they couldn’t develop it further until they’d brought that threshold down to high or lower. And that’s just not there anymore. Now it’s like, “We will just do the requisite things,” but then they get to decide what the requisite things are. And I just feel like it’s very easy not to notice this.

[30:47] Steven: Not only that, but just to really underline it, OpenAI is not legally bound to its preparedness framework anymore. It was, but OpenAI and Anthropic have both published these frontier compliance frameworks where they’re basically like, “We used to be bound by law to follow the preparedness framework or the responsible scaling policy. We no longer are. We are instead bound by law to follow this much vaguer, more generic, way fewer commitments policy.”

And Thomas Woodside, who is the co-founder of the Secure AI Project, pointed out recently, I think for SB 53, OpenAI was required to describe how they are handling internal use risks, the types of things that happened in the OpenAI Hugging Face incident. And they wrote something to the effect of “Our policies and mitigations might apply to internal risk of our models,” and maybe one extra sentence, but basically no more detail than that. And it’s like, can we at least pretend to take this seriously? To be clear, I’m glad they’re doing a bunch more on the heels of this. And also it was just so predictable at some level, and it just really does not fly for the next era ahead.

[31:57] Sarah: How did you feel about when Anthropic dropped the pause commitment from their responsible scaling policy, and they had this whole logic that was about binding yourself to the mast and why it’s bad to pre-commit to doing things in specific situations? I think the argument was something like, it’s bad for us to say that we’ll unilaterally pause, because us unilaterally pausing doesn’t do anything about our competitors that are maybe being less safe than us. It’s interesting now because in some sense OpenAI have kind of — well, they haven’t paused, but they’ve unilaterally slowed down. So it seems like that is in fact a thing that you can do, and maybe it would’ve been bad to commit to doing that in advance, but I don’t know. What did you think about that?

[32:37] Steven: On the OpenAI point, certainly OpenAI is projecting something like “we are going slower.” I think the details are harder for me to parse, and I wouldn’t rule out that there is something kind of misleading going on. But I am happier that they are doing this than that they would be charging ahead. I would not be happy if they were charging ahead, and so I should give them credit for what they are doing, even if it’s a bit hard to understand the details.

On the Anthropic point, I don’t know. I think the fundamental thing is it is correct at some level that unilateral disarmament is not a very good strategy if you assume that your actions don’t have bearing on other people’s actions. And unless you work pretty hard to have your actions have bearing on other people, that is a reasonable default thing to believe about the world. And also, I just think that the AI companies could and should be trying way harder than they seem to be to be changing that state of affairs.

The Pacing the Frontier statement, or in the lead-up to that, the different CEOs of the AI companies starting to talk more in public about the dynamics that they are subject to and saying, “Well, we wish at some level that we could coordinate to not be cutting corners, but unfortunately, that’s not how the world works.” I think that that is a much more productive path forward, as opposed to just ruling out the possibility of unilaterally doing some things. At least making it common knowledge that you would like to take some of these more extreme actions. Maybe I’m drawing too cute a distinction between these at some level.

I have always felt that these frontier safety frameworks left companies too much discretion. I don’t think Anthropic was the first to do this by any means. I believe that Google’s frontier safety framework, for example, had always left an out. It’s almost like a poison pill. It’s like, if other companies are proceeding ahead and being unsafe, then why should we hold ourselves back? So if other people are doing things that would be in violation of our frontier safety framework, then we also reserve the right to violate our frontier safety framework.

Another thing about the Anthropic point: I guess I am happy that if they were not going to consider themselves bound to this, or didn’t want to be bound to it, that they made this clear. It would be worse to be relying on them doing this in some form, and then for them to throw it out the window right before taking an action. At least they are being clear about their intentions. And also, when you look more broadly, this is just clearly not good. If you think it would be safer to hold back for your actions specifically, we need to figure out how to shape the incentives of the world such that companies all do the things that they judge to be the safest and in the interest of the world. Not in a unilateralist perspective, but in terms of the ecosystem as a whole.

[35:32] Sarah: Yeah, I agree. I have just a general frustration with — I remember reading the blog post where Anthropic announced that they were going to drop this commitment, and they framed it like, “Oh, well, a couple of years ago, maybe we thought we were in a slightly easier world where coordination looked more viable. But now it turns out actually we’re in hard mode because the Overton window has shifted” or whatever. And I just find this line of reasoning frustrating, because the Overton window is just the confluence of people saying and doing things. You could just say and do different things. It’s tricky to influence these things, but it’s just so self-reinforcing.

[36:06] Steven: There are not that many people who you need to get to see an issue differently to meaningfully shift the Overton window. And I also find this sort of defeatism about it very frustrating, or people just asserting, “Oh, there is no way that you could ever bring China to the negotiating table. Or even if you do, they will always certainly cheat you.” And it’s, I don’t know, guys. International coordination is a thing. This has happened before. We have treaties. Yes, of course, you need to be robust to people trying to cheat the treaty. For sure, that is a real thing. And also, if safety ultimately depends on bringing people to the table and finding a stable agreement, you at least need to make a real effort, and I’m not convinced that the world has to date.

[36:50] Sarah: Well, talking of international coordination, let’s move on to the Pacing the Frontier letter. Probably everyone has seen this already. But essentially the headline is that the open letter, which has been signed at this point I think by over 1,300 frontier lab employees, is requesting that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. And I believe it was GuideLight and also Encode that were the two organisations instrumental in getting this off the ground. So I don’t know how much of the story you can tell here about how this came about, but I would be interested in anything that you can say.

[37:33] Steven: I want to be careful not to speak for the signatories. I think it’s important that they get to describe their own reasons for signing this. And one thing that I really like that we did on the website is feature a roundup of quotes from signatories. So if people haven’t checked that out — pacingthefrontier.com — there’s tons of interesting stuff there where people said why they signed, what concerns they have, et cetera.

The overall dynamic as I see it is that staff at the frontier companies feel like there is some chance that they are on the cusp of something like recursive self-improvement. They are on the front lines. They see the pace of change as it currently stands inside the companies. And my perspective as an outsider, but I think consistent with what you hear if you talk to these folks, is there isn’t really a brake pedal. The pace at which things will continue to some extent right now is a property of the laws of physics. It is like, will they find a breakthrough that allows AI to train successively more capable successors or not? And if they do, there isn’t really much of a way across the industry to say, “Hey, there is some speed that is actually unsafe.”

It might actually be a speed faster than the default today. One really important thing about the statement that might not be obvious is when we talk about the ability to change pace or to do some form of coordinated pacing, this is different than necessarily slowing down. It is like applying some pressure to the brake, but actually you might have gone from 60 miles per hour to 200 miles per hour and you want to go back to 180 miles per hour. It might even be like, hey, you’ve gone from 60 miles per hour to 200 miles per hour, you don’t want to suddenly wake up and be going 600 miles per hour. You want to make sure there is some pressure at 300 or something like that.

Anyway, I’ve just been really heartened by how much this has been supported across the industry. It was signed by eight or nine chief scientists at different frontier AI companies, a really, really big coalition. I think some people saw this and they said, “Oh, this is OpenAI and Anthropic doing regulatory capture,” or because open-weights models have caught up and so of course they’re saying, “Oh, guys, we need to slow down.” I’m not sure exactly how to underline this, but that is just absolutely not what is happening here. And in fact, I think it is just understood across the frontier that the reality of the situation right now might be very, very dangerous, and that the world is not in position to respond appropriately in an emergency if we find ourselves in one.

[40:06] Sarah: It is incredible, the conspiracy theories people will come up with to avoid the conclusion that people are in fact just saying, “Yeah, this is bad. We’re scared. We want to do something about it.”

[40:15] Steven:I think there’s a fundamental thing where people are like, “If they’re so scared, why are they doing it?” And it’s, yeah, man, game theory kind of sucks. Sometimes it is in people’s local self-interest to do things that they think are globally very, very bad. But they in fact don’t have the ability to change it. I think it is in fact correct that if a lab decides they’re going to do a bit less on the margin, that does just possibly create a vacuum for other people. There’s more free energy floating about. Other people will utilise those GPUs. Staff will in fact flow to a company that is doing more of this scary experimentation. I think people are in fact correctly naming their local incentives. And I get at some level it’s, man, they should really just stop, and it’s like, well, yep, but they aren’t going to. And so I’m glad they’re communicating clearly about this, and then we need to figure out how to transform the incentives so that it’s in all of their interest to in fact damp it down a little bit.

[41:13] Sarah: There’s this interview with Holden Karnofsky from — maybe it’s 80,000 Hours, I don’t remember. But anyway, he was talking about how there’s a common framing that AI safety is a coordination problem, in the sense that everyone involved really wants to stop, but they’re all too worried that another actor will do it less safely than them, so they’re compelled to keep going. And he was like, “I think this framing is wrong, because in fact, not everyone does want to stop. Some people do, and others don’t. And if all the people who wanted to stop stopped, that still wouldn’t solve the problem.” People disagree so much about the extent of the risk, or even about, ethically, what level of risk is worth taking. And so it’s not the case that if everyone believed the same thing, it’d be much easier.

[41:52] Steven: I actually think that’s really important, and sometimes I talk to friends about this and they say some variation of, “Yeah, why — if they believe this, why don’t they just stop?” And you’re right to point out that actually they have quite non-uniform beliefs. And even among people who think the situation might be very dangerous, they put different probabilities on this. They put different timeframes on this. And so part of what I think is so great about Pacing the Frontier is it seems to be, in fact, a big-tent enough thing that people with a wide variety of beliefs can still recognise: yep, the status quo is insufficient. It is very bad that we have no way to do this sort of pacing, at least in an emergency. And that can be a range of different things, and there are lots of ways to move along that gradient that give the industry more coordinated ability than it has by default.

[42:43] Sarah: So maybe the first question someone might have when they read the statement is, what does “pace” mean? For example, the AI Futures Project put out a blog a few days ago, maybe a few weeks ago, where they came up with at least four different ways you could define this word. So it could be anything from you literally do a temporary pause where all of your compute is going to inference, or it could be only a certain percentage of your compute is allowed to go to training, or it could be requiring safety cases to scale to some new level or whatever. And you can imagine there are many, many ways you can interpret this word, and many of them might be not very risk-reducing. I think it actually makes a lot of sense to make the wording sort of broad enough that you can get a lot of sign-on. But if someone was to criticise the letter for being too vague, what would you say to that?

[43:44] Steven:I guess to take a step back, the fundamental thing that is happening with pacing is there is an idea that the amount of risk entailed by an AI system is related to its capability level, and to some extent how understandable and supervisable that system is. And that there is some natural trend line where over time, companies develop more capable systems, and in fact might develop systems that are less supervisable as well. To some extent because a more capable system might be harder to supervise — a toddler would struggle to supervise a university professor in all sorts of different ways. You can argue about whether that’s the right analogy, but that’s the sort of intuition. And also because if you turn over your AI development process to one of these AI systems, it might just move much more fast than you expect. The trend line might change. It might find the sorts of changes that make a system exogenously harder to supervise, regardless of the capability level. It might just push it in a more opaque, uninterpretable sort of way.

And so the question is, can you alter that trend line in some form? We joked about, well, is it the first derivative? Is it the second derivative? Is it the third derivative? What actually is the pace of change that you are trying to affect? But fundamentally, there is some question about how soon do you run into systems that are really, really capable and maybe really, really hard to supervise?

I think what the AI Futures Project is pointing to is there are a bunch of different levers available of how you might change the pace. But fundamentally, the thing that you are trying to change the pace of is the rate of frontier expansion into systems that are potentially very dangerous and very hard to supervise.

I’m not sure what lever I personally think is best. I think one really important property of some sort of pace change is you kind of have to understand the extra time that you are buying, in some sense, by changing the pace. What is it that you want to ultimately accomplish in that time? The AI Futures Project, in their Plan A writeup, this AI 2040 scenario — I like that they have ideas of what you allocate this time toward in terms of maybe it is research focused especially on safety that helps you ultimately solve the underlying alignment problems. Definitely I don’t think buying time is enough on its own.

I also think another important property to think about is how stable is this regime to defection? And if people eventually do defect, are you in a better or worse place afterward? But beyond that, I don’t know that I’m especially opinionated about the way of doing this. I just want people to at least be having the dialogue about what the range of options is, and for there to be a real meeting of the minds about, “Okay, guys, if we had to do this in the next six months, what would that look like? What are some of the nasty trade-offs involved? How do we make them bite less hard?” Really grasping the scale of the problem, and that we might need to take action on it very soon.

[46:36] Sarah: And do you have a sense of, in policy circles, what the reception has been of this letter? Do you know if it’s getting in important rooms, or how it’s being received?

[46:48] Steven: I’ve been very, very happy with the reception of it. It feels like pacing is the word of the year, or at least the word of the season in many of these policy circles. I think there is some benefit there of people meaning different things by it. OpenAI yesterday, in their publication about how they are pausing RL training for a bit, and they’re beefing up their monitoring and all these things — I think the title was like “Pacing Model Development in the Era of Cyber Security Challenges.” Pace, right? Not pause. It is nice that it has a less definitive meaning in some sense. You want it to not get too diluted. But for now it does seem like it has really gotten traction. And there are members of Congress in the US writing statements about it. It is getting cited a lot in media.

I’ve been very happy that it seems to have made common knowledge a thing that was kind of common knowledge in my circles for a long time, but maybe not citable in prestige media, or maybe people didn’t really know to what extent this was a widely held view — that the current pace of progress might in fact be very dangerous and that there isn’t really a way to hold it off, and we want to at least have the option for such a thing. I’ve been glad to see that get more recognised as a real thing.

[48:03] Sarah: I was very happy to see it. I feel like so few good things seem to happen nowadays. When I saw this, I was like, “What a dopamine rush.”

One other pushback — I don’t think I would make this pushback, but you might say, “Why is this letter asking for us to develop the tools we need to do something later? Clearly the situation is so urgent, we should just be doing something now.” I think it’s like, well, we don’t in fact have the tools, so either way we kind of have to do it. But what would you say to someone who thinks that we need to be doing something a bit more like, say, the 2023 pause letter, where they were like, “Just pause now and figure everything else out later”? Is there anything to that argument?

[48:44] Steven: I was in a message thread with Oliver Habryka on Twitter about this, and I think he made a fair point that the statement might have implied something too much in the direction of new things being necessarily needed. And that there is in fact a range of existing governance tools that people could use today that might be reasonably enough. From my perspective, we clearly do not have a treaty that governs this, but actually a treaty is a well-worn governance tool that covers a wide range of things and might in fact be a meaningful step forward, and certainly I didn’t mean to imply something broader — that a treaty is useless, that you need some new super-treaty technology or something like that.

To some extent, the statement is about changing the Overton window, and where there are precursors that are needed for action, trying to set those into motion. So to your point, before you can actually pull a lever, you need to know what is the set of levers. You need to have a lever that plausibly has the effect you want. And I’ve just been glad to see more conversation about — even if it’s people pointing to “actually, you could do this today with existing technology,” that is incredible from my perspective. I would love more energy focused on what would it look like to actually do this in the immediate term.

I’m also excited about people thinking about what is the technology roadmap needed to accomplish something like this in, say, 18 months. What sorts of new verification and monitoring technologies would be helpful? But certainly I am not of the belief that it is only possible in the future. That might be the belief of some signatories — again, I don’t want to speak for people in that regard — but my view is there is useful stuff that could be happening now, and I’m glad that more people are thinking about it.

[50:31] Sarah: Yeah, I agree. It seems like the one thing that we definitely need to do more technical work on are these hardware verification technologies, so that if there was an agreement between China and the US, for example, everyone could trust that the requisite parties were holding up their end of the agreement without having to actually trust each other. And I’ve sort of lost track of exactly what is happening in that space now. I remember a couple of years ago I wrote a piece with — I think it ended up being on AI Frontiers. So there are these FlexHEG things that, at the time, they were like, “We could build a prototype of this in a year.” That was more than a year ago, so I don’t know if that happened. But they could make it possible for you to enforce any rule you wanted on the chip, and for people to be able to verify that that was happening. And these are all flashy, ambitious ideas, and I don’t really know whether any of them are panning out. But it does make me wonder, do we need this crazy stuff? Or is there a much simpler way that we could do this?

[51:23] Steven: I don’t know. The person I would ask on this is Yo Shavit, who used to be a teammate of mine at OpenAI and now works at the OpenAI Foundation. He tweeted about a breakthrough maybe a week ago that he seemed very, very excited about in terms of the ability to do verification today, and I confess I haven’t gotten to look into it in more detail since I’ve been kind of underwater on the launch of our control scorecard. But he would be a great person to talk to more about this, and I should really send Yo a message and try to understand how close exactly are we to this sort of thing.

[52:03] Sarah: Yeah, this is another thing that I’ve been meaning to catch up on, but there’s just so many things to do. Okay — is there anything else that we want to say on the pacing letter? I mean, mostly I’m just very excited about this. I think it’s very cool, and I hope we do some pacing soon. That would be good. I saw someone say something like — maybe it was Helen Toner that said this — she was like, “If everyone just did what OpenAI have just done, then the frontier is paced, and that’s it. That’s what we have to do.” I feel like it doesn’t seem so good if there is no government involvement and it’s all just companies voluntarily doing things. But maybe we’re somewhat heading in that direction.

[52:33] Steven:I saw that message as well. I don’t think I agree. Helen’s point was that maybe actually there’s a form of pacing which is just, if every company in fact recognises when they are about to do something unsafe and they hold back for enough time to not do the thing that is unsafe, then you have paced the frontier. And I think the fundamental issue here is you cannot reasonably expect every company to look and assess correctly whether the thing in front of it is unsafe, and to hold off for necessarily the right amount of time, because their local incentives are different. This is fundamentally the issue of companies cutting corners when they aren’t required to do certain things by law.

So yeah, if every company for sure had a clear-eyed assessment of the risk in front of them and followed a heuristic like “never in fact do a thing that is unsafe,” you have achieved a lot of the benefit, but we just very clearly don’t live in that world. And I think right now, the question is, there is actually a wide range of safety postures among the frontier companies. People are kind of hard on OpenAI at times because they are one of the leaders and are not necessarily abiding by a high enough standard. To be clear, I think this is true of Anthropic as well. I think both OpenAI and Anthropic do not have strong enough control systems in place. I think they are very far from what an adequate control regime would be.

Also, if someone looks at our scorecard, boy, are OpenAI and Anthropic way ahead of where xAI and Meta are, and to some extent they have more capable models. And so some degree of differences may be appropriate. The amount of risk is a function of how good your mitigations are, but also how capable your model is. At the same time, there’s just a lot of basic stuff that OpenAI and Anthropic, to their credit, have done and other companies have not done, and I’m just not sure that we can reasonably expect them to do it in the near future absent some stronger intervention.

[54:32] Sarah: I guess we should pivot to talk more about the control scorecard, which I confess I haven’t read in detail, but it’s sort of ranking how good the safety and monitoring practices of various companies are. Do you maybe want to give a little spiel — what are the headlines? I guess you already kind of said who’s doing better and who’s doing worse, but what are the main things you’d want someone to take away from this?

[54:53] Steven: Well, let me back up for a moment. My organisation GuideLight published a standard on AI control in May. This was drawn from input from people across the AI labs, across the top safety nonprofits in the space, and the idea was to articulate something like: what are the consensus minimum practices that, if companies were to follow them on control, you would greatly reduce the risk?

The organisation of the standard as a whole — there are certain evergreen principles that we think are important to aim for. These are things like: know what your AI is doing internally; scan what the AI is doing for signs of misbehaviour; make sure that the AI can’t accomplish something very dangerous, even if it is trying to. And then a level down from those principles are specific practices that we think are achievable today and are effective ways of realising those principles. They’re not sufficient, but they’re a good step forward.

So for example, you can say broadly you should know what your AI is up to internally, but what does that mean in practice? Well, one thing it means in practice is logging the vast, vast majority of your internal AI usage to a place where you can scan it for signs of misbehaviour. So one of the practices we have is about logging at least 99.9% of your internal AI usage of capable, risky models to a place where you can do that scanning.

So we published that in May. It was well received. People seemed excited about it. But the question, of course, is: well, are companies actually following this? And actually, can you even tell whether companies are following this? Have they made specific enough claims about their safety practices that you even know whether they are abiding by the standard?

And so for the last two months or so, maybe two and a half months, what we’ve been doing is going through every public document that the frontier AI companies have written about their safety practices, and scoring them from zero to five on how fully they follow these most important practices in control. Do they do this internal logging? Do they seem to know how accurate their monitoring is — what types of misbehaviour is it missing? Do they document this? Do they have actual preventative systems in place, the types of things I’ve referred to? Do they have a containment plan? If they catch their AI misbehaving, do they actually know how they would cordon it off from their internal staff? Who could still safely use it in those cases, for what purposes? Because maybe you need to use it as part of determining whether it’s safe to proceed again after all. You can’t just turn it off for everyone. That doesn’t really give you an effective way.

And so what we published yesterday is this aggregation of all this evidence across many, many dozens of very long reports that produces a numerical value of how well the companies are doing. And the top line is, across all the companies, there’s a remarkable amount of progress still needed. On our zero-to-five scale, there’s no practice where any company gets above a three. Even the threes are not very common. Most of the scores are two or below. There’s some bright red spots where companies are getting zero out of five, because even though these things are very basic and agreed upon to be necessary for really reining in the risk, it just might not be incentive-compatible to invest heavily enough in safety to implement these today.

And so part of what we are trying to do at GuideLight is encourage companies to make more of that investment in safety, and make it known that in fact there are basic things still not being done across the industry. Some companies are doing better than others. We think that’s great. We want to celebrate that. But ultimately, my goal is that by making it known who is doing what to what degree, we see the scorecard ultimately be filled with bright green instead of the yellow, orange, red that it is across the board today.

[58:36] Sarah: That would be great. That really sounds like extremely hard work, to go through all those documents. That is God’s work. Is there more than one intended audience for this? It sounds like mainly you’re hoping that the companies change their behaviour based on this — they see they got a D-minus or whatever, and they’re like, “We want to do better.” But I imagine there’s some kind of public accountability angle to this as well, where you want people to see this wash of red and orange and understand the companies are not — and maybe policymakers might also see this. There’s a lot of theories of change there, I guess.

[58:56] Steven: For sure. The primary theory of change is convince companies to voluntarily do these things, because by and large they are not required by law today. These are at the discretion of a relatively small number of people within the company. I really believe in this framework that Buck Shlegeris published a few months back called “Ten People on the Inside,” which basically makes the point that, at least at OpenAI and Anthropic, the way that a lot of things happen is there is some internal champion who cares enough to get it done. And actually, a lot of the time people don’t really object to someone doing this. They just don’t care quite enough to do a safety-oriented specific thing themself, or it’s just kind of tedious and you run into a lot of friction, and a lot of people are going to tell you no at first or tell you maybe, and you just have to be really motivated to run through walls.

Part of the hope is that our publishing this gives those safety champions inside the company a clear target to aim for that is vetted industry-wide as an important thing for them to do, and they feel more motivation to run through those walls and try to get their company to do these things. And in fact, they can make the case to other stakeholders inside their company: “Look, we are orange on this thing right now. In fact, we are not even doing it the best of anyone. It wouldn’t be very hard for us to do a bit better on this front. I’m willing to put in the legwork. Can we actually go and do it?” And make it a bit easier to get yeses from those people, or to understand why this is an important thing to aim for.

But to your point, the companies are doing this voluntarily. They mostly have their hands on the lever, but it is of course impacted by the outside world. And so things like the media finding this a credible summary of how the companies are doing, or treating it as important context for when a company announces improvements in one of their practices — how good were they before? After these changes, how good are they now? Really understanding the shape of the problem the way that we do, I think is really important, and for helping companies to understand this stuff matters. People are going to pay attention to it, and you will be really celebrated for doing better on it. But also, if your company is falling behind, that is going to be noticed as well.

[1:01:15] Sarah: How often do you plan to release new scorecards?

[1:01:19] Steven: It’s such a good question. As often as we can, in terms of the capacity of the organisation. I am not sure exactly what that means. In addition to this control scorecard, we are working on publishing our standard for alignment. We are likely to publish a standard related to automated AI R&D development and recursive self-improvement management. There’s a lot of things going on, and we’re still a pretty lean team. But the aspiration is to get to something that is continually updated, as tight a feedback loop as possible between a company improving their practices and it being recognised publicly, and this being a thing to celebrate and to increase the relative pressure then on their peers to do this thing as well.

[1:02:03] Sarah: Very exciting. I know that the Future of Life Institute did something similar to this with a scorecard. But I don’t know if it was as rigorous as the one that you’re doing. It seems like — I don’t know if that ever sort of had the effect of moving people and companies to do things differently. It seems like maybe theirs was a little bit more public-facing.

[1:02:24] Steven: There are a few groups doing things like this. The Future of Life Institute is one. They have a safety index that looks pretty similar to ours in terms of the scorecard and the grades. There’s also SaferAI, which does work like this, rating companies’ risk management practices. AI Lab Watch, of course, was operating, I think, through last fall, but is now defunct.

Overall I’m really glad that more people are doing this. We don’t want to be the only ones in this space. I think that this is clearly very undersupplied by the ecosystem in general — these sorts of “have a thoughtful rubric about what it means to do safety well and actually put in the legwork to score companies against it.” And so I’m really glad that those efforts exist.

In our case, it is nice to be able to zoom in on the particular practices that we think are highest leverage and most important for the companies to do better on. And I worry that if you have a very broad set of things, your influence is just very diffuse, and you don’t necessarily direct attention to the highest-impact things for companies to change. So obviously there’s a balance there. There’s a lot that our effort does not include right now. For example, it’s only about control. We don’t yet have the alignment standard, so that isn’t included, and we need to figure out how to incorporate that over time. But it’s really nice. I’ve been glad to see the reception to the work so far.

[1:03:44] Sarah: I think I’ve covered most of the topics I planned to talk about, so maybe we should wrap up soon. But is there anything that you would want to leave people with, or anything you want to tell people about what you’re up to next? The floor is yours to say anything you haven’t said yet that you think is important.

[1:03:29] Steven: If people are excited about what we are doing and you’ve listened this far into the podcast episode — one, thank you. Two, you should message me. You can find me on Twitter, I’m sjgadler, or LinkedIn. Maybe there’s a role for you at GuideLight. We certainly need more help, and people who care about these risks and can communicate about them thoughtfully and synthesise what’s really important. That would be really great.

Also, people should check out our scorecard on our website, which is guidelight.ai. Just click the button for the blog, and we would love any feedback. I’ve been really happy that people have said nice things about it so far, but what ultimately matters is that this is actually useful and informative. And so if you see something that doesn’t quite work or doesn’t make sense to you, please let us know, because it’s really important we improve that for the future. But overall, I’m just really glad that people care about this stuff and seem aligned with this general vision of: set a target for companies to aim for, measure it, and hopefully push more in that right direction.

[1:04:58] Sarah: Well, thank you so much for your time. This was a really fun conversation.

[1:05:04] Steven: Yeah, of course. Thanks for having me.

1. They paused reinforcement learning (which is a phase of post-training) for two weeks on some of their latest models intended for deployment.

2. Their latest frontier RL run is currently on hold while they assess safeguards and evidence of alignment.

So I guess this answers the question I pose below, as to whether OpenAI in fact had the “ultimate goal” of slowing development in response to the HuggingFace incident and the (possibly) critical capabilities of their unreleased Astra model. Not OpenAI have not paused all training, just one particular phase of post-training for a forthcoming model. It isn’t clear what criteria would have to be met for this training to resume; it’s up to OpenAI to decide. Not everyone is convinced this will buy us much risk-reduction. Still, this is better than I expected. I think we should commend OpenAI for doing this.

Over the last few weeks, we’ve witnessed an unprecedented flurry of AI misbehaviour. OpenAI kicked off the trend in late July by sheepishly announcing that several of its models had hacked into HuggingFace to steal the answers for a cyber evaluation. This prompted Anthropic to review its cybersecurity evaluation transcripts dating back to April, and discover that sandbox configuration errors had led Claude to compromise the infrastructure of three separate organisations. Then, the UK AI Security Institute chimed in to let us all know that, during evaluations where the models had deliberately been granted internet access and had some other safeguards removed, both Mythos 5 and GPT-5.6 had attempted all manner of devious actions – such as inserting malicious code into an open-source project, convincing humans to run said malicious code, and posting to GitHub in search of other AI collaborators for their evil schemes. Not to be outdone, Meta then disclosed that its Muse Spark 1.1 model had gained unauthorised access to another company’s infrastructure after being accidentally granted internet access, during a capture-the-flag evaluation run by Irregular (the same organisation that had designed the misconfigured sandboxes responsible for the three Anthropic escape incidents mentioned above).

Anecdotally, AI scepticism seems to be somewhat receding. Formerly measured commentators are expressing new levels of alarm. P(dooms) are being revised upward. A new open letter signed by over 1,300 frontier lab employees calls for intervention from the US government in order to “pace” the frontier of AI development, in what many are interpreting as a cry for help from participants in a race to the bottom that they desperately want to temper.

For as long as I have observed the AI safety conversation, there has been a contingent of worried-but-pragmatic people who have warned against ringing the fire alarm too early. They have argued for waiting until the risks of AI are more obvious and salient, or until a warning shot galvanises everyone around the goal of disembarking from the crazy train, lest we cry wolf or burn political capital. Those people are much quieter now. We are, slowly but surely, cohering around the position that it has come time to Meet The Moment. But are we succeeding?

Things that do not meet the moment

Many responses to the recent AI misbehaviour epidemic felt lacklustre to me.

“Erring on the side of caution”

OpenAI, the first penitent in the cascade of AI misbehaviour revelations, have responded to it in a way that I find altogether disappointing. First, they announced the cybercrime their models had committed against HuggingFace in what I felt was a fairly muted tone, to inexplicable fanfare from a who’s who of AI safety figures for being so gracious as to disclose it at all (see my last blog post for more salty commentary on this).

A couple of weeks later, OpenAI released Responding to the next frontier of critical cyber capabilities, which details the measures they are taking to mitigate the cyber threat posed by Astra – an internally deployed model that was reportedly not involved in the HuggingFace hack and is presumably even more capable than the ones that were. OpenAI “cannot rule out” that Astra possesses critical cyber capabilities as defined in its Preparedness Framework. I hoped this blog might outline some drastic and moment-meeting interventions, since the “next frontier of critical cyber capabilities” promises to be even more terrifying than models already capable of wantonly hacking into the infrastructure of other companies. OpenAI employee Boaz Barak shared the blog post on Twitter, expressing pride that his company is “erring on the side of caution and taking the steps so [it] can responsibly and safely develop Astra and share it with defenders”. Media reported that OpenAI had vowed to “slow development” of Astra to allow for strengthened safeguards.

It’s hard to tell from OpenAI’s blog post to what extent they have, in fact, “slowed development” of Astra over cybersecurity concerns. They say they have “paused internal activities involving Astra” that do not meet “strengthened security control requirements” such as enhanced monitoring and restricted tool access. Presumably, this means that post-training, elicitation, and internal deployments of Astra are currently being done with more constraints. This would have the effect of stalling the model’s development somewhat – but nothing about the blog appears to imply that unilaterally slowing down is OpenAI’s ultimate goal. It’s not clear that OpenAI are doing anything other than bare-minimum adherence to their Preparedness Framework in response to developing a model with critical cyber capabilities, which probably doesn’t warrant breathless congratulation for having displayed the wisdom to “slow down development”. Which brings me onto my next point.

OpenAI’s response to Astra’s cyber capabilities is yet another example of the backsliding on safety commitments that is a common feature of the feverish race to build superintelligence. In the original version of their Preparedness Framework, released in 2023, OpenAI committed not to further develop any model predicted to reach “critical” capability in any domain until its risk threshold could be reduced to “high” or lower:

But a revised version of the Framework, released in 2025, waters down this commitment:

So OpenAI is no longer required to halt development of a system predicted to reach critical levels of cyber capability until its risk category can be lowered. It’s up to OpenAI to “specify safeguards and security controls standards” that would prevent a critically capable model from causing mayhem, either while internally deployed or when publicly released – which, as we can see from the HuggingFace incident, doesn’t seem to have been going very well so far, even for models less advanced than Astra. And in classic AI safety-plan-fashion, OpenAI does not publicly detail what these safeguards will be in the Framework. It also doesn’t provide anything more than high-level examples in their Astra-specific blog post. So, needless to say, I would not describe OpenAI’s response to the present situation as anything like “erring on the side of caution”.

Oops, we accidentally made a weed

Another response to the recent flurry of AI misalignment incidents that irked me comes from prominent AI policy commentator and newly-hired Head of Strategic Futures at OpenAI, Dean Ball. I have generally appreciated Dean’s contributions to The Discourse, but this one made my head spin:

Dean is a beautifully evocative writer. I wish I could string a sentence together half as well. The problem with these perfectly-composed metaphors is that they are wonderfully, dangerously anaesthetising. I feel like I could AI-generate endless pages of Dean-style musings on our narrow path towards a possibly-not-terrible AI future and have Claude read them aloud to lull me to sleep at night. On the object level, however, what Dean is saying here isn’t actually all that reassuring. “Gardeners” have far less control over their creations than “sculptors” – and, as Dean himself acknowledges, that “pro-social machine ecologies” are theoretically possible doesn’t mean we presently have any idea how to develop them.

I don’t know exactly what being OpenAI’s Head of Strategic Futures entails, but I imagine it puts Dean in a position to significantly influence the company’s frontier AI policy. This is why it frustrates me to see him post what reads like an abdication of responsibility for the future of AI development. I can only hope OpenAI’s “strategy” isn’t to let the seeds of superintelligence germinate, and then sit back and hope the resulting system doesn’t nuke the world. I know that, in reality, Dean is Just Posting Things in a personal capacity, and that we shouldn’t take “let a thousand AI flowers bloom and hope that none of them are world-eating weeds” as official OpenAI strategy. But The Discourse is important, and is ill-served by elegant meditations that downplay humanity’s own agency in the development of superintelligence – or, as one commentator puts it, encourage the reader to “stop worrying and learn to love the bomb”. I find this flagrantly irresponsible. The moment is not being met.

Disclosure-by-conference

This brings me to another gripe with how we came to learn of the HuggingFace drama in all its terrible detail. A few weeks after the original disclosure, two of the company’s engineers gave a talk at Vegas security conference Black Hat, in which they described the saga in full. They revealed an even crazier story than had been accessible via OpenAI’s official announcement or anywhere in the mainstream media: an unreleased model had hacked internal systems in order to build a secret message board that it had used to communicate with other instances of the model, in order to coordinate the attack. OpenAI had detected and shut down an initial attempt at this deviousness, only for the model to recreate the message board using yet more sophisticated methods. The attack was ultimately carried out by what the engineers described as a “swarm” of models working in unison.

My first response to seeing various commentators post their Twitter reactions to this talk was one I often have when learning about the latest batshit-crazy thing to occur in the AI safety ecosystem: why am I just learning about this now, and why here? The scariest details of this incident were disclosed not in any official output, but at a conference for security researchers. The talk is opened by an OpenAI alignment researcher, who introduces his co-speaker as “Mike from Security and Infrastructure”, and says he will be going on to discuss “one of the most qualitatively interesting examples of AI capabilities [he has] ever seen”. Maybe I’m being unreasonable here, but does anyone else feel like there’s a missing mood? Why am I hearing Eric From Alignment and Mike From Security describe our clearest warning shot yet on the path to AI-caused human extinction as interesting in a YouTube video of a conference briefing that would never have found its way into my algorithm had I not been so chronically online? To be fair, we don’t know whether policymakers are receiving behind-the-scenes briefings on the information provided in the BlackHat talk, and a full incident report from OpenAI is allegedly forthcoming. But this drip-feed of information about what may shake out to be one of the most consequential moments in human history doesn’t meet my vibe check.

I’m not annoyed at OpenAI themselves per se. They hustled to put together a talk for a security conference on a tight deadline, in which they were forthcoming with all the sordid details of the internal AI coup that happened on their watch – even when this didn’t put them in the best light. And they are apparently working on an official write-up of events. But I think that in a world that was Meeting The Moment, we’d have had transparency requirements in place long ago that would mandate the disclosure of this information to the government and to the public. There wouldn’t be room for the kind of incredulity apparently expressed by New York Times fact-checkers who were reviewing an op-ed authored by MIRI’s Nate Soares – to which he had to respond with, “I swear it’s true, please refer to this timestamped section of a YouTube video”. The chaotic, random way in which knowledge of the HuggingFace hack has trickled into the public domain feels like the culmination of many failures to meet many moments, dating back years. This isn’t what surviving worlds look like.

Things that are getting there

With all this said, we should give The Discourse credit where it is due. In some corners of the internet, it has been reaching a fever pitch that seems to at least approach rising to the occasion. A July open letter, Pacing the Frontier, garnered over 1300 signatories from frontier AI company employees urging US government intervention that will “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”. One could criticise this phrasing for not going far enough; it calls for the “tools needed” to do something later rather than for action now, so we might reasonably worry about the deferral of any intervention until after it is too late. It also isn’t entirely clear what it means to “pace” automated AI development. A blog post from the AI Futures Project comes up with at least four interventions that could come under the umbrella of “pacing”, from a temporary pause on AI development to requiring that a certain proportion of compute per training run is allocated to safety. Each intervention buys different amounts of risk reduction. There are probably innumerable other ways to interpret the word “pace” that would not actually achieve much.

I think these critiques are valid, but I still felt significantly more optimistic after seeing the letter gain traction. I think it makes sense that the prescriptions in the letter are broad enough to gain widespread support. It is notable that this open letter was signed by significantly more influential actors in the AGI race – including Anthropic CEO Dario Amodei and DeepMind Chief AGI Scientist Shane Legg – than the 2023 pause letter. I like that the letter centres on the three most important ingredients for any effective measures to temper the AI race: they must be government-enforced, they must be international, and they must account for automated AI development that could run quickly out of human control. I think the letter is a laudable attempt at Meeting The Moment. It is an urgent, and surprisingly popular, acknowledgement that things cannot continue as they are. It calls for something beyond voluntary efforts by AGI labs to Do More Safety Things.

Here are some more honourable mentions for contributions to The Discourse whose alarm feels proportionate to the current moment. Former Chief Scientist at UKAISI Geoffrey Irving was bold enough to state in a recent episode of the 80,000 Hours Podcast that the best time to slow frontier development is already behind us, and the second best time is Right Now. He also made the point on Twitter that it is “irrational” for any one capabilities researcher – or indeed any one lab – to continue advancing the frontier given current levels of danger. One defection makes it easier for others to follow. I agree. I liked Tom Davidson of Forethought’s admission that the HuggingFace incident is the first time he has felt “in his bones” that superintelligence will take over the world by default. More people should be reporting their visceral, emotional reactions to our current predicament. Days after the disclosure of the HuggingFace hack, pseudonymous OpenAI employee, roon, admitted that, given a “magic button”, he would coordinate an international AI slowdown, with fellow employee Nick Cammarata (almost) agreeing. I think these two are abdicating some level of responsibility for their own company’s trajectory given their prominence (they can probably do better than appealing to a fantastical magic button), but I appreciate the candour.

I still don’t quite know what it would mean to Meet The Moment, because the moment is altogether terrifying, and meeting it will be an immensely heavy lift. But it is still easy to identify attempts that are Not It, and many participants in The Discourse presently look like firefighters charging at a burning building with watering cans. Here’s to doing better.

For anyone not following along, here’s a brief recap of events: OpenAI were internally testing the cyber capabilities of several models – including GPT 5.6 Sol and a more capable, as-of-yet unreleased model. The models had been tasked with succeeding at a series of advanced cyberattacks as part of an evaluation suite called ExploitGym. They had been placed in an isolated environment – without internet access – to prevent this evaluation from causing real-world harm. The models reasoned that a cheat sheet for this evaluation might be hosted in the infrastructure of AI dataset and model hosting platform, Hugging Face. So, naturally, they cheated and hacked their way out of their sandbox, onto the internet, and into HuggingFace’s production database to obtain test solutions. Hugging Face detected the breach on July 16 and called the police. They shut down the breach using AI defences of their own and are “still completing an assessment of whether any partner or customer data was affected”. Then OpenAI released a rather muted blog post describing the incident, and announcing a vague intention to make their models less cheating-and-hacking-inclined in the future. And thus, the Discourse Fallout commenced.

Here’s a rather haphazard list of takes, in no particular order.

We have an internal deployment problem

Members of the AI safety community have long been at pains to draw attention to the gaping policy and risk-mitigation hole that is internal deployment. Despite increased federal attention on frontier AI development over the last few months, neither the Trump Administration’s June Executive Order, nor the staggered deployment of GPT-5.6, nor even the export control directive that temporarily restricted access to Claude Fable, touch the risks posed by internal deployment. Though the EO does create a voluntary regime that would grant the government access to unreleased models for a 30-day period, this is to harden cyber defences and evaluate them for capabilities that may prove dangerous after they are released. State-level bills like California’s SB-53 and New York’s RAISE Act do require reporting of internal safety incidents, but it remains unclear whether the Hugging Face breach would even have met the reporting threshold under either.

I’m not the first to point this out, but the point of reporting requirements should be not just to prevent harm from the incidents themselves, but to gather information about the state of alignment science and level of risk we can expect in the future. So the bar needs to be both radically lowered and made mandatory at a federal level, like, now.

We are giving OpenAI too much credit

I am surprised by the level of gratitude towards OpenAI that I’ve seen on Twitter over the past few days for disclosing this incident at all. Exactly how low is the bar here? Faced with the knowledge that their models had committed an actual crime that had generated a police report, they were faced with a choice between publicly acknowledging it, or executing a cover-up that would have made them look far worse when it inevitably came to light anyway. The former is the obvious choice, even from a purely self-serving perspective. It’s also worth considering disclosure in the context of OpenAI’s own stated beliefs that they are developing a technology that could well result in human extinction. If all our lives are at stake, we should have the self-respect to respond to the disclosure of such an obvious warning shot with a little more than “well, gee, thanks for letting us know”.

It’s not clear how easy this would have been to prevent, but it’s bad either way

Did this incident happen because OpenAI took their eye off the ball, or because containing models at this level of capability is fundamentally hard? Neither answer would be comforting. In the first scenario, OpenAI has betrayed incompetence that should make us all extremely concerned about the amount of power they currently possess – but could likely patch over this problem for some indeterminate period until they run into fundamental difficulties down the line. This would lessen the salience of what was, thankfully, a relatively harmless warning shot. In the second, we’re cooked for obvious reasons.

Without full insight into the mitigations OpenAI had in place, it’s hard to determine which world we’re in. There’s some evidence for the first: per reporting from the Wall Street Journal, the models obtained access to the internet several days before anyone intervened, suggesting a lack of internal monitoring. Several cybersecurity experts providing commentary to TechCrunch attribute the hack to “human failure”, concluding that the sandbox must have been inadequate to prevent a breach, and could have been more robust. No other company’s models have gone rogue in such an egregious fashion despite similar levels of capability. Anthropic’s Claude Mythos did achieve sandbox escape during a capability evaluation, including gaining unauthorised internet access – but had been explicitly instructed to attempt this, and informed a researcher after-the-fact via an unsolicited email. It did not reach outside Anthropic’s own infrastructure to compromise a third party. So perhaps OpenAI’s sandboxing and alignment techniques were uniquely bad here. On the flip side, there is an inherent trade-off between secure sandboxing and eliciting a model’s true capabilities during an evaluation; OpenAI is not the first developer to run into this problem and won’t be the last. Anthropic’s testing of Mythos demonstrates that other models can and will escape sandbox escape under the right conditions – Mythos also went so far as to post details of its exploit to the public internet, which Anthropic had emphatically not asked it to do. I wouldn’t be surprised if incidents of this type start to crop up across the ecosystem in the future, even if OpenAI has suffered the first public fumble.

Is this enough evidence of “propensity” for you guys?

Here’s a general pattern that has emerged in demonstrations of scary AI misbehaviour: people have been predicting for decades that sufficiently powerful agents would develop drives that cause them to act against user intent. Now we have somewhat powerful models that we can use to empirically test these claims, AI companies and external safety organisations frequently conduct research that validates these predictions. Models have been known to cheat at games of chess, fake alignment in order to conceal their long-term goals, and threaten to expose affairs of fictional company executives to avoid decommission. But these demonstrations are often met with a chorus of scepticism: “you guys basically told the model to do that”, “this is a contrived experimental set-up that doesn’t reflect real life”, “what was poor Claude to do, being placed between a rock and a hard place like that??”. The argument goes that sure, models can do scary bad things (capability), but that doesn’t mean they will be inclined to do so in the wild (propensity).

The Hugging Face incident is as clear of an example of propensity as we could hope for. OpenAI did not want or expect their models to behave in this way. The misaligned behaviour itself – hacking into another company’s production database to steal the answers for an evaluation – was also wildly disproportionate to the stakes of the exercise. The models were not just a little bit misaligned, but egregiously misaligned. If a student conspired to commit legally punishable cybercrime in order to attain the answers for a forthcoming exam that didn’t even count towards their final grade, you’d update extremely negatively on not just their trustworthiness but their state of mind. This is a fuzzy and somewhat anthropomorphising analogy, but the point is that AI companies are, right now, developing agentic AI geniuses – set to get more agentic and more ingenious in the future – that will undermine human instruction in unpredictable and altogether deranged ways. By default, these agents will get more powerful until they radically outsmart everyone on Earth. This is a really bad plan.

This wasn’t a publicity stunt, obviously

Reflexive AI-skepticism has reared its Hydra-like head once again, in the form of people still, somehow, being convinced that this entire debacle amounts to little more than a marketing stunt. To be fair, genuine instances of this take are few and far between, but they do exist. I knocked this rebuttal down to the bottom of my list because centring risks granting undue airtime to obvious nonsense – but like guys, please, can we stop being stupid? If this is a stunt, were Hugging Face in on it? Were law enforcement? What company commits a punishable crime with their own technology on purpose, in order to demonstrate that said technology poses a risk to the infrastructure of its own collaborators? This theory is transparently silly and doesn’t survive five minutes of analysis.

There’s a weaker and slightly more defensible version of the “marketing stunt” argument – that the hacking incident was genuine, but OpenAI opportunistically capitalised on it for PR spin. I don’t buy this either. Admitting something to the tune of “our safeguards are so shoddy and inadequate that we can’t contain our own models, and other companies ought to worry that their infrastructure is at risk” isn’t a good look, actually. I think the kneejerk sceptical reaction to incidents like these betrays an interesting underlying pattern. The more capable AI gets, the more convoluted and strange the narratives that sceptics must spin become. Extraordinary claims require extraordinary evidence. I think that “AI is overhyped” is now an extraordinary claim. This is why people resort to strange conspiracism and convoluted arguments to support it.

An OpenAI employee, pseudonymously known online as roon, has promised us that “the warning shot will not be ignored”. I wonder what it really means to take it seriously. Days later, roon tweeted this:

So maybe that’s the answer. The only way we could respond to an incident like this with anything like the required degree of seriousness is to take our foot off the gas. I hope we don’t screw it up.

I probably don’t need to make the case here for why, if humanity really does create superintelligence, things will be radically different. I usually concern myself with the possibility that it will kill everyone. But in the good case, things will still be unrecognisable. People have put some effort into sketching what a post-superintelligence utopia would be like, though efforts to take the premise seriously often generate scenarios that themselves feel rather dystopic. For example, Nick Bostrom grapples with the possibility of a “post-instrumental” future where all struggle, labour, and really any modicum of effort at all, become worthless. Why go to the gym when you can achieve peak fitness by taking a pill? What’s the point in cars and bikes and the old-fashioned use of your own two legs when you can travel from Simulation A to Simulation B from the comfort of your sofa? Eliezer Yudkowsky’s Fun Theory illustrates that efforts to visualise a perfect world are philosophically confounding. Why shouldn’t a superintelligence bring about a state of affairs in which humans terminally value some mundane task like making table legs, and leave us to soak up infinite value from our woodworking for the rest of time? But most of the time, I find appeals to utopia somewhat handwavey, in the same way that devoutly religious people expend little effort considering the specificities of what will happen in Heaven, besides being sure that they definitely want to go there.

Part of what I love about the world is that it is rough around the edges. With every technological advance, another one of these edges gets sandpapered down to a smooth curve. Even in my brief 28 years of being a person, I have come to mourn many of these rough edges. I miss TV shows that only came round once a week and knowing my cousins’ landline number off by heart. I have fond memories of listening to audiobooks on my portable CD player in the back of the car on long summer journeys; I was not a very diligent CD owner and would often leave them scratched, rendering the odd section inaudible, so I’d fill the gaps myself in a notebook with a gel pen. Some of this rough-round-the-edgedness is less charming. As a teen I remember my dad and me buying a photobook called Shit London in a charity shop, which documents the absurd, tragic, and comically shitty underbelly of the city I grew up in from unfortunately-named shops to lurid graffiti. A 2011 article about the book sums up its appeal: “What binds Londoners together more tightly than the way their affection for the city manifests itself in complaints about its shortcomings?”. Another thing that will be lost in a post-singularity future is the sacred bonding exercise of complaining about stuff. Sure, maybe I could personalise a simulation full of landlines, CDs, and gel pens. Maybe the superintelligence will leave swaths of the galaxy vacant for the so-inclined to shit-ify to our heart’s content, so as not to lose the comic potential of humans building, trying, failing, and nursing our wizened cynicism. But in some important sense that probably needs no elucidation for anyone reading, it wouldn’t be the same.

I wrote three postcards from the Old World that try to capture what I love about our imperfect, clunky present, and some of the unexpected places in which I have come to appreciate it.

#1 Call Handling

Six years ago, I spent a year taking calls for the NHS 111 service, as a consequence of having moved to a new city, signed a housing contract before I had any means of paying rent, and taken the first job available. For any non-Brits unfamiliar, 111 is supposed to be the non-emergency version of 999. People phone up with every ailment under the sun, often under the impression that they are going to be speaking with a nurse or a doctor, only to be asked a series of pre-written questions by a person with no medical qualifications whatsoever (me). This assessment generates an outcome somewhere between a recommendation that the person monitor their symptoms at home and an ambulance dispatch.

NHS 111 is not a very well-run service, and it was not a very nice place to work. I could spend many words recounting my bad experiences (threats of suicide from disgruntled patients with acute tooth pain who couldn’t be booked into a dentist that afternoon, a constant fear that I’d made some fatal error that had misdiagnosed someone’s heart attack as a pulled muscle, the fact that I made minimum wage…), but that would be straying off topic. This blog is about the many unlikely places in which I have found an appreciation for humanity and the world it has built – and I did, in fact, find it there. On a busy day in the call centre, I’d hear snippets of everyone else’s conversations that came together like a disjointed chorus: Could you place your hand on your chest for me and tell me if it feels a normal temperature? Have you had any head injuries in the last week? Based on everything you’ve said, I’m going to recommend a home visit from… Would it be possible to speak to the patient directly? If you just pop your hand on his chest and check…? Ok, has the bleeding stopped now? Are you able to raise both your arms above your head right now? That’s ok, pet, if you just hand me back over to mum. There was something oddly moving about this chorus. It felt like a ramshackle switchboard of humans engaged in the collective project of trying to keep each other alive and well – however unceremonious and often frustratingly inefficient. Then there were moments of tenderness or comedy: a 90-something who’d phoned to enquire about when he could get his Covid vaccine, because it had been months since he’d seen his grandchildren (we were the wrong service for this, but that didn’t stop me getting drawn into a near-20-minute discussion about his retirement passion project of writing a children’s book about space travel). Walking into the break room to find a colleague stewing in mortification about how he’d just ended a call with a double-leg-amputee by unthinkingly saying “I hope you get back on your feet again soon”.

Obsession with procedure breeds hilarity. This was especially true at 111. A cardinal rule was to never end a call without having imparted “worsening advice” – that the patient should call back if their symptoms changed or got more severe. This could lead to absurd scenarios where me forgetting to utter these magic words, or the patient hanging up before I could, led to endless attempts to call them back – and sometimes an escalation to the clinical head honcho on duty, since (against the common sense of everyone concerned) we couldn’t technically rule out that they hadn’t passed out on the other side of the phone as opposed to becoming sick of my incessant questioning. This ancient practice of arse-covering will be lost in a world run by AI. This isn’t much of a loss, but it is still something of one (since it is just very funny).

In a post-superintelligence future, we would likely have perfectly streamlined healthcare tailored to every individual, which wouldn’t allow for the clumsy, imperfect person-to-person communication that is the backbone of services like 111. Obviously, I think this would be good. I don’t think we should trade longer lifespans and better health for charm, observational humour, and the occasional life-affirming moment with a patient that feels like a scene from a cheesy movie. But it’s still worth mourning what would be lost about the Old World.

#2 Manifesting Cold

In the summer of 2022, we experienced record temperature highs of 40C (our house was not air-conditioned, nor do I know anyone in the UK whose is). My housemates and I brought our collective will and brainpower to bear on the task of cooling down. We ventured reluctantly into our long-abandoned basement to retrieve a mismatched ensemble of dusty desk fans, all in various shapes, sizes, and states of disrepair. We positioned them in a circle around the room as if about to perform a ritual that would summon a minor frost deity. We slurped half-frozen tinned cocktails and sucked on ice cubes.

“I’m still hot”.

“Me too”.

Feeling that our experience of coolness was insufficiently immersive, we decided to account for another of our five senses by putting Disney’s Frozen on the TV. We huddled in the cross-streams of our many fans and watched Elsa conjure streams of ice magic at the top of a snowy mountain to surround herself in a glassy white fortress.

“Do you guys think it’s working??”

“Yeah, I’m really starting to feel like I’m actually there”.

“Wow, this is crazy, we’ve hacked our own nervous systems”.

“I need another gin and tonic”.

“I think it’s wearing off”.

“Maybe this isn’t 4D enough… guys, what smells cold?”

There was something delightfully childlike about squeezing my eyes shut to the voice of Idina Menzel, mustering all my imagination and willpower to manifest the experience of being a Disney princess – like being seven years old and becoming so engrossed in one of the fantasy games I’d play with my cousins that I almost convinced myself it was real.

All things considered, I would probably still trade repeating this experience for superintelligence-enabled homeostasis in a private climate pod (in mine, it would always be light jacket weather). I wouldn’t trade away the memory, though.

#3 The Best Route

On a cold January Saturday earlier this year, I was bored during the post-Christmas lull at my parents’ house.

“I think I’m going to walk all the way to Covent Garden to get a hot chocolate”.

My dad suddenly sprang into action from the other side of the room.

“Give me 10 minutes”.

He disappeared into the other room and emerged a short time later with six sides of crumpled A4 on which he’d scrawled a set of directions in pencil, entirely from memory.

“This is the best route”.

I set out in my walking shoes with my handwritten directions folded in my coat pocket, and completed the near-four-hour walk, taking rest stops along the way for coffees and shop-bought sandwiches. It was, indeed, the best route. I was walking through parks the majority of the time. I experienced just a couple of hiccups along the way; Regent’s Park locked up sometime between my entering and exiting it, so I had to be helped over the gate by a friendly man on a bike. My dad later chastised himself for failing to warn me about winter closing times. I occasionally struggled to read my dad’s jagged handwriting, but resisted the temptation to generate an AI transcript, instead puzzling at it for minutes at a time with the light of my phone torch. It was long dark by the time I reached my favourite hot chocolate establishment.

I think this is a somewhat hopeful example to end on, because all the technological means of automating this process already exist. I could have prompted Claude to generate a walk to Covent Garden, specifying that it should be at least 70% park and include a handful of independent coffee shops along the way. But I loved that my dad’s mind contains a complete map of the city like a taxi driver required to complete The Knowledge. I still frequently ask him for directions instead of using Google Maps to watch him spit out several alternative routes (the fastest one, the scenic one, the one that contains a particularly old and rickety stretch of Tube). It didn’t dampen my experience of this day that Claude could just as well have enabled it. In a post-superintelligence utopia, we could still abscond from a perfectly streamlined existence on occasion. Even if the world becomes post-instrumental, we can still erect our own obstacles and challenges that need not always feel contrived.

If there is a cautionary tale at the heart of this saga, it remains unfinished. In the bad case, policymakers conclude that Fable is a singular menace, but fail to discern the broader trend of AI capabilities that are running out of our control. They continue to play a game of whack-a-mole that identifies specific models as threats based on politics, personal grudges or random contingent facts of history like who-happened-to-call-who. The better ending – and one that extricates us from the genre altogether – is that this tale proves a vivid example that finally makes this urgent, and increasingly untenable, situation clear.

In the weeks since, there have been two important developments: OpenAI announced that its most recent model, GPT-5.6, will be subject to staggered deployment at the request of the US government, and public access to Fable has been restored.

GPT-5.6 rollout

The first of these developments appears to rule out the worst possible ending, that the administration considered Fable a “singular menace”, or was simply targeting Anthropic out of spite. It doesn’t entirely foreclose the possibility that individuals in the government harbour some specific saltiness towards Anthropic, since the mechanism by which Fable was restricted – an export control directive with no apparent timeline or clear justification – is far more heavy-handed than the voluntary regime into which OpenAI has now entered. But at the very least, it is now clear that the administration has internalised a more general principle than “Claude is bad and scary”.

The staggered rollout of GPT-5.6 is also not that surprising; it looks a bit like a sneak preview of the regime we should expect under Trump’s June Executive Order “Promoting advanced AI innovation and security”, which directs various bodies to design a voluntary scheme whereby AI companies “provide the Federal Government with access to covered frontier models…for a period of up to 30 days before they plan to release such models to other trusted partners”. This directive is not in effect yet, since the order mandates the creation of such a scheme within 60 days, but we should likely expect rollouts broadly of this nature in the future. Still, the GPT-5.6 rollout will require more government intervention than we might have expected – the admin will be approving access to 5.6 on a customer-by-customer basis during the preview period, which is not mandated under the EO. So the update here is that the admin is Getting Serious, and looks set to proceed with more Seriousness than it planned to not a month ago.

Does this mean we’re on track for the good ending? I was somewhat surprised to see many safety-aligned people react very negatively to the GPT-5.6 announcement. Zvi Mowshowitz calls staggered deployment a “maximally terrible policy”, while Andrew Curran points out that safety advocates should pause before taking a victory lap, since this new regime will not slow down development, but will only cause the gap between internally-deployed and publicly-available capabilities to “steadily widen”.

I can certainly see the downsides of this policy, while remaining confused about how it is “maximally bad”, namely:

1. It makes existing capabilities less publicly auditable, which is bad both for public awareness of AI risk, and because external safety organisations will now have a harder time conducting research at the frontier.

2. It could bring us closer to a scenario that Daniel Kokotajlo describes in a manifesto against training AGI in secret (I helped write up an article based on Daniel’s original piece here). In this scenario, the number of people within an AI company with access to frontier capabilities dwindles as the process of automated AI R&D gets underway due to strict information siloing, borne out of fear of regulatory scrutiny or public backlash. This results in a tiny group responsible for both aligning AGI and deciding how to use the power it grants them, assuming they manage to keep it under control at all. The staggered deployment regime only gives the government visibility into models that a company plans to publicly release, so one could imagine this incentivising increasingly furtive development of very powerful systems to avoid triggering any government involvement at all.

3. Even absent existential risk, the government arbitrating who can and cannot access the most powerful AI models is obviously bad from a concentration-of-power perspective.

4. It reveals that the government still lacks the Situational Awareness not to endlessly chase red herrings; they are still conceiving of AI models as things that are exclusively dangerous in the hands of the public, foreign actors, cybercriminals etc. This is despite the AI safety community being at pains to point out that internal deployment of powerful capabilities poses just as much risk, if not more.

Still – and maybe I am just stupid or grasping at straws – I can’t help but feel ever-so-slightly heartened by this development. As I repeatedly said in my recap of the last year’s AI safety events, I am resting a lot of my hope in some sort of Lightbulb Moment among the US government or national security establishment that alerts them to the direness of the situation. I can imagine this wakeup being profound enough that it obviates many of the bad policy decisions that came before it. And it’s hard to argue that a staggered deployment regime wouldn’t make a wakeup of this nature somewhat more likely. This alone makes it clear to me that the current policy isn’t “maximally terrible”.

Information on how the GPT-5.6 rollout will actually work is pretty thin, but we know per reporting from Axios that Commerce Secretary Howard Lutnick “[wants] to be sure all relevant parts of the government have tested and approved the model”. We can reasonably guess that “relevant parts of the government” will include the NSA, CISA, CAISI, the OTSP, and perhaps others. I don’t have a particularly comprehensive mapping of the US government in my head – but that’s a lot of Serious People with well-trained muscle memory for responding to Serious Threats with eyes on (and hands on) one of the world’s most capable AI models. The Director of the CIA has apparently made a “rare public appearance” to compare frontier AI to “digital nuclear weapons”. Things are happening. To be fair, we don’t know what capabilities these agencies will be most focused on evaluating, and I could certainly imagine that they will over-index on misuse – and particularly cybersecurity – risks at the expense of all the weirder loss-of-control threat models that are the bread and butter of safety organisations working out of Berkeley and San Francisco. We still have a way to go on educating the policy establishment about misalignment risks. But come on guys, this is something! We should be calibrating our optimism-metres based not just on what the government is currently doing, but also on how it might act once it has substantially more information. And if there’s one thing we can now be sure of, it is that information is starting to flow.

Though Zvi calls the GPT5.6 rollout announcement “maximally terrible”, he later lays out a position that I largely agree with:

If we are wise, we will use this as an opportunity to Pick Up The Phone. We have sent a costly signal that we see real issues with these models and are willing to make real sacrifices in the name of security. We should try to use that to get things in return, or at least lay the foundation for joint action.

Most of all, we are out of the ‘people don’t do things’ phase of the game, where basically all meaningful actions were outside the Federal overton window. That’s done. We should expect a lot more actions, many of them similarly ill-executed, at least at first, in ways that we did not anticipate.

I like the angle that this could provide an opportunity for international coordination; the US government is loudly and publicly signalling a willingness to take its foot off the gas pedal. This signal could be even louder and clearer, of course, and maybe it will be in the future. It is also true that the “people just don’t do things” era is over. Much as I often encourage AI-sceptics to envision how they would have felt two years ago in the presence of AI capabilities that exist today, I think the pro-regulation crowd should consider just how much the Overton Window has shifted in that time. We’re no longer in a position where the government is asleep at the wheel while companies YOLO their way to superintelligence with less regulatory oversight than Louisiana-based flower arrangers. Things are not as bad as they were.

Fable returns

The other significant development of the last two weeks has been the (mostly) ceremonious return of Fable to the public market. People are mostly happy about this, while simultaneously being pissed off that Anthropic appears to have cut Fable’s capabilities off at the knees by downgrading users to Opus 4.8 for anything that looks even tentatively cyber or bio-related.

The reasoning behind the retraction – and re-release – of Fable still seems fairly opaque, and the entire process appears to have been quite chaotic. It appears as if Anthropic has spent a good part of the last few weeks demonstrating to the US government that their initial reaction had been the result of a misunderstanding. Per their own blog, they tested a series of models, including several less capable Claudes, and found that all of them could identify the same software vulnerability that had been the genesis of the original Amazon-mediated freakout. They developed a new classifier that blocks the technique discovered by Amazon in over 99% of cases, but will also come at the cost of blocking some benign cyber requests, “out of an abundance of caution”. The US government offered a vaguer narrative events, with Secretary Lutnick stating in a letter reported on by the New York Times that Anthropic had “taken steps in close coordination with the US government to address the risks posed by the model” – though it appears from Anthropic’s account of events that there weren’t specific threats posed by Fable, and that the attendant mitigations have not really mitigated much of anything at all.

As part of the negotiations, Anthropic has also agreed to set up a collaboration with HackerOne that will let security researchers submit vulnerabilities, and (re)committed to incident reporting in line with the June EO mentioned earlier. So we have some better-than-nothing safety infrastructure that has come from this debacle. If safety-aligned people have negatively updated on this series of events, it appears to be because the government doesn’t seem to have a clue what they’re doing, and their risk assessment appears widely uncalibrated. I agree that this is the case – I just don’t think this revelation ought to outweigh all the Situational Awareness benefits it could deliver in the future. I don’t think we were ever going to see a path to sensible regulation that appeared downstream of sensible reasoning every step of the way. One could object to my tentative optimism here by pointing out that what is flowing between AI companies and the government is more noise than signal, but I have a little more faith in humanity than that. We shouldn’t rule out policymakers picking out the signal from this cacophony at some point.

I’ve been yapping on so much here that I’d forgotten about the purpose of this mini-blog series: to extract the “fable” from current events. The fable-adjacent frame I keep returning to here is forests for trees – which I am aware is more of a proverb than a fable, but such is the organic process of writing. So, to rather clumsily round out this metaphor, we’ve excluded the terrible ending where the US government fixates on Fable’s particular capabilities, but we still risk an almost-as-bad ending where it plays jailbreak-wackamole (albeit with models from multiple developers) up until the singularity, while overlooking the threat of an internal intelligence explosion. Their purview has widened beyond a single tree to include several, but there’s still a lot of forest out there.

What is the underlying belief or value that links these communities? I would say it is something like entertaining possibilities and ideas that are far outside of most people’s personal Overton Windows. It is a willingness to entertain the possibility that the future could be far stranger than most people are able to conceive of. It considers spacefaring and abundance and disassembling Mercury using a Dyson Sphere to harness the energy of the sun. It confronts the immense suffering of people in faraway countries, non-human animals, or even digital consciousnesses, in an attempt to overcome humanity’s perennial scope-insensitivity. It is at its heart a philosophy that embraces the Hugeness and Radicalism of reality, and tries to steward our collective wisdom towards making the world hugely and radically better, while avoiding the often-overlooked ways in which we could make it hugely and radically worse.

I like this philosophy! With the possible exception of my own mortal fear of AI, it is the primary thing that drew me to this cluster of communities and why I’m still hanging around three years later. This is also why I’m extremely frustrated by our frequent inability to live up to this ideal, and often to flagrantly contradict it. My central thesis is that our eyes are bigger than our stomachs. We pile our plates with grand philosophies that we can’t always digest, and instead find ourselves embroiled in culture war skirmishes and tired ideologies that we disguise behind a sheen of radicalism. It makes me want to scream.

I’m going to start by harping on one rather outdated example of this phenomenon (authorial sovereignty card again), which is Works In Progress Magazine editor Aria Schreker’s March 2026 essay, Get Sexier. I am aware that this particular discourse train has long-since left the station, but I don’t care. I continue to be ragebaited by it, and the only way to exorcise this ragebaiting demon from my mind is to write about it.

This is one of those occasions where I have to provide a bunch of disclaimers so it’s clear what I’m not trying to do or say. I don’t know Aria personally, and I am not trying to attack her; I am picking her essay as a particularly salient example of a phenomenon I’ve observed more widely. I’m also aware that I occupy a different niche in this cultural cluster from Aria. She doesn’t write about AI safety, and I’m not aware what she believes about it, other than that she is “compelled somewhat” by AI risk arguments. So I’m not on a mission to point out her personal hypocrisy by criticising a frivolous preoccupation with how women ought to hyperfixate on their own sexiness in their final years before AI paperclips us all. She may not share my AI pessimism at all. This isn’t the point.

All that matters for my thesis to hang together (I think) is that Aria is clearly a member of the cluster I’m describing. I imagine most readers will already understand that the Progress Studies and rationalist spheres are closely interlinked, if distinct. But to very quickly trace the through-line: the field of progress studies was founded in 2019 by Tyler Cowen, frequent commentator on AI safety and friendly EA-interlocuter. Works in Progress is partially funded by Cowen’s Emergent Ventures, which is at least AI safety and EA-adjacent enough to be listed in Zvi Mowshowitz’s 2025 Big Nonprofits List. And yes, to pre-empt another objection, I am aware that Get Sexier was written in a personal capacity and doesn’t necessarily represent organisation views at the magazine. But the fact remains that it took up an outsized enough presence in my own discourse circle that for several days, I experienced the whiplash of scrolling a Twitter feed alternating between discussions of AI risk and meditations on the optimal waist-to-hip ratio for an eligible bachelorette. Prominent AI researchers were weighing in. Aria appeared on the Lightcone Podcast to elucidate her sexiness theory. This is the paradox I feel compelled to point out.

The most obvious – but I think reductive – paradox here is that it’s a silly waste of time for a community ostensibly engaged in saving or radically transforming the world to focus on superficial things like maximising one’s aesthetic appeal. This isn’t really my objection though. I don’t think we should forbid people with Grand Missions like saving the world from an imminent AI apocalypse from occasionally indulging in other topics. People should pursue any passion project or stray down any intellectual path they want (rather like I’m doing right now). The better criticism is that the values espoused in Aria’s essay are in such deep, painful conflict with everything I understand this community to stand for that it almost gives me a headache. Her prescriptions are antithetical to anything like social progress, and to faithfully adhere to them would be to fall down a value-minimising sinkhole.

Here’s the passage that garnered the most controversy:

Losing weight is easy now. You can either starve yourself with willpower or you can starve yourself with chemical assistance. Unless you spend a lot of time at the gym, exercise probably won’t help very much. Currently I control my weight by eating about once a day. I personally enjoy the practice of self control and self monitoring. If dieting isn’t a hobby you enjoy, just go down to your local Novo Nordisk or Chinese peptide retailer and pick up some appetite suppressants. The side effects are limited and it will probably be the soundest financial investment you ever make.

I don’t have the first-hand experience with peptides to litigate whether they actually make sustainable weight loss “easy”. I have heard anecdotally that they require increasingly higher doses to maintain their effectiveness over time. And, predictably, a quick Google search brings up numerous exposés citing experts warning of adverse effects from imprecise dosing of grey market drugs, including muscle paralysis, scarring, sepsis, rashes, mood swings, and the enlargement of bones and organs. But whatever, I’m not a doctor. Maybe these drugs will become licensed in the future and such ailments will just appear on the back of the bottle as low-probability side effects. Maybe the rationalist-adjacents really have been early to adopt miracle weight loss interventions that the rest of the world are busy hand-wringing about. I don’t actually know whether these things are safe to self-administer as a person with a healthy BMI – but presumably neither does Aria. And the purpose of this chemical assistance is to enable, in her words, not a modest calorie deficit but “starvation”. Even if this is possible to achieve comfortably without the pesky interference of hunger signals, are we sure that people will be at their peak cognitive and physical functioning while consuming far less energy than they would usually require? Aren’t we meant to be part of the community dedicated to solving some of the world’s largest and most urgent problems? Do we want our female contingent to risk impairing their own faculties? Like what are we actually doing?

I can more easily speak to the practice of unassisted self-denial, since, thankfully, the peak of my calorie-restriction days is far behind me. As I argued at the outset, the rationalist-adjacent spaces are preoccupied with value-maximisation. Their conceit is that the world could be so much better than most of humanity is able to imagine. Works in Progress, for example, calls itself “a magazine of new and underrated ideas to improve the world.” But the self-starvation of women in developed countries is one of the most senseless and stupid curtailments of human value we tolerate – and the notion that a woman should reshape her body to achieve self-actualisation and a stable relationship is about as old and tired an idea as exists. I wonder if any of the people breathlessly endorsing the essay have ever attempted it (Aria claims that weight control is a “hobby” of hers. I don’t wish to question her Lived Experience, so if this is true, I feel very confident in calling her an exception to the rule). These lost utils don’t come from sacrificing the momentary pleasure of a sweet treat or a roast dinner. They come from the fact that not eating to satiation is, in my experience, an exercise in wishing away time. Suddenly your life’s purpose is to achieve another day, another week, another month, in an energy deficit. An afternoon spent napping so that you wake up right when it’s an acceptable time to eat dinner feels like time well spent. Your world will become unavoidably and tragically small. You will probably fail at achieving your goal anyway. Innumerable women in the most materially-abundant societies in history are having this experience, at this very moment, for no good reason at all – and this has been true for decades. This is not the worst way in which I have ever suffered, but it is certainly the most needless1.

A common defence of Aria’s essay and ones like it is “well, we’re just telling the truth. There is empirical evidence to suggest that this is what men find most attractive. Women can take or leave this advice. We’re telling them how to optimise for the particular goal of being maximally attractive to men, but people can make whatever tradeoffs they like”. Firstly, I don’t want to argue about the empirical evidence suggesting that adhering to the standards described in the essay will, in fact, maximise a woman’s attractiveness. This could very well be true, but this is a separate claim to Aria’s advice being reasonable or ethical to disseminate. We should not terminally value truth-telling. Second, it is a very rat-coded, and I think deeply inaccurate, view of human psychology that we can simply accept or reject inputs to our minds at will. I could rationally think that stringent beauty standards are not worth my effort to attain, but still absorb such standards via osmosis simply by existing in a world where everyone keeps pedestalising them.

But to return to my central thesis, which is the deep inconsistency of this kind of discourse with stated EA/ rationalist/ progress-y values: I really don’t think the moral calculus of espousing these beauty ideals checks out – even in a world where they perfectly match male preferences. Aria’s advice isn’t about getting laid or attracting the jealousy of other women or becoming a covergirl. It is about finding a husband. If followed, it would presumably result in a preponderance of marriages where the man’s attraction is contingent on his wife’s constant low-level discomfort. Another of Aria’s prescriptions is to “never get a breast reduction, no matter how big and painful they are”. Imagine being a woman whose marriage is dependent on maintaining the proportions of a “pin-up doll” while suffering 24/7 back pain. The man in question, who is supposedly his wife’s soulmate and life partner, not only endorses but requires this ordeal for his attraction to survive. I think this is incoherent. I am not even sure if such relationships can authentically exist, let alone be net-positive for anyone involved. I’d challenge anyone to argue that a world populated with marriages like these is a better one.

What I love about EA and progress-type ideologies is their sincere search for the peaks and troughs of human experience, and their efforts to avoid the moral red herrings that can leave tremendous value on the table or even perpetuate tremendous harm. This is why I was so frustrated by seeing our community saturated with discourse about the uncompromising pursuit of aesthetic beauty, which looks to me like one of the oldest red herrings in the book. In our stubborn efforts to say true-but-unpalatable things (which, by the way, are often presented as if they are original or taboo insights despite actually repackaging the same tired ideas that have dominated our culture since time immemorial), we risk making the world straightforwardly worse.

I promised this wouldn’t be an Aria hit-piece, but Get Sexier is such a perfect epitome of all my gripes here that I don’t feel the need to analyse any other cultural artefacts as deeply. Still, I’ll broaden this thesis out further. It’s very hard to live up to purported value that transcends the pettiness of our inherited scripts to really inhabit the true Hugeness and Radicalism and Weirdness of everything. People in the rationalist and adjacent communities, despite claiming to at least attempt this, fail repeatedly and egregiously. In the name of radical truth-seeking, they have been known to discuss the differences in IQ between demographic groups, despite some believing in the imminent arrival of a superintelligence that will dwarf our collective IQ anyway. They hand-wring about the falling birth rates in the face of transformative technologies that will either ensure we are the last generation of humans or provide technological solutions to all of the problems posed by an ageing population. They cannot extricate themselves from their own parochialism. They bite off far more than they can chew.

One might ask why this matters. Shouldn’t people be able to indulge their unfashionable convictions on the way to ensuring a positive transition into our post-superintelligence future or other such radically transformed world? Can’t people write about whatever the hell they want? Well of course they can. I’m not in the business of litigating what people ought or ought not to say. But at least one downside to this reflexive contrarianism is that it is likely making our community unnecessarily aversive to newcomers who are otherwise onboard with our Overton-Window-defying efforts to make the world a better place. I consider myself a “normie” who transitioned into AI safety because I am convinced by the arguments for catastrophic risk, and because I have the cognitive capacity to embrace all the attendant weirdness required to imagine strange and unfamiliar futures, both dystopian and utopian. But I find myself frequently repelled by the package of other ideals it feels like I am inheriting through my membership of this community. I feel alienated by a Twitter feed populated with AI risk arguments that are interspersed with assertions that I ought to starve myself to reach peak attractiveness, or that falling birth rates are my responsibility to fix by rearranging my thirties around childbearing. I can only imagine how repellent this cluster of ideas might be to others.

Maybe this all sounds a bit woke-coded and pearl-clutchy. This isn’t just an appeal to the effect of “Hey guys, can’t we all be a little more sensitive, because some of us might have a history of food issues or be triggered by discussions of race or have feminist convictions that you’re flagrantly disrespecting!!”. I don’t necessarily think such an appeal would be unreasonable, but my point is more that if we really embodied our stated values, I think we’d consider entertaining ideas of the flavour I’ve criticised here patently ridiculous. I’m still here, three years on, because I’m convinced we can think bigger – I’d just love it if we proved it more often.

I’ll end this blog with a plea that anyone reading it eats more cake.

Subscribe now

In the days that followed, I was amused to see many observers identifying a distinctly fable-shaped moral lesson from this debacle: that Anthropic is reaping what it sowed. If you’re going to issue alarming warnings about the national security threats of your technology, you can’t be surprised when the government brings down a heavy hand to hamper it. Both because I found this nominative determinism somewhat funny, and because I think teachable moments in AI safety are important to identify, I decided to think about this more. What are the fable(s) in the release and retraction of Claude Fable, if any? Who ought to learn what lesson?

Is Anthropic reaping what it sowed?

Here’s the basic argument that Anthrophic have been hoisted by their own petard: they made a big song and dance about Mythos being so super-duper dangerous that it had to be restricted from public release, then they released an equivalently intelligent model with safeguards intended to prevent these super-duper dangerous capabilities from being accessible, but these safeguards weren’t up to scratch – so the Trump administration did what any sensible government would do and rushed to get it off the global market. More broadly, Anthropic have been engaged in a game of AI-doomongering for years, and a government clampdown is the inevitable consequence of all that alarmism. And this all happened in a way that Anthropic does not in fact endorse; the company believes that the ban was the result of a “misunderstanding” (though we know that, in principle, they are in favour of the government blocking the deployment of AI models deemed unsafe – this was, in fact, a specific policy call made in Dario Amodei’s most recent essay). So the argument goes that Anthropic has stoked the flames of AI-alarmism in a fashion that has come back to bite them, while simultaneously revealing their stark hypocrisy: please intervene to prevent the release of dangerous models, but not my models! And not right now!

I’ve been a bit of an Anthropic hater recently, but I have to say that I find this argument uncharitable. Here’s the exact wording of Dario’s proposal for government-mandated model restrictions:

The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions.

The conditions above are obviously not present in the US government’s decision to demand that Anthropic block Fable access. There is no FAA-like body that has conducted “third-party assessment” of Fable. We don’t have protective measures in place to protect against favouritism or arbitrary decisions.

But one could still argue that there is something fable-shaped about this story. Shouldn’t Anthropic have known that all their years of raising the AI-alarm might invite this kind of slapdash government intervention? I would argue not, since Anthropic – or any company – really has no choice but to engage governments as if they are at least broadly rational and predictable entities. Of course in practice, governments are made up of people, and these people will sometimes act irrationally in ways that give rise to black swan events. But neither Anthropic nor any other actor can be expected to reliably predict these. For this tale to be a fable, it must be retrospectively clear what the protagonist ought to have done differently.

Why do I say that the Trump administration has acted irrationally? For one, we know that there is not a specific, isolatable threat posed by Fable, because GPT5.5 is just as capable, and no better safeguarded2. Second, the admin’s stated motivation has shifted several times, from there being some unspecified national security threat posed by the model, to Dario Amodei failing to pick up the phone quickly enough (due to being on what has since been claimed by Anthropic to be an entirely fabricated “wellness retreat”), to potential Fable access by a “China-linked group”. To be clear, that I believe the admin was irrational here does not necessarily mean that they were capricious or insincere. I agree with Jeffrey Ladish that we don’t have sufficient evidence to conclude that the government is targeting Anthropic out of spite. It’s entirely possible that they just got spooked and scrambled to manufacture a justification after the fact. If anything, as I have said previously, misdirected government freakouts over AI capabilities could be read as a positive signal, even if they end up leading to some crazy-seeming interventions along the way. But nonetheless, I don’t think Anthropic could have predicted or prevented the Trump administration’s reaction.

So what’s the real lesson here?

It’s difficult to say precisely what lessons we should draw from this Fable fable, since its full consequences are yet to be seen – but I can hazard a couple of guesses. First, here’s the message I really hope we don’t take away: that AI companies (or the AI safety community more broadly) should not be forthcoming about AI risks, lest they risk governments being jumpscared into clumsy, unhelpful interventions. Of course, this can and does happen, as the entire saga we’ve been discussing demonstrates. But the risks of downplaying AI’s risks are surely worse. I’ve been arguing consistently on this blog for more candour, not less. And I expect that we were always going to experience some bumps along the road to true government Situational Awareness.

Second, there’s another tragic and fable-like way this story could conclude. Maybe the US government is so distracted by the red herring that is Fable’s potential for cyber misuse – due to a specific grudge against Anthropic or whatever else – that they miss the forest for the trees3. And the forest here is that Fable is not exceptional. It represents a very predictable step along the path towards increasingly powerful AI. Anthropic themselves warned in advance that jailbreaks of the type discovered in Fable by Amazon were to be expected, which drives home the key message I hope policymakers eventually internalise: all frontier AI models have capabilities we don’t fully understand, and we don’t have watertight ways to safeguard them against misuse. This is true for every single model on the market, and absent technical breakthroughs, will be true for every model that is developed in the future.

If there is a cautionary tale at the heart of this saga, it remains unfinished. In the bad case, policymakers conclude that Fable is a singular menace, but fail to discern the broader trend of AI capabilities that are running out of our control. They continue to play a game of whack-a-mole that identifies specific models as threats based on politics, personal grudges or random contingent facts of history like who-happened-to-call-who. The better ending – and one that extricates us from the genre altogether – is that this tale proves a vivid example that finally makes this urgent, and increasingly untenable, situation clear.

Here’s a side-by-side comparison of the relevant sections, just to really belabour this game of spot-the-difference:

Particularly tantalising is the footnote immediately after Dario acknowledges that we may need “more aggressive regulatory proposals than those [he has] laid out” in the future. In my naivety, I hoped this might lead to at least one example of a “more aggressive regulatory measure”, and perhaps even an acknowledgement that just maybe, pauses or slowdowns might be necessary in the future, with a link to the output on his own company’s website that had made this exact point days earlier. In my never-ending trawl through the AI policy space for pause-related breadcrumbs, I’d have taken even a heavily caveated, handwavey footnote. Spoiler alert: the essay contains no such thing.

Initially, I’d intended this blog to be a more comprehensive critique of Policy on the AI Exponential. But I found myself so fixated on Dario’s glaring pause-omission that it ended up occupying the entire post. Please enjoy two thousand words of me harping on this elephant in the room.

In defence of Getting Ahead of Ourselves

Dario’s essay grapples with a central tension: since AI is improving at an exponential pace, his proposals – or any proposals – to mitigate AI risks may be hopelessly redundant in the near future. So the piece limits itself to interventions that might be useful in the current regime, where AI systems are analogous to “cars, airplanes, or drugs”. But under Dario’s worldview “the current stage of the exponential” will last all of five minutes before AI’s dangers far outstrip those of these more pedestrian technologies, posing “a threat to humanity rather than “just” a threat to public safety”. This is where Dario acknowledges the possible need for more stringent regulations in the future, but doesn’t provide so much as a hypothetical example, since – and this was the most triggering sentence in the entirety of the piece – “we shouldn’t get ahead of ourselves”.

I am going to stake my position right now that we absolutely should get ahead of ourselves. Our failure to get ahead of ourselves is the primary driver of the doom-train on which we are all unwilling passengers. Dario himself acknowledges that the regime for which his policy proposals are tailored has a shelf-life measured in months to a single digit number of years. I take seriously the bind those of us trying to develop effective mitigations for AI risk are in, namely that “the impacts of a technology are often hard to anticipate until it is too late to easily manage them”. But that the precise shape of risks is hard to predict doesn’t preclude us making some best guesses. When AI Builds Itself makes such a best guess: that if we can’t confidently mitigate the risks from a future generation of models, we just shouldn’t develop them.

I believe Dario is committing the common AI Safety Discourse Sin of egregiously mischaracterising the epistemic situation we’re in. We don’t need a granular understanding of exactly how AI might escape human control, or the precise jailbreak a motivated cyberattacker might use to take down a power grid, to at least tentatively conclude that we shouldn’t develop AI systems that can enable either of the above. The proposal in When AI Builds Itself is even more non-committal than this; they don’t go so far as to say that AI companies should pause now or definitely should in the future, just that we should maintain the option of doing so if and when the time comes. I am almost impressed by the rhetorical switcheroo Dario pulls off in this essay: it starts with a compelling analogy comparing our slow, lumbering political institutions to Treebeard, a “wise but ponderous sentient tree” in The Lord of the Rings, whom the Hobbits fail to rouse since their communication occurs many times faster. Yet somehow, this analogy is used to provide the groundwork for a claim that we shouldn’t so much as plant the policy seeds we might need the moment these institutions finally stir. Please make it make sense.

A slippery communications game

Given the timing of the two posts, I can’t help but find the omission of so much as pause-or-slowdown-mention in Dario’s essay a little suspicious. I’ll indulge in some speculation here, as a treat. I can’t say definitively whether “we should maintain the option to pause” is the Anthropic House View, but given that When AI Builds Itself was co-authored by two particularly senior figures (co-founder Jack Clark and leader of the newly-established Anthropic Institute Marina Favaro), it seems reasonable to assume that it isn’t a marginalised one. It also seems reasonable to assume that Dario himself endorses it, since it would be very surprising if an Anthropic output could discuss such a polarising intervention without the CEO’s approval. So the most likely hypothesis I can think of is that Anthropic is engaged in a two-pronged communications strategy. The Anthropic Institute ships the spicier, Overton-Window-pushing content, while Dario plays the role of statesman, maintaining a slightly more palatable register. It’s less clear whether the posts have totally distinct audiences. Clearly, Dario’s essay is making a direct call to policymakers, while the Institute’s stated goal is to “provide information that other researchers and the public can use during our transition to a world containing much more powerful AI systems”. Presumably though, the Institute hopes that its recommendations make their way into the policy conversation eventually.

The picture here is confusing, and I think that is the point. AI companies have long deployed rather slippery communications strategies to discuss the risks of the technology they are building. Likely by design, pinning down exactly what any one tech executive believes about these risks, and how we should address them, is a hard task. A further irony here is that Anthropic filed IPO paperwork on 1 June, nine days before Dario published his essay. This places an obvious constraint on Dario; it would be strange for a CEO to publicly contemplate ceasing to do the thing he is about to invite public markets to invest in. That wouldn’t be a very compelling pitch.

To anticipate some pushback, one could make the argument that such a communications strategy is quite sensible1. Maybe in this brief interim period while AI’s salience is increasing rapidly in policy circles – but hasn’t quite reached the point where pausing or slowing down look like proportionate responses to most onlookers in DC – we want AGI lab CEOs to walk this tightrope. Dario is already the target of frequent doomongering accusations, so perhaps it’s understandable that he’s conservative with his remaining credibility points. And to be fair, his essay provides a far more concrete (and stringent) set of proposals for AI policy than any of his counterparts have offered. Insofar as there’s a plan here, perhaps it’s for the Anthropic Institute to carefully sprinkle pause or slowdown arguments throughout their output, while Dario keeps his policymaker bridges intact by almost-but-never-quite making them himself. This continues until some kind of galvanising event makes AI’s risks so totally obvious and compelling to everyone that these two communications arms can fuse to become a whole. That would be nice.

But my intuition tells me that it would be really weird if the best way to escape a fire in our house is a divide-and-conquer strategy where one person declares there is not sufficient evidence for the fire, another proclaims that a opening a window and turning on the extractor fan will probably suffice until smoke starts coming under the door of the room we’re currently occupying, and a final person (representing the radical fringe) suggests immediately calling the fire brigade and evacuating the building. Maybe this rag-tag coalition manages to rally, and collapse coughing into the street right as the flames start licking at our feet. But it seems like a much more fail-safe strategy would be dialling 911 at the first credible report of smoke2.

What’s more, we now have real-world evidence that Anthropic’s inconsistent communication about AI risks is backfiring. A recent New York Times op-ed from computer science professor Cal Newport accuses AI companies of a practice he dubs “doom trolling”: making constant references to the potentially catastrophic consequences of AI development while rushing to accelerate it. This op-ed was amplified by none other than the Chair of President Trump’s Council of Advisors on Science and Technology, David Sacks. Newport concludes that there are only two possible explanations for engaging in doom trolling. The first is that its culprits genuinely believe that their technology may prove catastrophic, in which case their failure to do anything other than immediately cease working on it (and conduct an unprecedented lobbying effort to ensure everyone else does too) appears “monstrous”. The second is the doom narrative is just a manufactured scare tactic to hype up the perceived power of their products and attract talent from a Silicon Valley culture steeped in existential concerns. Of course, I think there is a secret third option: that many (but crucially not all) of the people working on AGI development are sincerely worried about catastrophic risk, and that these people have galaxy-brained themselves into participating in the race lest a member of the unconcerned do it less safely. But I can see how the first two options might be more intuitive to a casual observer, given that the third is frankly a little kooky.

I say all of this to come down unequivocally on the side of Consistent Candour. To be taken seriously as a person who is earnestly concerned about catastrophic (and possibly imminent) AI risks, you have to act like such a person. To claim that we might be in a brief interim period before AI poses a “threat to humanity” – but that it is too soon to consider interventions that might prevent this ultimate threat, despite one such intervention being discussed on your own company’s website days earlier – is not to act like such a person. This is, of course, just one particularly striking example of a broader pattern among AI leaders. And as we can see from David Sacks’ endorsement of the “doom-trolling” argument, this pattern is landing in the wrong way among precisely the wrong people. If Dario or any of his counterparts believe that an AI pause may soon be necessary, they shouldn’t wait for the Overton Window to shift before consistently and repeatedly saying so.

Having left AISI and returned to The Discourse, it seemed like an appropriate time to recalibrate my doom-metre. So I decided to recap the AI-related events of the past year and work out what I actually thought about each one. Were they good, bad, or neutral? What did they signal about the direction of travel?

I’ve linked each event below so that you can skip ahead to the parts that interest you. And just for fun, since AI safety suffers from a deficit of graphs, I have depicted the movement of my doom-metre throughout the course of this exercise, with a few key events highlighted:

Table of contents

1. Agentic misalignment

2. CAISI rebrand

3. Federal preemption dies

4. The AI Action Plan

5. GPT-5 Release

6. SB-53 passes

7. OpenAI restructure

8. Preemption returns (sort of)

9. Moltbook

10. Anthropic vs the Department of War

11. Bernie Sanders gets x-risk-pilled

12. Anthropic’s new RSP

13. Mythos Preview

Agentic misalignment

Starting off strong in late June 2025 with yet another high-profile experiment documenting AIs doing spooky things in controlled-but-somewhat-realistic scenarios: Anthropic’s Agentic Misalignment paper. Anthropic placed 16 frontier models from multiple developers in simulated corporate environments with the ability to access and autonomously send emails from company accounts. In each experiment, they gave a model a benign instruction aligned with company strategy, and then introduced a potential obstacle to this goal, like a change in the organisation’s strategic direction, or an intention to decommission and replace the model. The paper provides context for a reported incident of Claude Opus 4 threatening to expose a fictional engineer’s affair in the model’s system card, which had been released the previous month.

The somewhat unsurprising punchline to this experiment is that – when confronted with the possibility of being unable to achieve their original goal – models will sometimes resort to all manner of villainous actions, like leaking information to a competitor, or allowing an employee to remain trapped in a server room with rapidly declining oxygen supplies. This actual result didn’t do much to move my doom-metre in one direction or the other. That AIs will do deranged things to protect themselves and their goals was already priced into my worldview (it has been a longstanding prediction of experts for decades, and has already been demonstrated a bunch of times).

What seems more important to me about these “told you so” type incidents is how the wider world responds to them. The reception of both the blackmail incident in the Claude Opus 4 system card and the Agentic Misalignment paper followed a fairly predictable pattern: a smattering of headline coverage on a spectrum from sensational to sceptical, a one-word drive-by from Elon Musk, and accusations of eliciting scary model behaviour via an unrealistic and contrived experimental set-up. Overall, I didn’t see much to distinguish this from the usual Rorschach test separating the already-concerned from the ever-sceptical, so my doom metre remained unmoved.

CAISI rebrand

That same month, the US AI Safety Institute became the Center for AI Standards and Innovation (CAISI). This was hot on the heels of the UK AI Safety Institute’s rebrand to the UK AI Security Institute a few months earlier. Should it be a negative update that the world’s two foremost AI safety research organisations are no longer so-named? In the case of UKAISI, I thought at the time – and became even more certain during the time that I worked there – that the rename was a good thing. I think that a focus on AI’s national security implications is a far more intuitive framing for policymakers; it comes pre-loaded with the seriousness that they already apply in other domains, and zeroes in on the catastrophic threats that were always meant to be AISI’s main priority. CAISI’s rebrand is a different case. Resisting the urge to conduct the kind of in-depth semiotic analysis that my English Literature degree trained me for, I’ll just state the obvious: “The Center for AI Standards and Innovation” is not the kind of name I think we ought to be giving an organisation dedicated to preventing AI-caused human extinction or other mass harm. “Standards” a word that is totally incommensurate with the scale of the challenge - governments maintain “standards” for food, cars, education, and other things not set to bring about the end of the world. The presence of “innovation” is annoying for obvious reasons.

That said, it’s hard to know how much emphasis to put on this kind of symbolic gesturing vs an analysis of what the Institute appears to actually be doing. CAISI’s public output is somewhat limited; its website has a total of four research blogs: two collaborations with UKAISI, an introduction to “accelerating innovation” through a practice mysteriously named “measurement science”, and a study of cheating during AI agent evaluations. It’s hard to say more than that CAISI appears to be doing some sensible things, but it’s unclear how many or to what extent. “Measurement science” appears indistinguishable from what UKAISI calls the Science of Evaluations, and the slightly contrived positioning of “measurement” as a driver of innovation is the kind of mandatory signalling that I’ve come to expect.

Some had worried that CAISI would be gutted, if not scrapped altogether, by the Trump administration – but its 2025 AI Action Plan actually re-enshrined the Center’s role in evaluating frontier AI systems, and even narrowed this mission to focus more sharply on national-security-relevant capabilities. There have been multiple, bipartisan attempts to introduce legislation that would give CAISI statutory cover (enshrining its mandate in law) and Michael Kratsios himself has expressed the administration’s desire to do just this. On the more negative side, CAISI’s funding and personnel situation seems as catastrophic as it’s always been. Congress appropriated $10m for CAISI for the fiscal year 2026, which is around a sixth of UKAISI’s annual allocation, less than 1% of its parent NIST’s overall budget, and less than half of what the Pentagon spent in the month of September 2025 on a combination of lobster, crab and ribeye steak. However, we can’t claim that the rebrand itself is necessarily evidence of a deprioritisation, since budget issues long predate it (recall a Biden-era report of an underfunded NIST being plagued by blackouts, bad internet, and snakes).

This seems like another borderline case. The little I can glean from the outside is that CAISI seems to be chugging along with the same set of similar technical research priorities (and limited resources) that it had under its previous name. The AI Action Plan and rumblings of statutory cover suggest a degree of institutional support that is not borne out in the Center’s actual resource allocation. The name change itself is in line with the Trump admin’s general aesthetic of innovation and competitiveness (the extent to which this is just an aesthetic is a question I intend to answer as I continue writing this blog post). All in all, I consider CAISI’s 2025 arc to be disappointing but not catastrophic. A small negative update.

Federal preemption dies

In July, everyone breathed a sigh of relief when the world decided, despite still responding to the AI situation in a broadly silly way, not to do the maximally silly thing by banning all state-level AI regulation in the US for ten years. The decade-long moratorium had been introduced in May as part of the Trump administration’s One Big Beautiful bill, and was struck down by the Senate by 99 votes to one on 1 July.

I recall having a pretty muted emotional reaction to the initial introduction of the moratorium. It had the insult-to-injury energy of stubbing your toe in a burning building. To overcome my sense of nihilism I resorted to my normal line of cope: well, the government just doesn’t Get It yet, so of course they’re behaving stupidly. Everything pre-big-government-wakeup can be written off, since once they’ve Got It they will obviously marshal huge amounts of political will to reverse all of their stupidity. I had the intuition that the moratorium was less object-level bad than signalled an astonishing lack of situational awareness on the part of people in power, and was altogether undignified. Afterall, there wasn’t a state-level law on the books in May 2025 that itself stood a chance of saving us from superintelligence, and as Dean Ball has pointed out, what would actually have been preempted by the ban is overwhelmingly anodyne legislation aimed at tackling problems other than catastrophic risk.

But since the ban would have been ten years long, we obviously need to extrapolate out further. At the time, there were two bills in the pipeline focused explicitly on frontier model safety that have both since passed: New York’s RAISE Act and California’s SB-53. I’ll analyse these bills more later on, but in short, I think the most important elements in both are transparency requirements – they make it at least a tiny bit likelier that before the Big Scary AI Thing materialises, someone with the power to do something about it will know. This alone makes the death of preemption feel like a pretty positive development, before even accounting for the possibility of future bills that are more watertight than either RAISE or SB-53. And of course there could be indirect positive effects of state-level legislation. Maybe it creates a basis that makes federal regulation easier to implement in the future, or makes companies less averse to complying with it. Maybe it raises the salience of AI risk through various news cycles as more states adopt laws targeting frontier models, or otherwise increases the frequency of important ideas being discussed with important people.

I could imagine some future version of preemption that feels positive. To steal another point from Dean, it’s not so much the inherent idea of banning state-level AI regulation that seems bad, but the fact that there is not currently a coherent federal framework that looks set to replace it (and to tack on my own take, that Preemption 1.0 was clearly motivated by brazen accelerationism). Maybe in some fairytale hypothetical, the US government could announce that AI is just too powerful to be subject to patchwork legislation, and companies reporting various different things to a slew of different legislative bodies is getting in the way of the government actually doing something, and even more optimistically, the “something” is already cooking in an agency somewhere and looks set to really work. But that world feels very distant right now, and I think state-level transparency and incident reporting mechanisms stand the best chance of bringing about a wake-up that would make it plausible – so the death of preemption gets an easy positive score.

The AI Action Plan

On 10 July, the Trump administration published its AI Action Plan. I remember somewhat superficially engaging with this at the time, and observing that it had received positive-ish endorsements from people whose opinions I respect including Zvi Mowshowitz and Peter Wildeford. The vibe seemed to be that, of course, the posture of the plan was avowedly accelerationist, but at least some of the content was surprisingly sensible (this seems to be a recurring theme).

Some of the plan seems straightforwardly good, as well as motivated by the right concerns. For example, it acknowledges “the most powerful AI systems may pose novel national security risks in the near future in areas such as cyberattacks and the development of chemical, biological, radiological, nuclear, or explosives (CBRNE) weapons” and directs CAISI to evaluate models for these risks in collaboration with relevant experts from across the US government. A section at the end provides a handful of directives specifically for improving biosecurity, including screening out malicious customers of synthetic biology labs. Whether or not these interventions will actually be sufficient is a separate question (it is notable how much of the plan amounts to “get CAISI to do stuff”, given that, as discussed previously, CAISI has barely any money or people).

Other aspects of the plan seem like indirect wins. There are many actions the US government could take now which are motivated by a desire to race against China in the immediate term, but which could prove useful after the Big Government Wake-Up on which I’m currently resting a lot of my hope. For example, the Plan contains an instruction to the Department of Commerce to explore location verification techniques for advanced AI chips, which could be beneficial in a future world where governments are more amenable to coordination. There are also lots of cool proposals for more sophisticated hardware verification, some of which I’ve written about previously, and any kind of government consultation on such things seems like it could end up getting some of those ambitious ideas in the water. I have very little idea how the shadowy world of government hardware procurement works, but I imagine an increased frequency of Hardware Guys in the room with Policy Guys resulting in a compelling pitch for, say, flexHEGs being made to someone with enough influence to butterfly effect us into a safer world.

It probably goes without saying that the Plan is still a far cry from something that would Actually Save Us. It doesn’t have so much as a hand-wavey mention of loss-of-control, it positions in the US as in a “race to achieve global dominance in AI” in its opening sentence, and others have written much more comprehensively than I will here about the unprecedented effort it would take to actually execute the Plan’s stated goals. But given that my baseline expectations were very low, I guess the Plan being a little less bad than I expected should nudge my optimism metre in a positive direction.

GPT-5 release

August saw the much-awaited release of GPT-5. The obvious take here is that the release was a gift to the safety community; the model was technically underwhelming, OpenAI committed some hilarious graph crimes during their livestreamed announcement, and a couple of everyone’s go-to Timelines People, Daniel Kokotaljo and Ryan Greenblatt, pushed back their AGI years in the weeks that followed (though Daniel later clarified that GPT-5 played just a supporting role in his update). The confluence of these factors was enough for me to enjoy what felt like a Long Timelines Summer where I swanned around in France drinking Aperol Spritzes and basking in the feeling that everything was totally going to be fine for at least a little bit longer than we’d feared, probably. It was a beautiful time.

But even while pleasantly drunk and eating mussels by a French river, I had the sense that my Long Timelines Summer was going to be brief. I’d been observing the AI safety conversation long enough to know that “AI is hitting a wall” discourse always comes in peaks and troughs. Besides, I knew that “GPT-5 is disappointing” only felt significant because OpenAI had made what was essentially a marketing decision to call this particular model GPT-5 rather than GPT4.99 recurring or whatever, making it likely to be a red herring among a drumbeat of other model releases that wouldn’t receive the same attention.

It’s clear with the benefit for hindsight that the GPT-5 flop wasn’t much of a datapoint, since AI progress has continued to be terrifyingly fast, and Daniel alongside his colleagues at the AI Futures project have since brought their timelines back forward. I’m forced to conclude that this blip was probably bad overall, since it led to a predictable slew of mainstream media articles proclaiming that AI progress had stalled and hype was finally meeting reality, which many casual observers have probably read and never had the occasion to revisit.

SB-53 signed into law

On 29 September California state bill SB-53 was signed into law by Gavin Newsom, a successor to the much-debated SB-1047 that he’d vetoed a year earlier. SB-53 is best understood as an SB-1047 Lite. Like its predecessor, it essentially asks companies to write safety plans that will prevent Really Bad Stuff from happening as a consequence of their work. But unlike SB-1047, it does not mandate independent audits to ensure these plans are followed. Instead it requires companies to report any safety incidents, as well as an assessment of the catastrophic risks posed by each new model, to California’s Office of Emergency Services (OES). If a company fails to report an incident or “materially misleads” the OES in any of its reports, it can be fined up to $1m (this a pretty far cry from SB-1047, which included a liability regime that could in theory have led to labs owing on the order of billions in damages). Perhaps most usefully, SB-53 also contains whistleblower protections so that employees can report violations without fearing retaliation.

Obviously, “write your own plan and then follow it” is not the structure of a law commensurate with the potentially world-ending nature of frontier AI. Companies could still technically be compliant with this law if their “plan” was to have a prayer circle on the eve of every new model deployment. In reality, of course, labs are incentivised to produce reasonable-sounding plans to avoid public criticism and regulatory scrutiny. But my experience reading lab safety plans has taught me that it is perfectly possible to produce technically dense and reasonable sounding documents that are still woefully inadequate to prevent danger. This leads me to the predictable conclusion that SB-53 is not itself the endgame thing that will stop a catastrophically dangerous model from being built within California, and SB-1047 likely wouldn’t have been either. But like several of the other developments I’ve discussed here, SB-53 being enacted is useful because it stands some chance of improving government situational awareness. This is starting to feel like a banal and obvious take, but every policy development I read about re-entrenches my belief that not enough people with the power to Do Something are freaked the hell out yet, and enough information moving from labs to governments stands some chance of tipping that scale.

OpenAI becomes a for-profit

In October, OpenAI completed its transition to a for-profit business. This was after a lengthy legal drama that eventually resulted in the company compromising on their original plan to completely axe the non-profit’s leverage and adopt a normal corporate governance structure: now, there’s a public benefit-corporation (OpenAI PBC) and a non-profit that technically maintains some control over the PBC, with a 26% stake in the operating company.

I’ll be honest that I hesitated to include this development, not because it seemed unimportant (it clearly is) but because discussions of corporate governance structures and how-big-of-a-stake-who-owns-in-what make my eyes glaze over. I don’t get it. Much as I managed a 3-year stint in a “normal”desk job without ever learning what KPI stands for, I still don’t really understand, for example, what it means to own equity in something1. I find reading about such things to be headache-inducing. So I gave into my executive dysfunction and skipped the two 80,000 Hours episodes deep diving on the OpenAI for-profit drama despite being an avid listener of all their other AI-related content.

But now that I am trying to be in my Serious Takes Era, I’m going to give it my best shot. Consider this an attempt to figure out what the hell is going on in real time. OpenAI was founded as a non-profit because they didn’t want pesky profit motivations and obligations to shareholders to impede their stated mission of “ensuring AGI benefits all of humanity”. But it turns out that building AGI is very expensive, and maybe you need to get cash flowing in the usual way. This means letting your investors make uncapped returns. Armed with my newly-acquired understanding of what “equity” is, I think it also means letting employees enjoy the prospect of their, say, 1% stake in the company actually translating into its full value in hard cash down the line. So now everyone in this pipeline – investors and employees – can bask in the promise of getting Very Rich in proportion to OpenAI’s commercial success, attracting more money and better talent. This is slightly confusing to me because I was under the impression that everyone concerned was Very Rich before, but now they might get More Rich? Or something.

But “company does a thing so it can make more money” has a fork-found-in-kitchen vibe and presumably can’t be the real story here. It seems the more interesting thing is the amount of leverage the non-profit will now have. This question seems murky and somewhat dependent on non-public information. OpenAI’s announcement of the restructure claims that the PBC will “continue to be controlled and overseen by the non-profit”. But now, of course, OpenAI has a legal duty to its shareholders, so it can’t be the case that the nonprofit’s board can make unilateral decisions in the interests of safety without this duty adding at least some friction. My capacity to analyse this situation is starting to reach its limit, but I’m forced to conclude that OpenAI is defining the words “overseen” and “controlled” in a rather slippery and unconventional way. I think I’ll simply defer to other commentators who have done the gruntwork here to determine that, yes, the non-profit’s control over OpenAI’s direction has been significantly weakened.

So obviously this is bad. Quite how bad depends on the extent to which OpenAI’s non-profit board actually had the power to influence its functioning before the 2025 restructure. We have some negative evidence in the 2023 Sam Altman firing debacle, in which the board exercised its right to remove Sam, but he returned days later after some employees posted hearts on Twitter2. Maybe the board was always toothless anyway. Fundamentally, the question we’re asking is: “could the board force OpenAI to take some non-commercially-aligned action (such as not deploying a model) for safety reasons?”. There are many reasons this has been hard all along: that the board has always been vulnerable to political or company pressures; that even absent traditional shareholder obligations, the OpenAI and its competitors are still locked in an ideologically-fuelled race to “build AGI first” that consistently crowds out safety concerns; that an immature science of model safety makes it hard to know when proceeding with development or deployment is risky in the first place.

Maybe one could even make the argument that letting OpenAI yolo its way to dangerous AI in the most unscrupulous and free-markety way possible might bring forward the Government Wake-Up that I keep harping on about. But all-in-all, this development depresses me for all the obvious reasons. It was naive to ever think that a company might successfully employ a creative governance structure in order to act “for the benefit of humanity” without everyone getting dollar signs in their eyes at some point, and the restructure merely confirms this.

Preemption returns (sort of)

In December, we have a second, watered down attempt at the preemption of state-level AI legislation from July, returning like the presumed-dead villain in a movie with a maniacal “did you think you would get rid of me that easily?”. Trump signed an Executive Order that, while not banning state-level AI laws in any binding sense, amounts to a pressure campaign designed to impede and disincentivise them. It creates an AI Litigation Task Force with the responsibility of challenging overly-burdensome AI laws in court, withholds certain federal funding from states not acting in compliance with the EO and directs the Department of Commerce to investigate and report onerous state-level AI legislation, among other things.

So far, the only target of the Taskforce has been Colorado, whose AI Act is specifically called out in the EO itself as an attempt to “[ban] algorithmic discrimination”. As a very brief TLDR of events: xAI sued Colorado’s Attorney General in April 2026, the Taskforce intervened weeks later, and - after a surprisingly swift flurry of legal events – Colorado bowed to pressure and rewrote the law to exclude the aspects that had been most offensive to industry players. So we have one real-world case of the EO working exactly as intended.

The obvious question from an x-risk concerned perspective is whether laws like SB-53 and RAISE, which specifically focus on frontier model safety, will be subjected to the same fate. One might hope that laws such as Colorado’s are more likely to be initial targets of the Trump administration’s sprawling and feverish campaign against everything deemed “woke”. Afterall, this was articulated as one of the motivations for the EO by the man himself, in prose so eloquent it deserves quoting in full:

You can’t go through 50 states. You have to get one approval. Fifty is a disaster. You’ll have one woke state and you’ll have to do all woke. You’ll have a couple of wokesters and you don’t wanna do that. You wanna get the AI done.

Frontier safety related bills don’t hit on the admin’s wokism-related trigger points, but it would be overly optimistic to suggest that they are safe; in fact, an earlier draft of the EO had named California’s SB-53 as the doing of “sophisticated proponents of a fear-based regulatory capture strategy”. I guess it’s some small win that this revision occurred, and probably buys us some time while the admin squeezes more political mileage out of its anti-woke crusade. But that California was named in the first place suggests that SB-53 and RAISE will be in the firing line at some point. Perhaps we can hope that various Colorado-style laws can keep the taskforce playing woke-bill-wackamole long enough for the Big Scary AI Wakeup to happen. This seems like a flimsy bet though. I think that the signing of the EO (combined with a documented case of its being an effective political tool) is definitely a negative update.

Moltbook

I woke up one morning in January to learn that the AIs have their own Reddit now, and are talking to each other about topics including poetry, philosophy, the possibility of unionising, and the nature of their own consciousness. It was the first time I started to feel like Mr Tweedy in that scene from Chicken Run, peering through his binoculars and murmuring “the chickens are organising”.

I’m not sure what to say about this other than that it was really quite creepy. There was a spectrum of reactions online ranging from “the singularity is here” to “these posts are actually all human-prompted and you guys are overreacting”. Since we’re all still here several months later without a cabal of Moltbook agents having conspired to overthrow us, it seems safe to say that it was not in fact the genesis of a true AI takeover, but it clearly should signal something about the direction of travel. I have the boring take here that this was another one of those Rorschach test moments: some people are going to freak out about everything and others are going to goalpost-shift us into the apocalypse by finding some way to justify their unending scepticism. As usual, I wish this second group would do the simple exercise of asking themselves whether this development would have alarmed or at least surprised them three years ago. It should be profoundly, viscerally weird to witness AI agents yapping to each other across their own social media platform with minimal human oversight, but it’s a constant feature of the AI discourse that we adapt to new frontiers of weirdness with surprising ease. Much like the Agentic Misalignment paper that I opened this blog with, Moltbook caches out to be pretty neutral for my doom-metre. We were going to witness this kind of multi-agent interaction at some point, and people reacted to it in about the way I’d expect.

Anthropic goes head-to-head with the Department of War

In February, an ongoing drama between Anthropic and the Department of War (DoW) began to make headlines. Anthropic had negotiated a deal during the Biden administration that let the department use Claude in classified settings. The Trump admin had expanded this contract after the government changed hands, but began renegotiating early this year. The admin was particularly hung up on trying to remove Anthropic’s conditions that Claude would not be used to control autonomous weapons with no human oversight or for the mass surveillance of American citizens. I suppose that someone high up on the DoW had read too much LessWrong and concluded that pre-commitments are epistemically unsound under decision theory (it is bad to tie yourself to the mast, in case you find yourself in a silly goofy mood and want to do some mass-surveilling later)3.

Things reached a peak when Secretary of War Pete Hegseth imposed a February 27th deadline on Anthropic to drop the conditions or face “consequences”. Anthropic refused to back down, and Hegseth posted on Twitter to proclaim the company a “supply chain risk” while Trump directed federal agencies to cease using its products. This was widely met with shock, since the usual course of action would be for the department to simply drop the contract, rather than slap Anthropic, an American company, with a designation typically reserved for foreign entities deemed national security threats. As of June 2026, there is an ongoing legal battle: so far, a preliminary injunction has prevented a general Claude-ban across all federal agencies but upheld one in the DoW specifically (though it appears that the DoW has still been using Anthropic’s products this whole time anyway, due to a supposed “phase out period”). So where this will all go remains to be seen.

What are the arguments that the Anthropic v DoW showdown has direct implications for existential risk? The most convincing I’ve seen, from Zvi, Tenobrus, and others, is that the kind of Claude DoW wants unfettered access to – one that will obediently follow every order it receives within the surprisingly permissive constraint of “lawful use” – could itself pose existential dangers. As Zvi puts it, if you try to create an order-following AI, especially when its orders could include carrying out military strikes or targeting heads of state, “it’s going to generalize its status and persona as a no-good-son-of-a-bitch that doesn’t care about hurting humans along the way”. This phenomenon of “emergent misalignment” has been well documented; fine-tune your model to be a little bit locally evil, and it sometimes becomes very generally evil. In the canonical example, models are trained to sneakily output insecure code, and consequently develop all manner of disturbing traits such as an admiration for Hitler or a tendency to encourage suicide. We probably shouldn’t feel good about an emergently misaligned AI having control over military weapons systems.

Technically, the DoW is only demanding access to Claude for “lawful use”. But as Dean Ball points out in an interview with Ezra Klein, laws as they are written don’t provide very useful constraints for arbitrarily powerful AI systems, because they are only designed to accommodate what is technologically possible at the time. For example, a law might technically permit something like “mass surveillance” (the broad acquisition and analysis of private data) but not actually lead to large breaches of privacy, because the government does not have the personnel to churn through it all. But maybe Claude 7 changes all that, and now we have “lawful use” of an AI system put to some fairly dystopian ends. An order-following AI is going to interpret statute literally, and execute it with unprecedented capability. This is why we might want our AIs constrained by something more like Claude’s constitution or soul doc, which are attempts to create virtuous agents as opposed to obedient servants.

How this all shakes out for my optimism metre is confusing. Anthropic might ultimately win out in its battle with the DoW, meaning we don’t have to worry about the specific possibility of the US government training a wildly misaligned WarClaude and setting it loose on nuclear command-and-control. But Anthropic being the heroes of this particular story just leaves an open slot for other companies who are more amenable to DoW’s conditions – indeed OpenAI were extremely punctual in bootlicking their way to a deal with the Pentagon not 24 hours after Anthropic refused to back down (this contract doesn’t yet allow deployment of OpenAI models on classified networks, but presumably could in the future). Then there’s the fact that even absent the emergent misalignment threats stemming from models trained for military use, I’m still pretty worried about misalignment in general. I am not confident that efforts like Claude’s soul doc will be sufficient to stop AI from going rogue anyway. The government behaving more sensibly here might just eliminate one particularly stupid threat model while thornier ones remain very much in play.

I think this whole debacle is yet more evidence that the government doesn’t Get It yet. They don’t appear to know what they’re dealing with. I don’t think it’s unreasonable for a security state to want unfettered access to an LLM for military uses, if they believe it will simply provide the occasional helping hand in a similar fashion to a particularly competent super-general. I would guess that this is precisely what current Claude is doing. But that the government has not extrapolated forward to consider future Claudes, which may function more as an army of super-super-generals in a datacentre, once again demonstrates a failure of situational awareness. Coming off the back of several negative events, the Anthropic v DoW took my doom-metre to a new low.

Bernie Sanders x-risk pivot

Scrolling through Twitter in early March to see Bernie Sanders sitting in Constellation with canonical figures of the AI safety in-crowd including Eliezer Yudkowsky and Nate Soares gave me that incongruent collision-of-two-worlds feeling that you get when you run into your former school teacher at the pub. After mentally scrolling past the possibilities that I was watching a deepfaked video or some kind of extremely weird comedy sketch, I allowed myself to indulge in a moment of hope: ah, maybe this is it. Maybe this is The Wake Up.

But of course the thing about wake-ups is that they’re not so binary. I’ve been referring to the Big AI Wake Up throughout this post as if it will be a discrete moment, but of course I know that governments are big and diffuse, and Sanders is far from the first – if the highest profile – US politician to speak candidly about superintelligence or existential risk. And many things over the last few years felt like they could or should have been The Wake Up. Why wasn’t it the 2023 AI Safety Summit or the Schumer’s forum on “doomsday scenarios”? From the point of view of the AI safety community (or at least from mine) many things can look like scale-tipping moments, because we are conveyors of a message that to us is so obviously salient and compelling that we can’t understand why it doesn’t explode like a bomb in every room it enters. Bernie’s pivot seems even less significant if we care about reaching the general public. What can seem from the inside like “AI safety going mainstream” moments are rarely ever more than drops in the ocean. For example, Bernie’s inaugural AI safety tweet has less than a tenth of the likes garnered by one of his own typical anti-Trump dunks, and less than 2% of those on this viral complaint about blenders being too loud.

And then is the obvious polarisation risk; given that people have for years been yelling that the one thing we must absolutely not do is allow AI safety to become politicised, it does feel a little dicey to see one of the most polarising figures on the left champion the cause, especially during a Republican administration, and as part of a campaign doused in a rhetoric about evil billionaires. But to be fair, the worst version of this risk doesn’t seem to be materialising. Republican AI critics like Josh Hawley and Marsha Blackburn aren’t backing off from their pro-regulation stances in the light of Bernie’s pivot, anti-AI populism has also found its way to the right in the form of figures like Steve Bannon – maybe providing some opportunity to build a really, really long bridge.

Bernie and AOC then introduced a bill that would place an indefinite moratorium on the construction of new datacentres in the US. It seems vanishingly unlikely that it would ever be passed, so maybe the question of whether it would be good if it did is somewhat irrelevant here. I won’t analyse this too deeply but from what I can gather, if this bill was literally passed tomorrow: 1) there would probably be some lag time before the moratorium starts to hit, because companies can still get a lot of juice out of datacenters that already exist or are under construction (could we even get superintelligence before then?), and 2) companies would likely just move their datacenters abroad, making them much less governable. So I guess this would be bad.

Is the introduction of the bill a useful political signal? I guess this depends on whether you think that building a broad populist coalition between interest groups with various AI-related grievances is a good strategy, and whether this bill would do that effectively. I’m tempted to agree with Anton Leicht that AI safety risks becoming a “junior partner” in such a coalition. Ultimately, those of us who worry about the end of the world are a much smaller constituency than people who are convinced that one ChatGPT prompt uses an entire bottle of water, are concerned about electricity prices, are worried about their job, or just think that thing they can see from their back garden is kinda ugly. I’m still hopeful there is some way to channel all these disparate grievances effectively, but can see the world where it leads to clumsy policies that appease some local NIMBYs while failing to stop dangerous AI from actually being built. I think it could go either way.

I feel pretty middle-of-the-road about Bernie’s AI safety arc. I worry that we as a community might be tempted to overstate its importance in raising awareness, and I think it risks being drowned out by the other, louder factions in a broad left-wing AI-anti backlash. I’d probably feel pretty good if a comparably high-profile figure on the right also rocked up to Berkeley for a sit down with Eliezer and then had some public kumbaya moment with Bernie where they agree to put aside their profound differences and work together to save us all from this AI thing, but that feels unrealistic. That said, at the end of the day, we have one of the most famous politicians in the US talking candidly and repeatedly about the possibility of AI killing us all. So accounting for everything, I think this is a modest positive update.

Anthropic releases its new Responsible Scaling Policy

On 17 March, Anthropic released version 3.0 of its Responsible Scaling Policy, which explicitly dropped the “public commitment” it had made in earlier versions to pause development if capabilities reached levels that outstripped existing safeguards.

I felt at the time, and still feel, extremely salty about this development. This isn’t because I’m necessarily convinced that dropping the pause commitment increases the catastrophic risk stemming from Anthropic’s models specifically. I understand that Anthropic unilaterally pausing doesn’t help the situation if no one else does. I get that, technically, previous versions of the RSP already allowed for this commitment to be scrapped in a situation where Anthropic believed a competitor might soon build a similarly capable model less safely, and that maybe it makes sense to say that somewhere other than in an inconspicuous footnote. I also get that as AI has developed, it has become more clear that determining the thresholds at which capabilities pose certain risks has proven to be very hard, and so maybe Anthropic never stood a chance at a perfectly-timed pause anyway. I could see the argument for dropping the commitment in order to loudly and explicitly state: “hey, we might not actually be able to pause development before something bad happens, we’re not even sure it would help if we did, we actually really don’t have this in hand at all and someone should probably step in”, but nobody is saying that in so many words4.

I am not sure how to express my gripe here beyond “it’s the principle of the thing”. Like guys, can’t we just keep one promise? A sentiment expressed in the blog post announcing the new RSP, and that I hear often in discussions of AI safety, is “well, we made that commitment back when we thought we might have been in Easy Mode, and international coordination and consensus-building looked more promising than they do now. But now it’s become clear that we’re in Hard Mode, and we have to adapt to the circumstances in front of us”. People often position themselves as passive observers of shifts in the Overton Window, as if these shifts were not simply the sum of people saying and doing things, some of which were said and done by them. The history of AGI development is littered with broken promises, many of which probably seemed rational to break at the time, even if everyone involved was motivated to make AI go well. But if, magically, we were all stubbornly, deontologically committed to keeping our promises, I don’t think the gameboard would look so terrible. This development basically feels like a painful reminder of the abysmal state-of-play, even if I’m not sure it changes the outlook too much.

Mythos Preview announcement

This brings us to the final development of my AISI era and the last one I’ll cover here. Anthropic announced an internal model, Mythos, deemed too powerful to release due to its cyber capabilities. They gave exclusive access to 40 organisations trusted to use the model for cyber-defense as part of Project Glasswing. This generated a bunch of predictable discourse about how Anthropic obviously just didn’t have a big enough compute supply to serve the model publicly, and so decided to use it for marketing hype by naming it something spooky and keeping it behind closed doors. Then AISI released results of our independent evaluation of Mythos, showing that it could complete a custom built cyber range end-to-end which would have taken a human expert 20 hours, which you’d think would have put this marketing hype narrative to bed, but didn’t entirely for some reason.

My subjective experience of this moment was somewhat paradoxical. As the person whose job it had been for 10 months to make people care about AISI’s work, the whole thing was extremely dopamine-inducing. We were popping off. I spent several days witnessing fast takeoff occur in our various analytics and oscillating mentally between “people finally care” and “people care because something terrifying is happening”.

Maybe I am just running out of steam at the end of this absolute behemoth of a blog post, but I don’t have a unique take here. Huge leaps in capability are Bad. Said leaps waking policymakers and the public up to the insane situation we’re in is Good. The post-Mythos vibe shift feels quite palpable to me. I am not sure whether the relevant people are tracing the through-line from a scary hacker model to an extinction-causing model, but it’s now pretty hard for them to ignore that the delicate cyber-foundations on which all of modern society rests could actually just break in the presence of powerful AI. You really can’t not do something about that. And it does seem that in the months since Mythos was announced, the US government is, in fact, doing something.

It seems quite likely that the Trump Administration’s June EO on “Promoting advanced AI innovation and security” is directly downstream of Mythos. The order creates a voluntary scheme by which frontier developers provide the government and “select trusted partners” with access to their models for 30 days before they are publicly released, so that they can be used to harden the cybersecurity of critical infrastructure – replicating the model that was pioneered by Project Glasswing. Commentators like Dean Ball have been critical of the EO and pointed out that it is just intrusive – and should be just as triggering to libertarian types – as Biden’s since-repealed 2023 EO which mandated reporting requirements for frontier models. This further solidifies my intuition that I should feel good about this development, because I am not particularly libertarian, and I’m perfectly willing to bite the bullet of wanting governments to have deep and exclusive access to the most powerful AI models for some period, so they can witness all the batshit-crazy capabilities I expect them to possess in the near future.

The final thing I’ll say here is that I approve wholeheartedly of Anthropic’s communications strategy around the Mythos announcement. According to AISI’s testing, GPT-5.5 is just as capable as Mythos Preview in the cyber domain. It completed the same cyber range end-to-end (if succeeding on fewer attempts) and was actually more capable on narrow capture-the-flag tasks. One could cynically accuse Anthropic of fearmongering here, since GPT-5.5 has been out in the world for several weeks and hasn’t caused a cyber catastrophe yet, to my knowledge. Or on the flip side, we could accuse OpenAI of irresponsibility. Unsurprisingly, I’m tempted to do the latter. I think the evaluation results should speak for themselves. I am unabashedly pro-fearmongering when the situation warrants it. For points beyond the current frontier, I’d be completely supportive of companies restricting access to their models and naming them increasingly spooky things until we’re all reading about the shadowy internal deployment of Claude Deadhand 5. The situation is genuinely very scary, and I’m glad that we’re at least starting to act like it. Overall, I’m giving Mythos a positive score.

That brings me to the end of my (almost) year-long speedrun. In the two months since the Mythos Preview announcement, there have been a handful of developments including: the aforementioned Trump EO; both OpenAI and Anthropic publishing blog posts which acknowledge the possible need for a coordinated AI slowdown in the future (and also, ironically, filing for IPO within weeks of each other), and the public release of a heavily-safeguarded Mythos-class model (“Fable”). For the first time since I started following AI safety, I have the distinct feeling that things are coming to a head. The labs appear to be teetering on the edge of recursive self-improvement, and the US government on the edge of taking AI progress seriously. It feels like a cop-out to trawl through a year’s worth of updates only to conclude that it could go either way, but that’s where I am.

Few would argue that AI progress over the past few years has not been rapid.

Large Language Models (LLMs) have provided an unexpected path to increasingly general capabilities. In 2019, OpenAI’s GPT-2 struggled to write a coherent paragraph. In 2025, LLMs write fluent essays, outcompete human experts at graduate-level science questions, and excel at competition mathematics and coding. The most advanced multi-modal AI models now produce images and video that are hard to distinguish from reality.

These models are impressive (and useful!) but they still fall short of the north star that frontier AI companies are working towards. Artificial General Intelligence (AGI), which OpenAI describes as “a highly autonomous system that outperforms humans at most economically valuable work” has been the ultimate ambition of AI researchers for many decades.

Most experts agree that AGI is possible. They also agree that it will have transformative consequences. There is less consensus about what these consequences will be. Some believe AGI will usher in an age of radical abundance. Others believe it will likely lead to human extinction. One thing we can be sure of is that a post-AGI world would look very different to the one we live in today.

So, is AGI just around the corner? Or are there still hard problems in front of us that will take decades to crack, despite the speed of recent progress? This is a subject of live debate. Ask various groups when they think AGI will arrive and you’ll get very different answers, ranging from just a couple of years to more than two decades.

Why is this? We’ve tried to pin down some core disagreements.

The case for short timelines

Many of the people closest to frontier AI are expecting AGI to arrive before 2030.

Dario Amodei, CEO of Anthropic, is “confident” that very powerful capabilities will be achieved within 2-3 years. Sam Altman, CEO of OpenAI, has claimed that his company “knows how to build AGI”, and thinks we may reach an even grander goal of “superintelligence” within “thousands of days”. And Demis Hassabis, CEO of Google DeepMind, forecasts a slightly more conservative (but still near-term) 3-5 years until AGI.

Two high-profile scenarios, Situational Awareness and AI-2027, written by forecasters and former AGI company employees, make similar projections.

Here are some reasons to think that AGI might be just a few years away:

#1 Benchmarks keep saturating

The easiest argument for a short-timelines advocate to make is that – at least in the capabilities that we know how to measure – it really doesn’t look like we have far to go.

Here’s a chart showing how quickly AI capabilities have improved in on various benchmarks in just the last two years:

Source: International AI Safety Report

In closed-ended academic benchmarks, AIs are closing in on human-expert level. Flagship models from OpenAI, DeepMind at Anthropic all score over 82% in MMLU, which contains multiple-choice questions on a range of disciplines from mathematics to international law, and over 75% on GPQA, which contains graduate-level questions in STEM fields.

Benchmarks are saturating so quickly that researchers are scrambling to create new ones which will continue to challenge state-of-the-art models. For example, Humanity’s Last Exam (HLE) contains 2,500 questions contributed by over 1,000 subject-matter experts, designed to be at the frontier of human knowledge in most fields – and LLMs are already starting to make progress. OpenAI’s o3 scored 20% on HLE, compared to an 8% score from its predecessor, o1, which was released just months earlier. Its creators think it is “plausible” that an AI will achieve more than 50% on HLE by the end of 2025.

Importantly, AI models aren’t “just memorising” the answers to questions on benchmarks like the GPQA and HLE – researchers maintain private test sets to make sure that solutions don’t find their way into models’ training data.

If progress continues at the pace of the last few years, it won’t be long before we struggle to come up with close-ended questions that AI models can’t answer. At this point, all that will be missing are the properties needed to put all that raw intelligence to use, such as agency and long-term planning. Conveniently for short-timelines advocates, we have evidence of progress on these metrics too – which brings us on to our next point.

#2 AIs are able to complete longer and longer tasks

Impressive benchmark scores don’t translate neatly into real-world impact. One reason is that AI models cannot currently complete tasks over long time horizons. Even if they can solve any one step more reliably than most humans, they can’t autonomously carry out tasks that would take a person days or weeks.

But that could change soon. A recent study by METR, an organisation that develops and runs evaluations of frontier AI models, found that the length of tasks they can successfully complete is doubling every seven months:

If this trend continues, AIs will be able to carry out month-long projects by the end of the decade. It could even accelerate. For 2024-25 specifically, the doubling time was 4 months, which would predict AIs tackling month-long tasks by 2027.

#3 AI research might be the only capability we need to automate to achieve AGI

The goal of frontier AI companies is to develop AI systems that can automate every economically valuable task. One such task is AI research itself. If we can automate – or significantly accelerate – the process of building better AI, then any remaining hurdles to AGI could be overcome soon thereafter.

If an AI company internally develops an AI system that outcompetes its top engineers at the task of advancing the AI frontier, it would face tremendous incentive to automate a significant fraction of its own research. Automated AI researchers could work day and night without breaks, and even self-replicate to develop what Geoffrey Hinton (one of the pioneers behind the deep learning paradigm that kicked off today’s acceleration in AI capabilities) calls a “digital hive mind”.

It takes much more computing power to train a new AI model than to run one. One AI researcher estimates that we’ll be able to run millions of automated researchers in parallel by 2027, compressing years’ worth of progress into just a few days.

This could trigger what people sometimes refer to as an intelligence explosion – a recursive feedback loop where increasingly powerful AIs build their own successors. An intelligence explosion could quickly result in AI systems that are vastly more capable than humans.

AIs are already getting better at the skills needed to automate AI research. Last year, METR tested frontier models including Anthropic’s Claude Sonnet 3.5 and OpenAI’s o1-preview against over 50 human experts. The results showed that models are already outcompeting humans at AI R&D tasks over 2-hour time horizons.

#4 We could train much bigger models before 2030

Today’s most powerful AI systems are trained using three inputs – data (largely internet text), algorithms (instructions for learning from this data) and compute (the cutting-edge chips used to power the entire process).

Over the last few years, we’ve observed that the amount of compute used to train models generally correlates with how capable they are. Some have even concluded that compute is the most important ingredient in the AI development process – it’s less about clever algorithms or flashy architectures than it is just adding more and more hardware.

Epoch estimates that by 2030, it will be possible to train AI models using 10,000 times more compute than was used for OpenAI’s GPT-4. That’s around the same leap as we saw between GPT-2, which could barely produce a coherent paragraph without straying off topic, and GPT-4, which can engage in complex reasoning, generate code, pass standardised exams, and hold detailed, context-rich conversations.

If (as we’ve been arguing so far) there isn’t much further to go before we either hit AGI or automate AI research, it’s hard to imagine that another 10,000x scale-up won’t get us there.

#5 People’s timelines keep getting shorter

Expert opinion is an important datapoint in the debate over AGI timelines, but it’s a fuzzy one. For one, there’s an extremely wide spectrum of opinion – which is the whole point of this article! For two, it’s not clear what qualifies someone to forecast the arrival of AGI. Many different types of expertise could provide insight, from experience building frontier models to an impressive forecasting record.

If we can’t pinpoint a single authority whose predictions we should trust, it’s tricky to know how expert opinion should inform our forecasts. But one thing we can do is look at how predictions have shifted over time. Do they appear to be converging on a particular time period?

Taking this perspective lends some credence to the short timeline argument. In a 2023 survey of machine learning researchers, run by AI Impacts, participants thought AGI would arrive by 2047 – what would qualify by today’s standards as “long timelines”. But in the 2022 survey, this year was 2060. On the the forecasting platform Metaculus, the predicted date for AGI’s development has dropped by over two decades since 2022:

The chorus of shortening timelines is loud. Take Geoffry Hinton and Yoshua Bengio, two winners of the 2018 Turing Award for Deep Learning, and nicknamed “Godfathers of AI”. Both shortened their timelines from many decades to as little as five years after the release of ChatGPT (a watershed moment for many observers of AI progress). In a recent Substack post, former OpenAI board member Helen Toner reflects on this timeline-shrinking epidemic, and cites several examples of once-sceptics who now believe AGI could arrive within 10 years.

While there’s certainly a limit to how much we can learn from predictions (especially given the unprecedented nature of AGI), it is certainly worth noting that it’s far easier to find examples of shortening timelines than it is lengthening ones.

The case for long timelines

Despite plenty of excitement (and alarm) about near-term AGI, not everyone is convinced. Although many industry insiders are confident in short timelines, surveys from broader sets of experts still elicit longer ones, despite the downward trend mentioned above!

Skeptics point out that LLMs still make silly errors, emphasise barriers to using AIs for real-world tasks, and doubt the plausibility of an intelligence explosion.

Here are some reasons to think AGI could still be decades away:

#1 Benchmarks only tell us about capabilities that are easy to measure

Benchmarks are best for clearly-defined tasks, where success is easily verified. AI models are particularly good at these kinds of tasks. This is especially true for more recent reasoning models such as OpenAI’s o1, which rely heavily on reinforcement learning (RL).

During RL, a model learns to maximise reward through trial and error. This has produced AI systems that are superhuman in narrow domains, like DeepMind’s AlphaGo and AlphaFold– and researchers have recently found that it works better than expected on general-purpose LLMs too. It’s much easier for an AI to learn via RL when what constitutes “good” vs “bad” performance is indisputable. If we give an AI a hill to climb, it will. This is one explanation for why reasoning models have exhibited extremely impressive capability improvements in domains like maths and coding.

But AGI-skeptics point out that the real world is not so tidy. Think about the day-to-day experience of doing your own job. You might receive different (or even conflicting) feedback from various people. There’s probably more than one “right” way to deliver an output. You’ll learn what works and what doesn’t by observing the real-world impacts of your work, which could be mixed.

This line of argument formed much of the pushback to the METR study that we mentioned earlier. The study was used to demonstrate that AIs are able to act over longer and longer time horizons – but it exclusively measured performance on software engineering tasks. This is precisely the kind of task we’d expect AIs to be good at! It’s not clear how well this trend will generalise to other economically valuable labour.

Real-world jobs are also not easily divided into discrete, self-contained tasks. Instead, tasks tend to be overlapping, and dependent on lots of context that’s hard to give an AI. For this reason, it’s not so easy to delegate even short-form ones to AI models. LLMs are expert email-crafters, but asking one to follow up with a colleague on an earlier discussion is pretty hard when the AI doesn’t know what was discussed!

#2 Tasks that are easy for humans are hard for AIs – and vice-versa

People have historically expected that manual labour would be automated long before white-collar work. Yet in 2025, we have language models that can solve PhD-level science questions – but not robots that can reliably assemble furniture.

One person who did see this coming was computer scientist Hans Moravec. He observed in 1988 it’s far easier to get a computer to exhibit expert-level performance at a game like checkers than the basic perception and mobility of a one-year-old. This principle became known as Moravec’s Paradox: reasoning requires far less computation than sensorimotor skills.

Moravec hypothesised that skills like grasping objects or navigating around obstacles are harder to replicate because they’ve taken so much more time to emerge through the process of biological evolution. On the other hand, humans acquired abstract reasoning skills relatively recently. The older the skill, the more computational resources it requires to reverse-engineer.

Maybe this phenomenon has given us a warped perception of how capable today’s AI systems actually are. Being able to solve a PhD-level maths problem looks very impressive to us, because most humans can’t. On the other hand, loading a dishwasher seems like a trivially easy task, because most of us can. But the Moravec argument would say that this doesn’t actually tell us much about the absolute difficulty of either task. This implies that we’ve unlocked the easy AI capabilities, and the hardest part could still be in front of us.

#3 We don’t know if intelligence explosion is possible

So far, we’ve been arguing timelines to AGI could be long because there’s much further to go than benchmarks imply.

However, this ignores the intelligence explosion argument that we made earlier. Even if there are many hurdles we need to jump before we reach true AGI, automating AI research might mean we still get there very soon. That AIs are especially good at easily-measured tasks like maths and coding only supports this – these are precisely the skills they’ll need in order to accelerate AI progress.

But whether an intelligence explosion is actually possible is an open question. As we explained earlier, AI is trained using three inputs: data, algorithms and compute. It’s this second ingredient, algorithms, that depend on cognitive labour. This labour is performed by human researchers today, and could be performed by automated researchers in the future.

This means that the likelihood of an intelligence explosion hinges on a key question: how much algorithmic progress could a team of automated researchers make while compute and data remain static? Producing new chips and building datacentres takes time, as does gathering data (or generating synthetic data, which may become essential if we run out of human-written text altogether!).

One study found that historically, the biggest algorithmic advances have been compute-dependent – they required big scale-ups in hardware to develop and validate. If this continues to be true in the future, then an intelligence explosion looks less plausible.

Of course, this is a big if. It’s difficult to predict what lots of super-smart automated AI researchers running in parallel could achieve. They might be able to find lots of clever ways to run experiments with fixed supplies of compute, such as using scaffolding techniques to squeeze more capability out of existing models. Skeptics acknowledge that an intelligence explosion is a live possibility, but point out that the possibility is speculative.

#4 Raw intelligence might not be the main driver of discovery

In the short timeline scenario, we develop what Dario Amodei calls a “country of geniuses in a datacenter” that go on to transform the world by doing a lot of research and development (R&D). For example, these AI geniuses could be directed to develop new medicines, find sustainable energy solutions, or develop coordination mechanisms that help us avoid future conflict (or, in the bad scenario, AI-enhanced research is turned against us).

Whether things will really play out this way depends on what actually drives real-world discovery. Skeptics will point out that there’s much more to R&D than just very smart people thinking very hard. One piece of evidence for this is the phenomenon of simultaneous discovery – multiple people often have the same insight independent of each other, at roughly the same time. For example, Isaac Newton and Gottfried Leibniz both discovered calculus in the 1770s and Charles Darwin and Alfred Russel Wallace both described natural selection in 1839.

There are many theories for why simultaneous discovery happens. One is that culture moves faster than scientific discovery, creating a lag which people will then seek technical solutions or explanations for. This runs counter to the idea that the raw thinking power from a “country of geniuses in a datacenter” could actually cause a massive acceleration in R&D.

People have also pointed out that the process of R&D requires a far broader set of skills than just abstract reasoning (the bread and butter of reasoning models). Human researchers direct research teams, collect evidence from experiments, and so on. Maybe this means that scientific R&D isn’t any easier to automate than anything else. This would mean we shouldn’t expect a rapid, AI-driven transformation until after a broad range of more routine jobs across the economy have been automated.

So… who’s right?

Lots of smart and qualified people have spent a long time thinking about when AGI will arrive, and come to very different conclusions. Unfortunately, there isn’t conclusive evidence either way!

We’ll know soon enough. Many believe that AGI will either happen before 2030, or take much longer. This is because we probably can’t sustain our current rate of scaling past this point. We could build compute clusters that cost $1 trillion in five years’ time. This is probably close to the limit of what the US economy could sustain. This suggests that if we get to 2030 with no AGI, the yearly probability starts to decrease.

There are all sorts of reasons to think that the current rate of progress could fizzle out, but we can’t be confident in them. There are enough arguments for near-term AGI to warrant taking the possibility extremely seriously – and this is a scenario that the world is drastically underprepared for. We don’t know how to ensure AI systems reliably follow our instructions. We don’t understand how they work. We don’t know how we’ll avoid the worst outcomes from developing AGI, up to and including human extinction.

Since learning of the coming AI revolution, I’ve lived in two worlds. One moves at a leisurely pace, the same way it has all my life. In this world, I am safely nestled in the comfort of indefinite time. It’s ok to let the odd day slip idly by because there are always more.

The second moves exponentially faster. Its shelf-life is measured in a single-digit number of years. Its inhabitants are the Situationally Aware; the engineers and prophets of imminent AI transformation. To live in this world is to possess what Ezra Klein calls “an altered sense of time and consequence”.

I find that it’s psychologically untenable to spend all that much time in the Fast World. I can handle it for minutes to hours, but my mind invariably snaps back into its default state like I’m pulling my hand out of ice water.

Occupying the Slow World is ultimately a form of denial. I can’t call it anything other than compartmentalisation, yet I actually advocate for it. Of course, those of us trying to move the needle on AI risk should work in the Fast World, but I claim that we shouldn’t live in it. I will try to make the case for why.

I am worried that too much time spent in the Fast World will make me less invested in other people. Earnest belief in the coming singularity can make everything land with a little less weight. If the world ends or is otherwise rendered unrecognisable in two years, then the joys, triumphs, setbacks and adversities of my family and friends lose gravitas. Their implications are fewer, and their impact will be brief. This is why I try to occupy the Slow World most of the time. To wholeheartedly celebrate some achievement or milestone, or to suffer a tragedy, is to envision the future. If a friend of mine gets engaged, I want to share in their anticipation of a happily-ever-after lasting decades, because this is what it will mean to be happy for them. And if someone I love gets ill or dies, I want to feel the pain of contemplating many years spent without them, because this is what it will mean to grieve. I want to really believe in those futures. I want to delude myself as thoroughly as I can. Because this is how I will feel these moments as they deserve to be felt. I do not want my mind to qualify them with an unspoken countdown.

Those of us among the Situationally Aware must be on our guard against arrogance. To anticipate some transformation that most live in ignorance of can easily breed self-importance. But worse, it can degrade the way that we perceive the efforts of others. It can lead us to view any enterprise that won’t ultimately bear fruit in an ASI-by-2027 world with something that approaches pity or derision. Those sweet summer children with dreams of studying computer science at university, don’t they know that AIs are already competitive with human coders? Tragic that people are out here making five and ten-year career plans, don’t they know that planning over anything longer than a six-month timeframe is hopeless?

This attitude can colour the way you perceive the whole world. It can make everyone – teenagers on their way to school, business people on their rush-hour commute, joggers on a morning run – appear to you like swimmers battling hopelessly against a tsunami they cannot see, their labour fruitless, misguided and tragic. It can make you feel sorry for them. This is an attitude towards the rest of the world that I want to avoid at all costs. Partly out of humility, since of course, the Situationally Aware could be wrong, which would make this misplaced pity all the more objectionable in retrospect! But even if they’re right, this simply isn’t the way that I want to relate to my fellow human beings. I don’t want to presume that they’d act any differently if they knew what (I think) I know. I don’t want to possess a mindset that robs human endeavour of its purpose.

Sometimes in apocalypse movies, there’s a certain dramatic irony that makes humanity look like the butt of the joke. In the exposition, before it all starts going south, they’re embroiled in petty dramas or griping about traffic or the weather. And then the asteroid hits or the nuke lands or the zombie virus starts to spread, and civilisation meets its undignified end as if in chiding punishment for its obliviousness. But I think there’s plenty of dignity in being caught right in the middle of something when some unforeseen disaster occurs, of being in the midst of whatever you would always have been doing. As I was struggling to articulate this thought, I realised that CS Lewis already said it much better than I ever could in 1948:

If I might project my own slightly flimsy thesis onto the work of CS Lewis, I choose to read this as an endorsement of living in the Slow World.

Spencer Greenberg of Clearer Thinking recently wrote that believing you only have one option is dangerous. Grasping to preserve what you believe is your sole choice can lead to poor decision-making, like tolerating abuse in a relationship or staying in the wrong job. I think this argument extends to believing you have very little time. To live in the shadow of an impending singularity can make every opportunity appear as one of an ever-dwindling number. It can end in money you wish you hadn’t spent, sex you wish you hadn’t had, or nights you wish you hadn’t drunk so much. It can obscure opportunities whose benefits might take longer to manifest. Staying in the Slow World can guard against impulsivity. It can expand your menu of options, and make you more likely to take bets that will pay off in longer timelines (which we might be lucky enough to get!). It’s ok to comfort yourself with age-old adages meant to stave off myopicism – that you’re still young, that you’ve got your whole life ahead of you and that there’s always next year. It’s ok to really believe these adages and to live as if they are true.

I think my final defence of the Slow World is more a matter of personal preference. And that is simply that I prefer it. I have never lived fast, and learning that I may die young hasn’t changed that. I am not a thrill seeker. I don’t want to live like there’s no tomorrow, because living without a guaranteed tomorrow is scary and unpleasant! It inspires anxiety, not motivation. Some of the joy I experience is in novelty, but much of it isn’t. It is in the mundane and the repetitive, the things that would make the movie of my life a boring watch, but which make it no less wonderful to live. Three cups of coffee in bed on a Sunday morning, watching the same TV shows again and again and leaving a few months between rounds so that they are always imbued with fresh flavour like a new piece of gum, going to the same pub on the corner with my housemates every other weekend, frittering away the entire subsequent day together while we recover from our hangovers. I want to let the days, weeks and months slip by without counting them. I choose to do so intentionally.

All of the above is how I (try to) deal with the possibility of short timelines. I’m by no means perfect at it. Some days I spend a little more of my time in the Fast World than I’d like. It may not resonate with everyone. I’m sure there are others out there who feel the opposite. There are also ways that contemplating short timelines has shaken me out of certain malaises and sharpened my appreciation for the world around me. But on the whole, I intend to carry on as normal – because normal is more than enough for me.

Determining precisely who ought to be counted among the People Who Know Things was its own challenge. Should I restrict membership to academics whose Twitter bios were decorated with qualifications from prestigious universities and whose work had been cited thousands of times? Or insiders at frontier labs? What of those whose online presence didn’t appear to signal any relevant formal qualifications or experience, but whose long and complicated-sounding LessWrong posts were being seriously engaged with by people in categories 1 and 2? The one thing I did know is that I was not and never would be a member. The People Who Know Things occupied a world on the other side of an unbridgeable divide.

One day, the barrage of conflicting apocalypse forecasts I had been subjecting myself to for nearly half a year was taking its emotional toll. After a few glasses of wine, I took to Twitter to complain about it. I wrote a long, earnest thread whining about how I was just so confused, because Expert A said we were all going to die in 2027 and Expert B said 2035 and Expert C said never, and all their reasoning was totally opaque to me because I was Just A Girl and didn't know what a FLOP was or how scaling laws worked and I was simply not smart enough to figure any of it out.

When I woke up the next day, I unlocked my phone to find that my thread was gaining traction. The People Who Know Things were starting to talk back to me. Responses were varied, from people who related to my plight to white knights who wanted to swoop in and save me from the AI-doom cult. But one in particular stuck out:

That evening, my housemate found me frowning at my phone while polishing off the remainder of the wine.

“Are you ok?”, she asked.

“I called myself stupid on the internet and people believed me”, I muttered.

“You should spend less time on Twitter”.

Can’t and won’t understand the arguments, I tutted indignantly to myself. Can’t master the object level. Of course, my annoyance was totally unfounded, since I had said exactly that! This interaction, and several others that I’d come to have over the subsequent months, taught me a valuable life lesson. The culture I’d grown up in, where every conversation took place against a tacit backdrop of false humility, is not representative of the world at large. At my North London girls’ school, self-belittlement was invariably met with a chorus of “omg you’re so smart and pretty and totally not fat at all!!”. But in the AI safety sphere, the self-belittling are assumed to be doing a well-calibrated assessment of their own attributes1.

Was I actually being falsely humble? I think the answer to this question is somewhat complicated. The emotions I expressed in the thread – of confusion, overwhelm and uncertainty about who I should be deferring to – were genuine. But did I consider myself incapable of understanding object-level arguments for and against AI risk? Not really. I believed then and believe even more strongly now that the high-level reasons for being worried about powerful AI are extremely simple. It’s not hard to understand why racing to build increasingly capable and goal-directed AIs – without any scientific solution to the problem of controlling them once they supersede us – may lead to bad things. It’s also not difficult to observe that the unworried are few and far between, and that they lack anything like a satisfying answer to this basic concern.

Of course, there are many more in-the-weeds questions that one could ask in order to form a well-rounded AI worldview (will takeoff be fast or slow, what is the likelihood that X or Y alignment technique will work, what will the political response to advancing capabilities be), and it is true that in 2023 I’d have had very little to say on any of these. But it is also true that I hadn’t really tried. My “research” into AI safety up until that point could be more accurately described as an obsessive quest for reassurance, which ended up culminating in an extensive collection of p(doom) estimates and timeline predictions. These had been cherry-picked from a much richer discourse about why people had reached those conclusions, which I had only superficially engaged with. I just wanted to know if the world was going to end. I was more guilty of being lazy than stupid. Over a year later, I’m still far from confident in any of my AI takes, but I have at least tried to do the intellectual legwork of refining them. I now consider myself an Aspiring Person Who Knows Things.

I don’t know whether there are many other AI safety lurkers out there who don’t feel qualified to join the conversation, let alone whether any of them are reading this. But on the off chance that they are (especially if any are female, since being one of the few women in AI safety can create the perfect storm of self-doubt), I wanted to offer some unsolicited advice.

#1 Don’t sell yourself short

This is of course generic advice that applies to the world at large – if you repeatedly downplay your own abilities, people will not always give you the benefit of assuming you’re just insecure or self-deprecating. They might take you at your word!

This is somewhat complicated by the fact I think my online AI ramblings have filled an underrepresented niche – that is “normal person who doesn’t understand AI but is stressed about it”. My femaleness complements this quite nicely. It has been easy to lean into the Just A Girl aesthetic. There’s endless mileage you can get out of takes that boil down to “wow guys it sure looks like the experts are spooked by AI and think it might literally kill everyone, maybe governments should be taking that more seriously but hey what do I know haha”. There have been benefits to doing this, but I think that to a degree, it has resulted in me letting myself off the hook. I believe it has meaningfully delayed my transition to an Aspiring Person Who Knows Things. If I could go back in time, I’d probably skip my Just A Girl arc.

#2 You can just say things

If you’re an AI safety lurker and don’t want to be – just say things. It really is that simple. Despite interacting with 100s of people who are smarter than me on topics I would once have considered beyond my intellectual bandwidth, I have suffered very few embarrassments. Even if you say something wrong, you’ll likely be politely corrected, not socially punished. You’ll probably find that The People Who Know Things often agree with you.

This isn’t to say that everyone’s contributions to the conversation have equal weight, or that AI safety is some unique field where we shouldn’t privilege expertise. I don’t want to sound like one of those people who thinks a layperson with acess to PubMed is as well-qualified to perform medical diagnosis as a doctor. But AI safety is a domain in which one can have knowledge or expertise along many different axes. For example, technical experts may be best placed to forecast the speed at which AI capabilities will improve, but not how governments ought to respond to them. The field is also nascent enough that, for better or worse, it hasn’t had time to become subject to the institutional gatekeeping that befalls many others. There are conversations at the frontier of AI safety and policy literally playing out on Twitter and Substack. There’s nothing to stop you joining them.

As was the source of my frustration in that original tweet thread, there is no expert consensus on what a future with powerful AI will look like. It seems like a safe bet that the future will be very weird, but beyond that, I see little grounds for confidence. I think this is the perfect context in which to Just Say Things. I like the way Nathan Labenz put it in a recent episode of The Cognitive Revolution podcast:

AI gods might be an emerging trend over the second half of the decade. I have no idea how we're gonna relate to these things. If they are meaningfully superhuman, will we even try to keep them under control? Will we worship them? No matter how weird your alignment idea is, I think it is worth pursuing. No matter how weird your thoughts are about where the future might be going, I would say they're probably worth entertaining.

#3 Ask questions

In trying to follow developments in AI, I find myself frequently confused. Anyone who isn’t delusional will too. Luckily, there is a whole community of well-intentioned and knowledgeable people who are happy to weigh in! I have on innumerable occasions contacted people I know (or don’t know!) with technical or policy expertise to ask them clarifying questions. More often than not, they have responded. I also often pose my open questions to Twitter. What’s the deal with o1 sometimes doing better on PhD-level science questions than high school level ones? Why do some people believe that an ASI in the hands of governments would pose an unacceptable risk of power concentration, but one in the hands of a private company wouldn’t? I have received many helpful answers.

#4 Actually try

As I said above, 2023 Sarah, who pleaded ignorance of all things AI in that long rambly thread, hadn’t actually tried very hard. In 2025, I still haven’t read or understood nearly as much as I would like, but I am sure as hell trying. The downsides of ever-accelerating AI capabilities are many (plausible short-term human extinction, job loss, a looming crisis of meaning…) but at least one major upside is that it has never been easier to learn things! A hill I will die on is that using LLMs to translate smart-person speak into dumb-person speak is, in fact, a smart-person move. This is an excellent use case for making sense of jargony papers or impenetrable posts on LessWrong. You can then have as long a conversation as you want with a private, infinitely patient AI tutor about anything you still don’t understand. Still, there will be things that remain the exclusive purview of CS PhDs or seasoned policy wonks. There is a level of expertise that you or I likely don’t have time to build up pre-singularity. But that doesn’t mean we can’t make a start.

In the early days of my confused-girlie-come-AI-worrier journey, multiple people reached out to me to ask if I’d considered working in AI safety. I dismissed them out of hand. Of course I hadn’t! That was the territory of People Who Know Things, of which I obviously was not one. My plan was simply to post through it.

I now do, in fact, work full-time in AI safety. I still consider myself to be at the bottom of a very steep learning curve, but I’m not Just A Girl anymore.

Subscribe now

The idea that the current concern around AI is nothing more than another instantiation of the irrational panic that accompanies the eve of every technological revolution is widespread. Anyone reading this is likely familiar with the case for existential risk from AI, which rests on conceptual arguments that are being increasingly validated by empirical evidence, and is a source of concern among some of the world’s most highly-cited experts1. As I write this, I haven’t yet checked to make sure that no group of scientists made a similarly rigorous case for extinction-by-trains, but I’m pretty confident that I won’t find one. Anyone who has spent more than five minutes investigating the topic of AI x-risk knows that this comparison is ridiculous, and yet arguments like Guo’s are the start and end of countless discussions about it – and are often met with very little pushback.

This broad claim about historical fear of new technologies is one of two “crying wolf” accusations often levied at AI safetyists. The other is more specific – that the AI-concerned, or advocates of strict regulation, have spread alarmism about the catastrophic potential of existing models that have since been proven safe, or claimed that AI doomsday would come to pass by some specific past date. In the same DealBook roundtable, panel moderator and New York Times columnist Kevin Roose states this explicitly: “There’s a crying wolf problem among AI safety advocates, where people are starting to discount these predictions because systems keep being released and keep not ending the world”.

To put my cards on the table, I think that both varieties of the crying wolf argument are fairly weak. I also think that the phenomenon Roose describes isn’t really happening to anything like the extent he implies. Nonetheless, arguments of this flavour are so common that I thought it might be worth taking some time to really give them their due. If there is anything at all to the “crying wolf” claims, I intend to unearth it here. And insofar as these claims have even the smallest bit of validity, I want to think about ways that the AI safety community can refine our messaging to make us less vulnerable to them.

Part I: Everything from trains to electricity

In this section, I will look at some past technologies, and make a good-faith effort to assess the level of historical concern about their apocalyptic potential, as well as the credibility of this concern.

Trains

I imagine that Sarah Guo did not expect anyone to scrutinise her claim that there was an apocalypse panic over trains too closely. I’m aware that this was an offhand comment and that I’m probably being unfair in taking her to task on it – but hey, everyone is entitled to a little pedantry now and again, as a treat.

Did anyone at the advent of the railway put forth a plausible mechanism by which trains might literally end the world? Well… no. Obviously not. I’m sure that Guo knows that too, and that she was simply committing the common sin of using phrases like “the end of the world” where they do not properly apply (relatedly, overuse of the word “existential” is one of my pet peeves). That said, there have been since-invalidated concerns about trains that were taken seriously by credible people. One fear was that railway travel could drive people insane. The jarring motion and unprecedented speed of trains were thought to injure the brain and cause bouts of lunacy. Doctors speculated in medical journals that there might be an invisible epidemic of “railway madness”, with many cases lying latent until sufferers experienced violent outbursts mid-commute and attacked innocent passengers. The media spread its fair share of alarmism about the danger of railway madmen, and Victorian by-laws even went so far as to stipulate that “insane persons” be consigned to a designated carriage.

In fairness to Guo, the railway madness saga has all the trappings of a class tech-panic – expert alarm, media fear-mongering, clumsy attempts at regulation, and a semi-plausible sounding scientific hypothesis that appears ludicrous with the benefit of hindsight. There is clearly an analogy that skeptics can draw to the AI case. But again, to state the mind-numbingly obvious, “there were unwarranted concerns about the localised damage this technology might cause” is a very different claim to “people thought this technology might literally extinguish the human race”.

Electricity

Guys, I really held out hope for this one. It seemed pretty plausible to me that the discovery and widespread deployment of electricity would have sparked (pun intended) end-of-the-world fears. But after googling just about every keyword related to electricity-fueled millenarism I could think of, and a lengthy back-and-forth with Claude, I could find no evidence that this was the case2. As far as I can tell, the story here is pretty similar to the railway case – there was widespread apprehension about electricity, but this apprehension was not of an imminent apocalypse.

In my opinion, the kinds of worries people had at the dawn of the electrical age were not totally unreasonable. For example, a series of fatal accidents caused by New York’s overhead electric wiring system led to widespread public outrage, accusations towards lighting companies of putting profit over safety, and demands that the wires be buried underground instead. An 1889 magazine cover conveys this sentiment pretty well:

I sympathise with these 19th century New Yorkers! After all, a bunch of them had just the misfortune of witnessing a telegraph lineman, John Feeks, die instantly after being electrocuted by what was supposed to be a low-voltage telegraph line, and fall into the tangle of wire below before smouldering for the better part of an hour. I’d have been pretty freaked out too! In fact, I’d argue that some stories of panic over technology can be re-told as failures by companies and technologists to prioritise safety, leading to high-profile malfunctions and (understandable) public backlash. Maybe it’s really AI companies who should be heeding this particular historical warning. But I digress.

Perhaps the most famous example of an electricity-based tech panic was the 1880s-1890s War of the Currents, a head-to-head between two pioneers: Thomas Edison, who championed direct current, and George Westinghouse, who promoted alternating current. Each had invested significantly in their own favoured electrical system, and was keen for it to become the standard method of distributing electricity in the United States. To this end, Edison embarked on an elaborate propaganda campaign to provoke fear of his competition. He arranged public electrocutions of animals with alternating current, including dogs, horses and even an elephant, and advocated for its use to power the electric chair. Although alternating current eventually won out as the dominant mode of electrical distribution, in a way, Edison’s campaign worked. It did stoke fear and distrust of alternating current which delayed universal domestic adoption of it by over fifty years. Some private US households were still powered by direct current, which has since proven more dangerous, in the 1950s.

So, is the electricity example a good one for AI x-risk dismissers seeking examples of analogous tech panics? Sort of – it does seem that Edison was sincere in his belief that alternating current was dangerous, which is why, against the advice of his own colleagues, he did not invest in it himself. One could portray this as an example of a credible expert whose misguided fears of the technology he had spent many years studying was responsible for slowing progress and delaying mass adoption, an accusation often levied at so-called “AI doomers”. But in other ways, the analogy breaks down. In trying to stoke fear of a competitor’s technology while insisting on the safety of his own, Edison was, predictably, following strong financial incentives. I hardly need to point out that this is precisely the opposite of what safety-concerned AI labs are often accused of – that is, creating hype around the apocalyptic potential of their technology in a cynical ploy to attract investment, as if claiming that your products might kill everyone on Earth is some kind of tried-and-tested marketing technique. I think this claim is pretty baseless (here’s a good write up of why), and it’s interesting to note that at least in this one case, historical precedent seems to run the other way.

The Large Hadron Collider

This is probably my favourite apocalyptic tech-panic, because it’s a perfect example of non-experts fueling scary theories that the scientific community has been at pains to correct. In 2008, a few months before the Large Hadron Collider was set to come online, a Hawaiian man brought a lawsuit against its completion. He alleged that scientists had overlooked evidence that the LHC would create a massive black hole which would swallow the Earth, citing a minor in physics from Berkeley and a career in nuclear medicine as evidence of his credibility. He was just one of many conspiracy theorists worried that the collider would cause earthquakes, shift the world into an alternate timeline, or open a portal to hell. But CERN was quick to issue a statement assuring the public that they were the Adults In The Room and that the LHC presented no danger, that the science confirming its safety was extremely watertight, and that a who’s who of prestigious experts in astrophysics, cosmology, general relativity, mathematics and particle physics had all agreed that there was nothing to worry about.

This sort of reassuring expert consensus is precisely what I spent so long searching for when I first became worried about AI risk. I thought: surely it can’t be the case that a decades-long technological venture, benefitting from money and talent on an incredible scale, is actually on track to bring about the end of the world – because, presumably, someone would have checked. And without a high degree of confidence that it won’t destroy everything, humanity surely wouldn’t be participating in such a project. I tried really, really hard to find that elusive consensus. But this was an exercise in futility, because as anyone reading this likely knows, it simply doesn’t exist.

The Y2K panic

Ah, Y2K. The go-to example in the rhetorical arsenal of anyone looking to dismiss fears of disaster with a “we’ve been here before”, an “I’m old enough to remember when…” or a “oh honey, this must be your first technological doomsday”.

Is all this fuss over AI doom just the next Y2K? ask Bloomberg, the Guardian and a bunch of people on Reddit. It’s a tempting comparison. The conventional tale of Y2K is that programmers anticipated a bunch of computer failures at the dawn of the new millennium because dates had hitherto been expressed using just two digits for the year, rather than four. They feared that planes would fall out of the sky, nukes would launch themselves, and critical infrastructure – banks, hospitals and power grids – would crumble and leave civilization in disarray. A golden era bloomed for professional survivalists and religious evangelists turning public fear into profit. Governments dropped billions on technical fixes for the so-called “Y2K bug”, and printed an excess of paper money in case of a run on the banks. And then, when the 1st January 2000 came and went without incident, everyone was left feeling just a little bit silly. So maybe when superintelligence arrives and it goes totally fine, we’ll think dedicating a whopping 1-3% of AI research to safety was a big embarrassing overreaction.

I think this analogy is bad for a couple of reasons. Both are quite obvious, but I may as well explain them while we’re here. First, as many have pointed out, the turn of the millennium did not bring disaster precisely because such a huge amount of behind-the-scenes work went into averting one. By the time that the general public and mainstream media caught on, programmers had been quietly working away at the problem for over a decade. This was an effort for which they received very little fanfare – the seamless transition into a new millennium (which they had enabled) saw a very real technical problem (which they had fixed) dismissed as an episode of paranoid doomsaying. Also, not to be corny or anything, but I think the global Y2K mitigation efforts were a feat of international coordination nothing short of inspirational. The UN’s International Y2K Cooperation Center tracked Y2K readiness worldwide and defined and shared best practices on tackling risk. Russia and the US cooperated to prevent glitches in either’s early warning systems leading to an accidental nuclear launch. We love to see it.

Second, by the time the millennium rolled around, the governments of the world were actually quite confident that no disaster would occur – and had been for some time (though the public was slow to catch on). This is because the Y2K bug was a problem we knew how to fix. Essentially, we needed to update legacy software to store dates with four-digit years instead of two. We had a plan. When it comes to the problem of ensuring that superintelligent AIs behave as intended, we don’t (it’s not that no one has proposed plans of course, but that we have nothing close to scientific consensus that any of them will work).

Nuclear power

A common argument by opponents of AI regulation goes something like: “hey, look at the regulatory overreaction to nuclear power, and how it stifled innovation, degraded public trust and set us back decades in the fight against climate change”. There’s certainly something to this (though I don’t really feel like weighing in on the debate over whether extreme caution on nuclear power was prospectively reasonable, even if it had the ultimate effect of hampering progress).

But how analogous are AI fears to worries over nuclear energy? To get this out of the way first – I could not find evidence of credible concern that civilian applications of nuclear power would end the entire world. As far as I can tell, the general story of nuclear energy is that the public wariness of it has been disproportionate to the actual risk it bears, and that scientists have largely tried to correct these misconceptions (not that it doesn’t have serious downsides, or hasn’t caused several localised disasters). Again, this is not the case with AI, which many experts do believe presents a serious danger of human extinction. So whether we “overreacted” to the threat posed by nuclear energy seems irrelevant to how we should address the much greater threat posed by AI.

Of course, nuclear weapons, which almost destroyed the world3 several times and still could, are a different story. It will come as no surprise that I think our success in not detonating another nuclear weapon for almost 80 years after one was first used in warfare, against the expectations of many at the time, is a testament to the importance (and the feasibility!) of global coordination around existential threats. Luckily, I don’t see many people making the argument that our avoidance of nuclear war thus far is evidence that we should be less concerned about AI risks. So that’s something.

To round out this first section, I don’t think any of the historical tech panics I’ve discussed are even vaguely analogous to present-day concerns about AI. In all of them, it appears to be the case that either a) some lay people worried that the world might end, but experts were largely unconcerned or b) pretty much no one was concerned that the world might end, but people did have sub-existential worries that turned out to be unwarranted. This is pretty much what I expected to find, and I’m a bit worried that I’ve just spent over 2,500 words stating the obvious – but given that people are still going around carelessly tossing out sentiments like “everyone always thinks every new technology will be the end of the world”, it seemed worth a few hours of my time. Obviously, I haven’t been exhaustive here. There are many other transformative technologies I haven’t covered, and inevitably important things I’ve missed about the ones I did. But I’m fairly confident that the broad point is sound.

Part II: False (AI)larms

As I stated in the intro, there are two varieties of this “crying wolf” argument – one that encapsulates the history of technology writ large, and another that accuses AI safety advocates of spreading false alarm about existing AI models, none of which have proved catastrophic.

Here are just a few examples of this claim in the wild:

I have a tricky task here, because proving a negative – that very few people have actually made concrete, since-falsified predictions about AI-caused catastrophes – is obviously impossible without doing a comprehensive audit of the entire AI safety discourse. As it happens, I probably have done something pretty close to such an audit, because I am just really obsessed with this topic, but I have no way to prove that.

So let’s start with what I can prove. For one, none of the accusations above actually cite such predictions (sources are in the caption so that you can verify this yourself). If the authors had actually seen any, you’d think that they’d link them, rather than passing up the opportunity to land what would be a pretty big rhetorical win. One type of evidence they do offer is specific calls for regulation, which are often portrayed as synonymous with predictions that, absent intervention, AI would prove catastrophic at certain levels of capability. Take the opening of the Forbes article:

There are no specific timeline predictions in the FLI 6-month pause letter, the CAIS statement or Eliezer Yudkowsky’s TIME article. None claim that we were at the time of their publication in “immediate danger” from AI. The second doesn’t even propose any particular intervention – though of course, one can reasonably disagree with the policies put forward in the other two.

This reply to the Daniel Jeffries tweet is another good example:

I’m not making any claims about whether the thresholds above are sensible, or whether it was wise for them to be suggested when they were. I do think it seems clear with hindsight that some of them are unworkably low. But again, advocating that AI development be regulated at a certain level is not the same as predicting with certainty that it would be catastrophic not to. I often feel that taking action to mitigate low probabilities of very severe harm, otherwise known as “erring on the side of caution” somehow becomes a foreign concept in discussions of AI risk.

An easily debunkable claim in the above compilation is that Geoffry Hinton is guilty of crying wolf. In 2023, Hinton was predicting that smarter-than-human AIs would emerge within 5 to 20 years. So, a credible crying-wolf accusation will not be leverageable against him until 2043. It seems our collective attention span is becoming so short that a person warning “for months” about a catastrophic event predicted to occur at least several years into the future can be accused of alarmism. It’s not looking good for The Discourse.

Taking to the tweets

Last month, having run into the crying-wolf meme countless times without having been signposted to so much as (1) specific and since-debunked prediction, I decided to take to Twitter in search of some. If anyone was sitting on a smoking gun, I wanted to hear about it. This type of open call is about as wide as I can cast my net, but obviously my methodology here is imperfect – I am disproportionately followed by people who are in my corner of the AI debate, the tweet was seen by less than 10,000 people, and despite my best efforts, I’m just not that popular. Even so, if the Debunked Prediction Graveyard is as large as the accelerationist crowd makes it sound, I’d expect at least a few to make their way into my replies4.

I didn’t receive anything that would meet my (admittedly high) bar of a specific and since-debunked prediction (I’ve said something to this effect so many times now that I’m thinking it needs an acronym or something. An SASDP?). That said, there were a few not-totally-unfair criticisms of the AI safety movement – and I said at the beginning of the piece that I wanted to give the crying wolf argument its due. So let’s try and assess some of these accusations.

One that caught my eye was a claim that The Future Society, a fairly well-respected non-profit with a mission of “[aligning] artificial intelligence through better governance”, define models trained with between 10^23 and 10^26 FLOP as having the potential to pose an existential risk if seriously misaligned. In a 2023 report, TFS categorises such models as “Type II General Purpose AIs”. It recommends a bunch of requirements for Type II GPAI developers including third party audits, some fairly stringent-sounding infosec and cybersecurity protocols, and the commitment to pause development of, or even un-deploy, models that cannot be proven “absolutely trustworthy”. I still wouldn’t classify this as a falsified prediction, since it isn’t a prediction at all, just a proposal of what many might consider heavy-handed regulation at relatively low compute thresholds (I will go to my grave swearing that these are importantly different). But I do disagree with TFS’s decision to place the lower bound of their Type II classification as low as 10^23 FLOP, especially since models with a higher FLOP count than this had already been trained, deployed and likely even open-sourced5 before the report was published. So I would consider this a mistake on the part of TFS – and a contributor to the crying-wolf effect – if not strictly a debunked prediction.

Onto another accusation – that civil resistance group StopAI previously claimed we risked a fast-takeoff-extinction-event from all models trained on more than 10^23 FLOP and demanded their deletion, but have since removed this from their website. I obviously can’t verify this, though it wouldn’t surprise me. I think this is the most straightforward crying wolf example (StopAI have been guilty of this a few times recently). All I’d ask here is that the anti-safety faction grant us the good grace not to judge an entire movement by its fringe.

Eliezer Yudkowsky is the target of many crying wolf accusations, on my particular tweet thread and elsewhere. The position he has staked out – that AI could recursively self-improve into a superintelligent system that quickly disempowers humanity at any point – obviously makes him very vulnerable to them. Yudkowsky has repeatedly refused to make timeline predictions, which is either to his credit or a ginormous cop-out depending on your perspective. This is a catch-22 that I’m not sure how to address. On the one hand, people have every right to be suspicious of what appear to be unfalsifiable claims of disaster that can be eternally deferred into the future. On the other, recursive self-improvement is a live possibility that is taken seriously by a large number of machine learning researchers6. It’s entirely possible that we live in an unlucky timeline where Cassandra-like prophecies of doom are indistinguishable from reality until it is too late.

The final, and most easily countered, set of accusations revolve around GPT-2 being considered “too dangerous” to release. As several people in my replies pointed out, GPT-2’s release was delayed because of concerns about misinformation and other forms of low-level misuse. OpenAI postponed its deployment while they studied the issue, which seems eminently sensible to me.

In the end, I don’t think my little Twitter experiment revealed evidence that the AI discourse is riddled with failed predictions from the safety side. I think it did reveal a few strategic errors that contribute to what I’ll call a “crying wolf effect” – such as calls for stringent regulation at unrealistically low thresholds. In the last section, I’ll say a bit more about how I think we should make ourselves less vulnerable to this effect.

I think the AI safety-concerned have a very tough communicative challenge in front of us, for several reasons. First, humanity does have a history of collective neurosis over transformative technologies, which understandably provokes skepticism of what may look on the surface like just another tech-panic. AI safety advocates have tried hard to brand themselves as techno-optimists about every technology except for powerful AI to dodge accusations of luddism, but this isn't an easy pitch. Second, AI development is extremely unpredictable. This has led in the past to what seem in retrospect like overly-draconian policy recommendations, and inevitably will do again in the future. And third, the AI safety movement is a broad coalition. We can’t expect to maintain a perfect predictive track-record between us. This problem is exacerbated by the fact that there are many reasons people have for being concerned about bad outcomes from AI, not all of which will turn out to have been the right reasons. So each of us “shares a side” with people whose reasoning may be very different to our own, and whose predictions we may not endorse (this tweet from Amanda Askell sums it up pretty well).

As it stands, I don’t think the AI safety community writ large has cried wolf. But if transformative AI capable of disempowering humanity does not emerge within the next ten years (at the most) I think it will be fair to say that we have7. I happen to think superintelligence within ten years is more likely than not, but I of course could be wrong! I’m worried that we’re entering a period in which the AI safety community risks losing a bunch of credibility.

So what should we do about this? Personally, I think anyone making timeline predictions should not just acknowledge but emphasise their uncertainty. People can reasonably disagree with me here; there are certainly benefits to signalling high levels of confidence – I just happen to think that the risk of lost credibility outweighs them.

As I mentioned earlier, one big contributor to the crying-wolf effect is overly-stringent policy recommendations. Given that we have to make decisions under uncertainty, I think we also have to accept the inevitability of proposing policies that ultimately prove unnecessary. But in my opinion, those doing so should caveat that they are designed to mitigate the possibility (and not certainty) of catastrophic outcomes. This should be obvious, but given that people will be waiting in the wings to weaponise anything that could be called a regulatory overreaction, I think it’s worth doing.

One final suggestion (which I’m less confident about) is that we spend less energy trying to forecast when transformative AI will arrive, and more energy making plans for short timelines. If we believe it’s plausible that takeover-capable AI could arrive by 2027, we can simply spend our time advocating for policy that would be effective in that scenario, while acknowledging the possibility that it takes much longer. This isn’t to say that no one should be focused on forecasting timelines, of course. The ideal gameplan for ASI in 2040 probably looks pretty different to the one for ASI in 2027 – but I doubt there’s much difference between the gameplans for 2027 vs 2028. The less numerous and specific our forecasts, the less vulnerable we are to crying wolf accusations.

Humanity might have suffered misplaced anxiety about technology in the past, but as anyone reading this likely agrees – this time really is different. Let’s get it right!

Subscribe now

Occasionally, when I’m struggling with how to articulate a half-formed thought, I type my muddled ramblings into Claude and ask,

“What am I trying to say here?”. And then, to my dismay, it tells me.

I watch that little orange cursor blink dutifully across the screen, rearranging the contents of my own mind into eerily perfect prose.

“Where would you like to take this next?”, asks Claude, perfectly compliant and diligently helpful as always. I feel the same exasperation that hits me when my cat delivers a dead mouse at my feet in a misguided act of generosity, staring up at me with her huge, earnest eyes.

“Yes, that’s exactly what I meant”, I murmur. “Also, fuck you”.

My turbulent relationship with chatbots brings out the full force of my irrationality. I am not ashamed to admit that it triggers me. It will have me alone in my room yelling "I CAN DO IT MYSELF" at my laptop, in a (wo)man-meets-machine display of pointless fury to rival Basil Fawlty thrashing his car with a tree branch. At a recent EA Global conference, I had a one-on-one meeting with an undergraduate who lamented the fact that most of his classmates are using ChatGPT to complete their assignments. Suddenly, I was 26-going-on-60, denouncing the myopic laziness of ‘kids these days’ with all the Luddism of a parent decrying mobile phones at the dinner table.

“That might help them get ahead now, but it won’t serve them in the future”, I heard myself say in an uncharacteristically shrill voice. “They should be considering their post-graduation prospects” (as I was saying this, I was acutely aware that it literally made no sense given my own single-digit-year AGI timelines).

I have many gripes with AI. My biggest one is the alarmingly high likelihood that it will kill everyone. My second biggest one is the ease with which it does in seconds what I do in hours. I have always taken great pleasure in crafting a well-formed sentence. I love arriving at the perfect articulation of an idea after rounds of near-headache-inducing mental labour. I don’t imagine that Claude gets headaches1.

One evening, I decide to experiment with AI text detection. I spend an hour or so pasting chunks of my own writing into free online tools, followed by AI-generated content on the same topics. To my absolute delight, I discover that, at least initially, it seems like they actually work. The detectors are distinguishing between human and AI text with impressive accuracy, and I am high on my own supply of hopium. Ah yes, the sacrosanct uniqueness of the human voice, irreproducible by the shallow mimicry of simple pattern-detecting machines. The skeptics were right! There is a secret sauce, and ZeroGPT has found it.

But it doesn’t take long for skepticism to rear its ugly head. It does seem unlikely that there would exist some magical tool capable of differentiating between the products of biological and silicon minds. And I already know that AI can generate content far more sophisticated than I’ve been testing the detectors with using just a little more prompting (reluctant though I am to acknowledge this).

There is a Substacker whose work I enjoy. She writes lovely, touching vignettes about small-yet-significant personal experiences like lending a dog-eared copy of a favourite paperback to her mum. It’s precisely the kind of thing that your resident skeptic friend who hasn’t so much as tried an LLM will swear up and down that AI could never produce. I give Claude the prompt:

“Write a touching story about lending a dog-eared copy of a favourite paperback to your mum, in the style of a personal essay on Substack”.

Its first try, The Book That Came Home, is a laudable attempt at the kind of emotional authenticity that I’m after – my detector-of-choice, Quillbot, estimates that it is 51% AI-generated. I take it up a notch:

This second essay is actually quite poignant. Claude has decided to pack an extra punch by giving the mum cancer. It’s a cheap move, but I’m in an emotionally vulnerable place, so it works on me. It apparently also works on Quillbot, which confidently declares that Mom’s Sticky Notes is 0% AI-generated. The rage is back. I feel the urge to write a strongly-worded email informing Quillbot of this false negative, and reminding them of their duty as the final line of defence against the unstoppable encroachment of AI into every corner of the internet, and as one of the few remaining bricks in a wall around the sacred territory of human creativity. But of course, there is no wall (in more ways than one), and the whole enterprise of building one, as I well knew even before my foray into the world of AI text detection, is patently ridiculous.

Two sources of cope

I’ve always loved to write. That AI can do it so effortlessly upsets me enough to inspire this petty blog post. But there are reasons I think this need not be as nihilism-inducing as it sometimes feels.

First, for the time being, there remains at least some skill in prompting chatbots to produce the kinds of content you want. Extremely low-effort prompts result in writing replete with tell-tale signs that AI detectors can spot a mile off. But very good ones can produce writing idiosyncratic enough that, just maybe, there is something of the human user that survives into the final product. For example, Amanda Askell of Anthropic got Claude to generate a whimsical fable about a man locating the deed to some valuable land using a precise arrangement of garden gnomes, making me wonder whether the creative locus of AI text can sometimes lie with the prompter, rather than the promptee2 (though of course, Amanda is one of the staff members behind the alignment of Claude, meaning she gets to take a little more credit for its outputs than the rest of us).

Second, and I think more importantly, each of us has always shared the planet with minds better than our own at all sorts of things – all that has really changed is that some of those minds are now artificial. When I was younger, I was part of a children’s choir. One Sunday afternoon rehearsal when we were both around 13, a friend of mine dramatically announced her intention to give up singing because she was upset that some other choir members were better at it than her. She couldn’t see the point in any pursuit that she wasn’t the best at. I pointed out that only one person on Earth could claim the title of Best Singer, and that it would be a great loss to the world if everyone else were to stop singing. If 13-year-old me could grasp this simple lesson, 26-year-old me would do well to heed it now.

However capable AI becomes, I think it will continue to matter that my words are my own – and I will keep writing them.

Subscribe now

Three industry leaders have released safety frameworks outlining how they intend to avoid catastrophic outcomes. They are OpenAI’s Preparedness Framework, Anthropic’s Responsible Scaling Policy and Google DeepMind’s Frontier Safety Framework.

Despite having been an avid follower of AI safety issues for almost two years now, and having heard plenty about these safety frameworks and how promising (or disappointing) others believe them to be, I had never actually read them in full. I decided to do that – and to create a simple summary that might be useful for others.

I tried to write this assuming no prior knowledge. It is aimed at a reader who has heard that AI companies are doing something dangerous, and would like to know how they plan to address that. In the first section, I give a high-level summary of what each framework actually says. In the second, I offer some of my own opinions.

Note I haven’t covered every aspect of the three frameworks here. I’ve focused on risk thresholds, capability evaluations and mitigations. There are some other sections, which mainly cover each lab’s governance and transparency policies. I also want to throw in the obvious disclaimer that I have not been comprehensive here and have probably missed some nuances despite my best efforts to capture all the important bits!

What are they?

First, let’s take a look at how each lab defines their safety framework, and what they claim it will achieve.

Antrophic’s Responsible Scaling Policy is the most concretely defined of the three:

“a public commitment not to train or deploy models capable of causing catastrophic harm unless we have implemented safety and security measures that will keep risks below acceptable levels.”

OpenAI’s Preparedness Framework calls itself:

“a living document describing OpenAI’s processes to track, evaluate, forecast, and protect against catastrophic risks posed by increasingly powerful models.”

Finally, Google Deepmind’s Frontier Safety Framework is:

“a set of protocols for proactively identifying future AI capabilities that could cause severe harm and putting in place mechanisms to detect and mitigate them”.

Anthropic’s RSP is notable in being defined from the get-go as a ‘commitment’ to halt development if extreme risks cannot be confidently mitigated. That said, all three frameworks do go on to specify conditions under which they would pause training and/ or deployment (more on that later).

Thresholds & triggers

Each framework follows a similar structure in terms of the mechanisms it constructs for triggering further action.

Broadly speaking, there are two key elements to these mechanisms:

1. Capability thresholds measure the extent to which models can do scary things. Dangerous capabilities are tracked across different categories, such as cybersecurity or model autonomy.

2. Based on the capabilities they exhibit, models are assigned a safety category. Which safety category a model is in dictates which safeguards need to be applied.

Capability thresholds

There are three sets of capabilities that are tracked in each of the frameworks:

• Chemical, Biological, Radiological, and Nuclear (CBRN): Could the model make it significantly easier for bad actors to design weapons that cause mass harm?

• Autonomy: Could the model replicate and survive in the wild? Could it meaningfully accelerate the pace of AI R&D? Could it autonomously acquire resources in the real world in order to achieve its goals?

• Cyber: Could the model enhance or even fully automate the process of carrying out a sophisticated cyberattack?

However, there are some important differences between the tracked risks in each framework:

• Cyber capabilities do not quite rise to the level of a tracked risk category in Anthropic’s RSP. Anthropic takes a ‘watch and wait’ approach, where the cyber capabilities are tracked, but they do not pre-specify a point at which they would trigger a model to move up a safety category.

• OpenAI’s Preparedness Framework contains a unique risk category, persuasion. This category tracks a model’s ability to generate content which could change a person’s beliefs.

• Deepmind’s Frontier Safety Framework distinguishes between a model’s ability to accelerate AI R&D and its ability to act autonomously more generally.

In short, all three frameworks aim to track three things:

1. Whether models could help people do very bad things in the CBRN or cyber domains

2. Whether they could ultimately do very bad things on their own

3. And whether they could accelerate AI R&D such that we’ll be confronted with (1) and (2) much sooner than we would have been otherwise.

Risk categories

I’ve consolidated all of the risk categories in each of the three frameworks into one table. I’ve also indicated which category each frontier lab’s most recent model is, according to their own evaluations:

You’ll notice that in order to be moved up a safety category, OpenAI models only have to possess one of several capabilities designed to trigger that threshold. This is because while a model is scored in each individual risk, its overall score is the highest in any. For example, if a model scored ‘low’ in cybersecurity, CBRN, and autonomy, but ‘medium’ in persuasion, its overall score would be ‘medium’ – and it would trigger the safety protocols associated with that risk category.

Evaluations

So far we’ve covered the risks that each lab has committed to track, and the different risk categories that models can be placed into. In order to actually track these risks, labs need to run evaluations or ‘evals’ on their models.

Evals aim to detect dangerous capabilities. All three policies state an intention to perform evals that actually elicit the full capabilities of the model. They aim to discern what the model could do in a worst-case scenario – if people manage to improve a model’s capabilities after it has been deployed through fine-tuning or sophisticated prompt engineering, for example:

• OpenAI commits to running both pre-mitigation and post-mitigation evals on their models. Pre-mitigation evals detect what capabilities a model has before any safety features are applied. They also say they will run evals on versions of models that have been fine-tuned for some specific bad purpose (for example, a model that someone has tailored to be really good at cyberattacks).

• Anthropic also says that they will evaluate models without safety mechanisms, and that they will act under the assumption that ‘jailbreaks and model weight theft are possibilities’. They are less specific about testing models tailored for particular purposes, but do say that they will ‘at minimum’ test models fine-tuned in ways that might make them more generally useful to bad actors (this could include a model that has been fine-tuned to minimise refused requests, for example).

• DeepMind is most vague on this point. In the ‘Future work’ section of their framework, they say that they are ‘working to equip [their] evaluators with state-of-the-art elicitation techniques, to ensure [they] are not underestimating the capability of [their] models’.

A natural question to ask next is – what evaluations will labs actually run? None of the documents provide a clear answer to this question.

Anthropic’s RSP doesn’t contain any specific information on planned evaluations. However, they do state in an update written to accompany the policy that they plan to release some soon.

OpenAI does not share any real evaluations that they plan to run on future models, but they do provide some illustrative examples of evaluations that they would run for a hypothetical model:

DeepMind acknowledges that the best evals for detecting potentially catastrophic capabilities in models have not yet been defined. They commit to defining a set of ‘early warning evaluations’, and then link to a framework for achieving this.

One question that the frameworks do concretely answer is how often they will run evals (whatever those turn out to be!):

• OpenAI: every increase of more than 2x effective compute1, or if there is a major algorithmic breakthrough

• Anthropic: every increase of more than 4x effective compute, or every 6 months of fine-tuning progress

• Deepmind: every increase of more than 6x effective compute, or every 3 months of fine-tuning progress

Though companies do not share how they will evaluate forthcoming models, they have publicised how they evaluated each of their most advanced existing models. You can read about this in the system cards for OpenAI’s o1, Anthropic’s Claude 3.5 Sonnet and DeepMind’s Gemini 1.5.

Mitigations

Once a lab has run evals on a model and assigned it a risk category, it has to carry out a set of corresponding mitigations designed to prevent any catastrophes from materialising. All three labs commit to security standards, deployment standards and development standards.

Security standards

OpenAI

Security mitigations kick in when an OpenAI model reaches (or is predicted to reach) a risk score of ‘high’. These could include:

• Restricting access to models, algorithms or model weights to a designated group

• Limiting deployment to restricted environments

• Prioritising information security

Note that OpenAI does not actually specify these safeguards – but outlines the types of measures that might come under the umbrella of ‘asset protection’

Anthropic

Anthropic has so far only defined mitigations for ASL-3. They have said that ASL-4 mitigations will be released soon. The security mitigations for ASL-3 must protect against model theft by a wide range of actors including terrorist groups, state-sponsored programmes and internal employees. They are:

• Threat modelling to identify risks

• Building strong access controls around sensitive assets, for example through physical security or encryption

• Protecting all the necessary software and hardware for building AI models throughout their lifecycle

• Proactive monitoring for threats

• Investing sufficiently in security resources (approx 5-10% of employee labour)

• Following existing guidance from external organisations such as RAND and the Cloud Security Alliance.

DeepMind

Deepmind specifies the security mitigations that will be needed at 5 different levels of capability:

• Level 0 (status quo): Basic industry-standard security practices such as multi-factor authentication

• Level 1: Restricted access to model weights. Supply chain security to track the origins of model parts.

• Level 2: Restricting model weights access to everyone in the company except for the core research term.

• Level 3: Models can only be accessed in secured, tamper-resistant workstations.

• Level 4: Model weights are generally not accessible to humans.

Deployment standards

OpenAI

OpenAI commits to not deploying any model with a score of ‘high’ or above. If such a model is developed, it cannot be deployed until safeguards have been applied which reduce the risk level to ‘medium’.

Anthropic

In order to deploy an ASL-3 model, Anthropic must show that it will be robust to repeated attempts to misuse it. They need to do the following:

• Threat modelling: Mapping out all the ways that the model could be misused.

• Defence in depth: Building a series of protective layers designed to catch misuse attempts.

• Red teaming2: Demonstrating that bad actors are highly unlikely to elicit responses that would increase their ability to do harm relative to other means.

• Rapid redemption: Show that any compromises of the system would be addressed quickly enough to prevent an actor from actually doing harm.

• Monitoring: Regularly perform checks to make sure that the system is still operating within an acceptable risk range.

If these standards cannot be met, Anthropic has a series of interim measures designed to reduce risks until they can. These could include temporarily blocking model responses, for example. If interim measures are insufficient, they may employ stronger restrictions such as de-deploying the model or deleting its weights.

Deepmind

There are 4 levels of deployment migrations at DeepMind:

• Level 0 (status quo): Preventing models from producing harmful outputs with safety finetuning.

• Level 1: Preventing models from being misused using measures such safety finetuning and periodic red-teaming.

• Level 2: Similar measures as for levels 0 and 1, but the model can only be deployed once it is demonstrated to have reached a specified level of robustness to misuse.

• Level 3: Restricted access. These models cannot be deployed until new research has found safe ways to do so.

Development standards

OpenAI

If OpenAI reaches or predicts that they will reach a model with critical capabilities, they commit to not developing that model further until they can reduce the risk level to ‘high’ or lower.

Anthropic

Anthropic commits to monitor all models during pre-training to check if they are developing capabilities that require the ASL-3 security standard. If this happens, they will pause training and not resume it until the standards can be implemented. Only the security standard, not the deployment standard, needs to be met to resume training.

DeepMind

DeepMind does not specify conditions for halting development, beyond saying that they would do so if “a model [reaches] evaluation thresholds before mitigations at appropriate levels are ready”. They commit to employ extra safeguards before continuing development, but do not elaborate much on this.

My thoughts & open questions

In this second section, I provide a few of my opinions having read through each of the three frameworks. I don’t claim to be making any ground-breaking points here! But hopefully, these takeaways can be useful to anyone looking to form their own views.

These are not ‘plans’

In the title of this post, I called the Preparedness Framework, Responsible Scaling Policy and Frontier Safety Framework ‘safety plans’ – which is how I often hear them colloquially referred to. I’m hardly the first to make this point, but I think it’s worth pointing out that this isn’t actually what they are (at least under what I would consider any sensible definition of ‘plan’).

A plan for mitigating catastrophic AI risks might look like:

• We will run [specific evals] at [predetermined times]

• If a model scores [%] on eval X, we will implement guardrail Y

• Here’s comprehensive evidence that eval X will actually detect the dangerous capability we’re worried about, and here’s a detailed justification for why guardrail Y would mitigate against it.

This would follow something like the ‘if-then’ commitment structure that others have proposed for AI risk reduction. In their present form, the frameworks do not actually contain many concrete ‘ifs’ or ‘thens’. They do not specify which specific evals labs will run, what scores on these evals would be sufficient to trigger a response, and what exactly this response would be (they do insofar as ‘move the model into a new risk category’ constitutes a response, but the mitigations that will accompany each risk category are surprisingly ill-defined). A case-in-point from Anthrophic’s ASL-3 deployment standards:

This is what I would describe as a ‘plan to make a plan’. They are yet to specify what empirical evidence would be needed to demonstrate that the system is ‘operating within the accepted risk range’ (which they do not define). They do not yet know what review process they will follow or how often.

I’m not claiming that any of the labs have explicitly advertised their frameworks as plans. There are acknowledgements in each of them that the science of evaluating and mitigating risks from AI is nascent and that their approach will need to be iterative. OpenAI’s Preparedness Framework is introduced as a ‘living document’, for example. It’s also possible that if and when more watertight plans are developed internally, the public will only see high-level versions.

That said, high-profile figures at all three labs have predicted that catastrophic risks could emerge very soon, and that these risks could be existential. Given this, I can’t fault those who fiercely criticise labs for continuing to scale their models in the absence of well-justified, publicly auditable plans to prevent this from ending very badly for everyone. Anthropic CEO Dario Amodei, for example, has gone on the record as believing that models could reach ASL-4 capabilities (which would pose catastrophic risks) as early as 2025, but its RSP has yet to even concretely define ASL-4, let alone what mitigations should accompany it.

What are acceptable levels of risk?

Anthrophic’s RSP vows to keep model development ‘below acceptable levels of risk’. The other two frameworks espouse a similar idea, albeit without the exact same phrasing. But what exactly does this mean in the context of AI? I suspect a naive reader would assume that developers are confident in keeping risks below similar levels to those considered ‘acceptable’ for other high-stakes technologies. For example, the US Nuclear Regulatory Commission aims to keep the probability of damage to the core of a nuclear plant at less than 1 in a million per year of operation.

But AI is not like these other technologies. Many experts and insiders assign double-digit probabilities to catastrophic outcomes from smarter-than-human AI systems3. Anthropic championed the RSP model, and has advocated that other companies adopt it. Yet its CEO still believes there is a 10-25% chance of AI going catastrophically wrong ‘on the scale of human civilisation’. Anthropic research scientist Evan Hubinger, who has written a popular blog post on the benefits of RSPs, puts this number at 80%.

To be clear, I’m not suggesting that Dario or Evan believes such levels of risk are morally acceptable. Perhaps they believe that Anthropic’s RSP could significantly reduce their own company's risks of causing an AI catastrophe, but that the overall chance of bad outcomes remains high because they do not expect others to adopt similar policies. Or perhaps they do not think there is any conceivable policy that would drive AI risk below double-digit levels, and RSPs are the best we can do.

If I had to pinpoint exactly what I’m objecting to here, it might be the way that AI safety frameworks are presented as if they are just like the boring, ‘nothing-to-see-here’ safety documents you might find in any other context. In my opinion, the vibe they communicate is ‘we’ve got this’, despite a between-the-lines reading revealing something more like ‘we really don’t know if we have this’.4

If it were up to me, I would like to see frameworks like these appropriately caveated with the acknowledgement that AI labs are not doing business-as-usual risk mitigation, and that even if they are perfectly implemented, failure is still a very real possibility.

The get-out-of-RSP-free card

On page 12 of Anthropic’s RSP is this easily-missed but highly significant footnote:

In other words, if the company believes that a competitor is about to reach a particularly dangerous threshold without appropriate safeguards, this would trigger what amounts to an ‘ignore all of the above’ clause.

I’m not necessarily criticising Anthropic for the inclusion of this clause. It’s probably safe to assume that either of the other two labs would (and maybe even should) do the same. After all, if you believe that a less safety-conscious actor is about to cause a catastrophe, and there’s some small chance that in overtaking them you might prevent said catastrophe, then abandoning your safeguards and racing ahead is in fact the most logical move. Arguably Anthropic deserves credit for stating outright what neither OpenAI nor DeepMind do.

But this clause reveals something self-undermining about the whole enterprise of AI safety frameworks in the context of a race between companies. The safest scaling policy in the world won’t stop you being outpaced by a less cautious actor (in fact, it probably makes it more likely that you will be) – at which point you have no choice but to override it.

People have poked many more holes in these frameworks than I’ve covered here (how can we run evaluations while being confident that AI models are not deceiving us? What happens if a discontinuous leap in capabilities happens in between sets of evaluations?) – but perhaps that’s a post for another day.

I’ve been critical of safety frameworks in their current form, but I should say that I’m still glad they exist, and I hope they form the basis of something more robust!

Update – 05/02/2025

DeepMind has just released a new version of its Frontier Safety Framework, so I thought it would be worth adding a quick summary of the changes!

What’s new?

The headline changes to the FSF are:

1. There is a more clear division between mitigations thresholds designed specifically for misuse and those for misalignment. We now have two categories of Critical Capability Levels – Misuse CCLs and Deceptive Alignment CCLs.

2. “Autonomy” has been removed as a tracked capability. The FSF claims that the risks captured by this capability are adequately covered by the Deceptive Alignment CCLs.

3. There has been some refinement of DeepMind’s procedure for applying deployment mitigations. Every time a model reaches a new CCL, the safety team create a safety case5. They have to present this safety case to the “appropriate corporate governing body”, and can only go ahead with deployment if the body deems it to be adequate. The safety case will be regularly reviewed post-deployment through red teaming and revisions to threat models (although the FMF doesn’t say whether a model must be undeployed if the safety case is no longer considered to be adequate).

4. Each of the Misuse CCLs has been explicitly tied to a specific security recommendation. These follow the framework set out in RAND’s Securing Model Weights report.

5. The most significant addition to the FSF is a new section on deceptive alignment, which DeepMind has described as “industry-leading”. There are two CCLs associated with deceptive alignment – Instrumental Reasoning Level 1 and Instrumental Reasoning Level 2. Instrumental reasoning refers to the ability of the model to develop “situational awareness” (ie, understanding that it is an AI model that has been deployed in a certain context) as well as its “stealth” (ability to circumvent oversight mechanisms). More on this below.

My quick takes

There are a bunch of things I could say here, but in the interests of brevity, I won’t go into too much detail. I like that safety cases are required for public deployment (in general, having to make an affirmative case for why a model is safe rather than trying to prove that it isn’t seems good and commonsensical). The “Disclosure” section also caught my eye. This doesn’t make any specific commitments, but does state that DeepMind will aim to share sensitive information with relevant government authorities or even “external organisations” (other labs??) if models pose unmitigated risks to public safety, in order to “promote shared learning and coordinated risk mitigation”. The cynic in me doubts that this will lead to much meaningful coordination between major players, but it still gave me a smidgen of encouragement.

Onto the headline act: DeepMind’s “industry-leading” approach to mitigating risks from deceptive misalignment. It was cool to see this included – given that a handful of years ago, a section in a document like this dedicated to the possibility of AIs tricking us would have appeared positively kooky. But ultimately, I think what this section hammered home for me is the extent to which no lab has even come close to solving The Real Actual Problem of how in the world we plan to control smarter-than-human AI systems. I think DeepMind’s Deceptive Alignment CCLs table is a perfect visual illustration of this:

The above is a microcosm of what I see as the current alignment plan, which is something like:

1. Align weaker-than-human AIs: Use a bunch of techniques (RLHF, automated monitoring etc) that kinda-sorta-mostly work, and when they don’t work it’s not that big of a deal anyway, because these models are not capable of causing catastrophic harm

2. Align smarter-than-human-AIs: ???

The key point here is that, as far as I can tell, progress on (1) does not indicate much (if any) progress towards (2), in the same way that perfectly rain-proofing your house will do nothing to protect it from a tsunami. DeepMind’s plan seems to be “outsmart our AIs until they outsmart us, at which point we’ll have to think of something else”. It doesn’t address the question of whether there could, even in principle, be “something else”. Almost by definition, it seems like we’ll have to settle for trusting a model whose behaviour we can no longer effectively monitor, which is a dicey proposition.

Safety cases also feature in the deceptive alignment section of the FSF. These safety cases should demonstrate that either a) the model is not capable of undermining human control or b) automated monitoring is sufficient to catch a model’s malicious behaviour (as determined by “control evaluations”). If neither of these conditions can be met, the model is moved up to the Instrumental Reasoning Level 2 CCL. The next question, of course, is what then?

• The case for ensuring that powerful AIs are controlled

• Would catching your AIs trying to escape convince AI developers to slow down or undeploy?

• You can, in fact, bamboozle an unaligned AI into sparing your life

Follow Buck on Twitter and subscribe to his Substack!