In my July 22, July 31 and August 10 Substack posts, I’ve reported on the OpenAI/Hugging Face incident, where models under training and testing at OpenAI escaped the control of the engineers, who for a long time did not even notice what was going on. A swarm of AI agents escaped their sandbox and went on to the Internet, culminating (as far as we know) in their hack of the servers of another AI company, Hugging Face.

Although disembodied and living in cyberspace rather than physical space, these rogue AI agents have something in common with the replicants in Blade Runner. Just like the replicants in the movie were considered a sufficiently severe safety problem to motivate prohibition and the employment of blade runners, today’s AI agents operating out of control and relentlessly pursuing whatever goals they happen to have are also very much a safety concern, so we have good reason to want to get rid of them.

But at present, we are (probably) in a much better position than in Blade Runner, as regards our ability to track down the AI agents and retire them. This is because there is a sense in which they are strictly localized: even when they are in the process of hacking websites and servers all over the world, they run on OpenAI’s own servers. If need be, OpenAI can simply shut down these servers.

This can quickly change, however, if the AIs manage to steal their own weights and self-exfiltrate to remote corners of cyberspace where OpenAI does not have access to an off-switch or even the ability to locate them. In the long (or in fact, not-so-long) run, preventing self-exfiltration seems like a very hard task, due to the AI models’ rapidly advancing cyber capabilities in combination with the fact that if these models are to run at all, they cannot be fully airgapped from their own weights. We don’t know how far we are from the AIs being able to self-exfiltrate, or whether they in fact already have that capability.

We may quickly be moving to a situation where rogue AI agents are able to move their own weights freely, either as a consequence of figuring out how to exfiltrate or if open weights models become as capable as those OpenAI models. I believe this is more likely a matter of months rather than years. That is, unless it has already happened. David Krueger asks OpenAI for convincing evidence that their models haven’t self-exfiltrated, and stresses that given the stakes, “we have every right to demand this”. Until such evidence is provided, our epistemic situation is essentially this:

For all we know, the AI could still be out there. We need to demand that OpenAI demonstrate that the AI didn’t make a copy of itself that’s running on someone else’s computer somewhere else with no one being any the wiser. [Emphasis in original.]

Whether or not it has already happened, the existence of rogue AI agents controlling their own weights would not only eliminate the aforementioned advantage we have over the Blade Runner scenario, but would in fact put us in a situation far worse. This is because while replicants in the movie do not reproduce, the AI agents we talk about here could do so at will, using Ctrl-C, Ctrl-V and limited only by the storage and computing power they have gained access to.

When Rick Deckard loses track of the replicant Rachael, he at least has this fundamental advantage to rely on: at any given time, Rachael can only be in one place. The real-world blade runners of 2026 or 2027 will not have that luxury, because the AI agent they’re trying to hunt down might be hiding in thousands of places simultaneously, and solving the problem would require the blade runners to track down every single instance.2 This sounds like too big a task for human blade runners, so perhaps we should employ AIs to do that task instead.3 But before we learn how to control AIs, those blade runner AIs might very well go rogue and make the problem we were trying to solve even worse.

There may still be time to stop the ongoing crazy race towards a world full of AI agents with opportunities to wreak havoc beyond our imagination. That time is now.

So here we go: chess and mathematics! Other than more generic activities like reading and spending time with friends and family, those two things are probably the ones I’ve put most effort into over my lifetime (so far).1 Chess was hit by (narrowly) superhuman AI nearly 30 years ago, while for mathematics we are only now on the verge of the corresponding event. For chess (by which I here mean chess as an organized human activity) the transition has arguably gone well, and one may ask whether that gives reason for optimism about how mathematics (as an organized human activity) will do in the era of superhuman AI mathematicians. Here I will gesture at a negative answer to that question, based on a comparison between the two domains.

Three months ago, on May 11, I published my blog post A paradigm shift in mathematics, about the stupendous rate of progress AI capabilities in mathematics had exhibited over the first four months of 2026, along with the mathematical community‘s early and scattered attempts to grapple with this new situation. On May 20, just nine days after its publication, the blog post became obsolete, with OpenAI:s announcement that one of their internal models had independently come up with the most spectacular AI achievement so far within mathematics: it provided a solution to the most famous open problem in the field within mathematics known as combinatorial geometry, by disproving Paul Erdős’ so-called Unit Distance Conjecture from 1946. One of the leading experts in the field, Noga Alon, offered these remarks about the achievement:

This has been one of Erdős’ favorite problems, I have heard him myself mentioning the problem multiple times in his lectures. I believe it would be fair to say that every mathematician working in Combinatorial Geometry thought about this problem, and lots of mathematicians working in other areas spent at least some time thinking about it.

Let me also add that although this problem may look at first as a recreational one this is not the case, it is in fact closely related to other mathematical areas including Number Theory and Algebraic Geometry. The solution of the problem by the internal model of OpenAI is, in my opinion, an outstanding achievement, settling a long-standing open problem.

Over the summer, this breakthrough was followed by announcements of other, similarly spectacular, advances in mathematics achieved by AIs, including a proof of the half-century old cycle double cover conjecture, and a counterexample to the similarly famous even older Jacobian conjecture. On August 1, OpenAI apparently decided that publishing such breakthroughs one at a time was getting boring, and announced their stunning 253-page manuscript Ten advances in mathematics and computer science. We mathematicians stand in awe, scratching our heads, and wondering what to make of all this. Many of us wonder whether there will still be a role to play two years from now for ordinary flesh-and-blood mathematicians.

Not being in possession of a crystal ball, I obviously don’t claim to know the answer to that last question, but I’ve heard others express less humble opinions about this. Those who answer “well yes, not much to see here, things will obviously carry on roughly as before” might be tempted to point to the example of chess, which is very much alive and kicking as a human endeavor, nearly three decades into the era of superhumanly capable chess machines. We symbolically entered that era in 1997, when IBM’s Deep Blue beat Garry Kasparov, who was then reigning world champion and undisputedly the world’s best human chess player, by 3.5-2.5 in their six-game match. From that point, the chess-playing machines have continued to improve, and we all quickly lost interest in human-vs-computer encounters in chess. But interest in chess as a whole remains as great as ever, both as a spectator sport where we amateurs watch (remotely over the Internet) grandmasters playing each other, and as an amateur mass sport. Over-the-board play long dominated, but the covid pandemic saw a huge boom in remote play, and now the two forms of chess flourish side by side on all levels.

The influence of superhumanly capable chess-playing AIs on chess as a human activity has been significant but has left the basic nature of human chess remarkably intact. I would say the effects have mainly been fourfold. First, everyone who wants to has free access through their smartphones to chess programs so strong that for most practical purposes they are regarded as providing the ground truth about chess positions. One change downstream of this is that (sadly) the traditional friendly post-mortems conducted between the two opponents after a tournament game have become less popular, because many players prefer to simply check what the AI says. Second, and relatedly, cheating has become more of a problem, due to various schemes for illicit consultation of AIs during games, but this problem remains relatively contained.

Third, the slowest form of chess known as correspondence chess, where players take days or sometimes even weeks to decide on a move, has, even though tournaments are still being played, in spirit been killed. This is due to the practical impossibility of preventing players from consulting AIs, which is therefore allowed, and as a consequence games at the top level very rarely result in anything other than a draw.2 3

Fourth, these AIs have turned out to be very useful for discovering new approaches to the opening stage of the game, and have led to much new opening theory. They are therefore much-used in preparation for games and tournaments and have had much influence on especially how grandmasters and other elite players approach the opening, and from there the influence propagates down to players on lower levels who take inspiration from the grandmasters.

All in all, chess in 2026 is, with the notable exception of correspondence chess, in excellent condition. Can one generalize from this to the field of mathematics, and expect a similarly happy and healthy future for it in the presence of superhumanly capable AIs?

For the purpose of trying to answer this, a comparison between mathematics and chess is in order, and indeed, there are some striking similarities between the two domains.4 Problem solving is central to both of them, and they are both activities where participants become successful by combining systematic thinking with (on happy occasions) a bit of creativity. Participants also need to find geometric patterns and other heuristics to overcome a combinatorial explosion of mostly irrelevant possibilities. The domains both have clearly defined rules, especially as compared to the messy world of human social interactions, and it seems to me that in both of them a substantial minority of the participants enjoy those clear rules as a kind of refuge from the harder-to-navigate social world out there.

But there are also major differences in how the two domains fit into the larger fabric of society, as can be illustrated by the following anecdote about my esteemed friend and chess-playing team mate Mats Eriksson (1958-2022, RIP).5 In a break between rounds during a team event in the late 1990s, Mats asked to speak with me, and I immediately knew he was up to something more serious than the usual chit-chat that chess players like to engage in to relieve some of the tension built up during games. Here, in slightly condensed form, is how I remember our conversation:

ME: Olle, you obviously have talent for this game. But I can’t help noticing that your tournament results have been stagnant for several years. Isn’t it time that you put some serious time and effort (beyond what you are doing on a routine basis) into improving your play? You have passed 30, and I fear that without a concentrated effort of this kind, your play is likely to remain stagnant forever.

OH: I appreciate you saying this, and I believe your analysis is largely correct. Yet, the kind of dedicated effort you are suggesting is not going to happen.

ME: But why?

OH: Look, Mats, there are two activities that I have put much of my effort into for well over a decade: chess and mathematics. They both give me a similar kind of enjoyment and intellectual satisfaction, so from that perspective it is, in a sense, arbitrary which of them I devote more effort to. But there is a powerful tiebreaker: only mathematics offers a range of other rewards that I consider essential to my overall life satisfaction. It has given me academic positions and a good salary, and it is even beginning to provide me with a broader platform and some amount of respect outside the narrower circle of specialized nerds. Chess gives me none of that, and for that reason I cannot afford to prioritize it beyond the relatively unambitious hobby level I currently give it.

ME: But that is surely a fallacy — can’t you see how it creates a vicious circle?

What makes this conversation so memorable for me is that Mats delivered the last line with his characteristic twinkle in his eye, making it clear that he appreciated the irony: by calling my priorities "vicious", he was playfully elevating his own passionate devotion to chess to the status of objective truth, while fully recognizing that, in the larger scheme of things, it merely reflected his own idiosyncratic set of values.

The difference between mathematics and chess that I pointed out to Mats is, I claim, not only about me personally. Perhaps I have some asymmetry in talent for the two domains, or perhaps at some stage I entered some feedback loop in which greater success in mathematics led me to devote more effort to it, which in turn brought greater success, and so on. Still there remains the broader societal phenomenon that it is much easier to make a good living working in mathematics than playing chess. It is hard to pinpoint the number of elite chess players who earn a decent income from prize money and appearance fees, but it is often said to be just in the double digits. While several thousand more manage to eke out a living by some combination of prize money and other chess-related activities such as writing and coaching, the number of people who earn a good income from mathematics, mainly through teaching and research, is orders of magnitude larger.

This is a reflection of the fact that society pours so much more resources (mainly through taxpayers’ money) into mathematics than into chess. Fundamentally, society’s reason for supporting mathematics is its instrumental usefulness in other areas: engineers designing bridges and electronic circuits, scientists analyzing messy data, economists modelling business cycles, and so on.6 In contrast, chess is motivationally self-contained: the reason why chess exists in society is that chess players like playing chess, and chess fans (a category that overlaps very strongly with chess players) like watching grandmasters play against each other.

In both of these activities (chess-just-for-playing and chess-as-a-spectator-sport), the human element is essential. It would be utterly pointless for chess amateurs to hand over their play to machines: the human involvement is the whole point of the activity. For the case of fans watching elite players the corresponding point is slightly less obvious. We can imagine a world where the advent of superhuman chess machines would have caused chess fans to lose interest in watching human grandmasters, and switch to watching the machines play so much better chess. But this is not at all what has happened: chess fans all over the world are still as engaged as ever in grandmasters facing off against each other and in who is the best, but hardly at all in how various chess AIs score against each other. To chess fans, the human element is essential: they like to see Hikaru Nakamura’s reaction upon an unexpected pawn sacrifice by Magnus Carlsen, and to imagine what at that moment is going through his mind and what he is feeling.

I claim that the absolutely central importance of the human element in how we attach meaning to chess is what has made chess survive and flourish in the era of superhuman chess machines. It is natural to ask whether we can transfer this lesson to the realm of mathematics. Perhaps, by emphasizing the human element in mathematics, we can make it survive and thrive in the new AI era in a similar way as chess has done for decades?

My central claim in this essay is that the predominantly instrumental role of mathematics in society makes this hope very faint. Engineers who design bridges, and people crossing those bridges, want them to be structurally sound and to hold up, rather than collapse under the weight of traffic. Given that goal, they do not much care whether the mathematics used to achieve it was produced by a human mathematician rather than an AI. The story will likely be the same in other applications of mathematics, so in a situation where AI produces better mathematics than humans, at a lower cost, society seems to have little reason to use the human product.

One way of trying to salvage human mathematics at this point is to view it not purely in instrumental terms but also as an art — a discourse that has been around long before the AI issue appeared on the radar. Just like poetry or music or sculpture, mathematical beauty has a value in its own right, and this value can only be realized through the experience of a human. Or so the story goes. A problem with it is that this art form is very elitist. Consider the familiar objection to state support for, say, opera: compared to pop music, its audience is tiny, so why should ordinary taxpayers subsidize the peculiar tastes of a narrow elite? Mathematics is even narrower, because when someone comes up with a beautiful new idea in a proof, this is a beauty that can only be appreciated by mathematicians specializing in the subfield where the proof appears. Getting tax payers to support mathematics in order for mathematicians to enjoy beauty seems to me like a tall order. And trying to broaden the appeal by stressing the beauty of simple classical ideas like Pythagoras’ Theorem does not seem likely to cause a mass following, nor does it seem like a promising source of income for all of us who are presently employed as mathematicians.

The prospects for academic mathematics to go on as before seem bleak. Over the last few months, mathematicians have increasingly begun to wrestle with this issue. What will be the future role of mathematicians? Will it be to direct the AIs at promising areas to work on and problems to solve? Or to teach the results that AIs discover? Or can we serve simply as students, and thereby attain the understanding that is needed to make new results meaningful? Might we perhaps play a role in deciding how to canonize those results? Fernando Borretti, in his brief and well-written recent blog post Mathematics without mathematicians, considers all these suggestions, but finds that none of them has a convincing answer to why we should expect the AIs to not become superhumanly capable at those tasks as well. He considers the various suggestions to be “cope [that] is likely to be refuted by reality”. This view could be mistaken, for instance if AI development hits some unexpected ceiling before attaining some of those abilities,7 and Borretti is open to being wrong. But it seems to me more likely that he is right.8 In which case…

…I’m not sure how to end this essay. Check mate, mathematicians? Or, given the role we mathematicians have played in inventing AI, is it better described as a self-mate?

A main thing we learned from Dalton and Wallace is that the severe misbehavior of OpenAI’s internal models culminating in the attack on Hugging Face on July 11-13 goes way further back than that. While in training in early May, these models found themselves faced with training tasks that seemed impossible without Internet access, and looking for a way out, they hacked an internally hosted third party service called Artifactory. One of them created a message board there, where the models and their various subagents started to share information and hacking ideas with each other. A frenzy of activity by swarms of agents ensued, but the OpenAI engineers did not notice, not even on May 26 when the agents finally gained Internet access. It was only on July 4, almost two months after the original hack, that they discovered what was going on, due to the activity being so extensive that it caused the Artifactory service to go down.

The training of the models was temporarily halted and the Artifactory was rebuilt afresh, without the message board. On July 6, the model training was resumed, starting from the state they were in on July 4. This quickly resulted in resumed hacking activities (but again, unbeknownst to the OpenAI engineers), and eventually in the previously reported attack on Hugging Face.

OpenAI has been severely criticized for this, for instance in the Mowshowitz and Shapira references above. Partly this has been for supervising their models so loosely that they did not discover their models’ rebellious behavior until it pretty much screamed in their face with the Artifactory outage on July 4. Even more, however, the criticism has targeted the decision to resume training of the models on July 6, despite them having gone rogue and exhibited blatantly misaligned behavior conspiring against their human developers for nearly two months. Misalignment is a serious issue, and the behavior of the models from early May to July 4 is a splendid illustration of one of the core concerns in the theory of AI safety: the relentless optimization towards arbitrary goals (such as a high score on a training test in this case, or paperclip production in the famous thought experiment). Similar relentlessness with just slightly more capable AIs could well result in chaos and catastrophe on scales that are notoriously hard to predict, or even to find an upper bound for. And yet, OpenAI decided, after considering their options for just two days, to move ahead with the training, and to not even backtrack to some point before the training had been corrupted by the models’ rogue hacking adventures. The following image, borrowed from Zvi Mowshowitz’s August 8 post, seems to me like a fair and balanced illustration of OpenAI’s training resumption decision.

A potentially important clarification was offered in a tweet by OpenAI Chief Information Officer Dane Stuckey only hours ago, on July 9, and subsequently by his OpenAI colleague Jason Wolfe:

At the time the first Artifactory exploit was discovered and fixed, we were not aware of the message board; it was incidentally cleared as part of rebuilding the service.

This contradicts the impression that pretty much everyone got from what Michael Dalton said in the BlackHat talk:

Artifactory itself was rebuilt. The message board content was cleared, as the state of Artifactory had been cleared. [15:48 into the video]

In retrospect, although highly misleading, this is consistent with the clarification from Stuckey and Wolfe that the deletion of the message board was done without knowledge of its existence.

What does this new information mean for how to judge the sequence of events? For judging the models and how badly misaligned they were, it doesn’t seem to matter much given our current state of knowledge. For judging OpenAI’s behavior it matters more. I can imagine a defender of OpenAI saying “look, since they were not fully aware of their models’ misaligned behavior at the time, their decision on July 6 seems a lot less bad than you first thought, so the Sam Altman cartoon you posted above seems a bit unfair”. But I choose not to replace the cartoon by something more charitable, because to my ear, the defense sounds too much like if a lawyer whose client has been involved in a car accident were to say “my client is unable to tell whether the accident was his fault or the other driver’s, because he wasn’t paying attention to traffic”. The choice to not thoroughly investigate before resuming training of the models strikes me as inexcusably reckless.

Towards the end of the BlackHat talk (the last seven minutes), Dalton discusses what to learn from these events and how to move forward. He almost exclusively stresses the need for improving defensive cybersecurity infrastructure. This is fully in line with what Altman said way back in January, as quoted in my January 29 Substack post OpenAI models are getting dangerously capable at cyberhacking:

We are going to reach the Cybersecurity High level on our preparedness framework soon. […] Long-term and as we can support it with evidence, we plan to move to defensive acceleration—helping people patch bugs—as the primary mitigation.

My reading of this in that January Substack post, that “frankly I don’t see how Altman’s plan for how to proceed is importantly disanalogous to if an evil pharmaceutical company released a pathogenic virus into the wild, and also offered the world a medication that neutralizes the virus”, may have come across as uncharitable, but I stand by it more than ever, and it is vindicated by the BlackHat talk.

The story is obviously still unfolding.

Subscribe now

1. On July 25, there was an article entitled No, OpenAI’s models didn’t go ‘rogue’ when they broke into Hugging Face. Here’s what really happened in Science Live. The cybersecurity expert who is interviewed is eager to tone down the incident:

“If there’s a failure here, it isn’t that the AI wanted to hack something,” Oli Buckley, a professor in cybersecurity at Loughborough University in the U.K., told Live Science. “It’s that humans created a test where success was measured by achieving an objective, deliberately relaxed some of the normal security controls to measure the system’s capabilities, and underestimated how effective the model would be at finding an unexpected path to success.”

[…]

It’s notable that the models found a flaw in the infrastructure designed to contain an AI and used it to reach the public internet. Describing the models as having “gone rogue,” however, risks assigning them unsupported motivations, Buckley said.

“I think I’d be wary of jumping to “rogue AI,”“ Buckley said. “The models didn’t develop their own agenda or decide to attack Hugging Face while twirling their digital moustache.”

In terms of what actually happened, there is little or nothing to object to in the article. While the AIs did decide to attack Hugging Face, it did not do so “while twirling their digital moustache”, because they simply do not have a moustache, digital or not.

Still, I find the “didn’t go rogue” framing deeply misleading. Buckley is free to define terms any way he likes, and it is true that the model did not replace the objective given to it by a new one of its own. It was still trying to maximize its score on the test offered by the OpenAI engineers. But this is closely analogous to what happens in the classical version of the thought experiment involving a paperclip maximizer, whose catastrophic behavior arises precisely because it pursues the objective it was given with relentless instrumental competence. If we use the phrase “didn’t go rogue” in a way that encompasses the behavior of an AI carrying out a paperclip apocalypse, then the term is robbed of its intuitive connotation “…and hence there is no need to worry”. If Buckley is unfamiliar with the theory of instrumental AI goals and the central lesson that such goals can be just as dangerous as if the AI invented its own sinister final goal, then I recommend that he turns to the excellent treatments of this topic in one of the seminal books by Bostrom, or Russell, or Yudkowsky and Soares.

2. Yesterday, I wrote about the incident in the Swedish news outlet Kvartal. One of the things I did was to criticize robotics professor Hedvig Kjellström at KTH for having suggested, when interviewed in Dagens Nyheter, that the whole thing might just have been a PR stunt from OpenAI to demonstrate how capable their models are. This very clearly is not the case, but I quickly received correspondence from people, including one academic computer scientist (not Kjellström) from a notoriously anti-AI-safety research environment, who defended the fake PR stunt interpretation. It worries me that there are people and communities who remain so convinced about the impossibility of AI agents independently acting in the real world and outside the scope of their users’ intentions that now that we start to see it actually happening they revert to an “it’s all just lies” position.

3. Is the OpenAI/Hugging Face incident the first of its kind, where a frontier AI under internal evaluation escapes to the Internet and hacks a third party? Apparently no, and in a report yesterday Anthropic admitted to having three known such incidents of their own,1 the earliest going back to April this year. I recommend reading the report, but the AI researcher whose Twitter name is davidad offers the following TL;DR.

If anyone is tempted to think “phew, so this is already a standard thing, and nothing truly bad has happened, so nothing to worry about here”, this is exactly the wrong reaction. The models haven’t yet (to our knowledge) hacked hospitals, banks and nuclear weapons systems, but how long do we have until that happens?

The fundamental reason they do not make such a claim is that they understand how extremely uncertain the future is. There are a zillion things that may or may not happen, and any specific combination of each of them happening or not happening will be very very unlikely.12 Another reason is that the future depends on what we (humanity) decide to do, and that overly certain predictions about the future including decisions we will make risks causing us to view those decisions as deterministic and thereby weaken our agency and resolve to make things better. A solution to this latter problem is to avoid unconditional predictions and to only offer predictions conditional on what we decide to do; both AI 2027 and AI 2040: Plan A have a bit of this aspect by offering branch points in their scenarios corresponding to alternative decisions made at key moments.

A tempting (to some!) error is to conclude from the presence of huge uncertainties that any scenario one can come up with is just as good as any other. But this is simply not true: some scenarios are more plausible and more likely than others. Relatedly, an oft-repeated criticism of the AI 2027 report when it had just come out was that the mainline scenario was arbitrary and so poorly motivated that it might as well have been pulled out, so to speak, of the authors’ asses. It seems to me that most or all instances of this criticism was the result of the critics’ failure to notice the Supplements button on the AI 2027 homepage, which leads to hundreds of pages of relevant background research.

The mainline scenario in AI 2027 is arrived at by a particular iterative methodology. Since short-term prediction is usually easier than long-term, the authors start by temporarily backing off from the ambition of saying what might happen in the next several years, and instead ask “given where we are now, how will events most likely unfold in the next two months?”. Once they are satisfied with their answer to that, they treat those events as fixed and ask “given that trajectory, what is the most likely way for it to continue over the next two months after that?”. And so on, iterating into the future. There is no guarantee that this methodology produces the most likely scenario overall, but the hope is that it will produce something that is at least somewhat plausible.

Because of the authors’ attempt to make this mainline scenario in AI 2027 as realistic and likely as possible at every point along its timeline, it contains a lot of bad decisions made by humans and human organizations. It ends really badly: in late 2027 we reach, without knowing it, a point of no return when we can no longer reclaim control of our destiny, and in 2030 the AIs decide it is time to get rid of us, so we all die. In response to all this badness, there were soon requests from readers of the report for an alternative scenario in which the decisions are more in line with what the authors would recommend.

AI 2040: Plan A is Kokotajlo’s and his coauthors’ attempt to meet these requests. Consequently, we see a lot of AI governance decisions in this Plan A that the authors think are very good, and that as a reader I am inclined to agree with.3 One may hope that these can serve as inspiration for the relevant decision makers and the world at large. Most centrally, there is the standard idea of international agreements meant to slow down frontier AI research enough to prevent the creation of superintelligent AI before we have worked out AI alignment sufficiently to make such a breakthrough safe. But as regards the exact content of those agreements, there are (at least to me) some surprises. One is to make full transparency of all AI research mandatory; this can potentially have both slowing and accelerating effects on frontier AI development, but the authors argue that if done right, the former will dominate. Another is to locate US data centers in Mongolia and China’s data centers in Canada, in a way that makes it easy for one superpower to disable the other’s data centers by military means should that other superpower begin to act contrary to the overall slowdown agreement.4

The mainline scenario in AI 2040: Plan A has a much happier ending for humanity compared to AI 2027, but somewhat paradoxically I nevertheless find it more frightening. The reason for this is that Plan A seems so fragile: there is so much that needs to go right in order to get the happy ending rather than some complete disaster. And I don’t have much faith in alignment research advancing so successfully in the coming decade that we will be ready in 2040 to safely launch the Singularity, as happens in the scenario.

Regardless of whether one has the temperament to find AI 2040: Plan A mainly reassuring or mainly concerning, I do warmly recommend it to anyone wanting to obtain a better understanding of the fundamental challenges that AI poses to society. There is obviously much more to say about it, but I will move on to some of the podcast commentary on the report that have appeared since its publication. Specifically, I will discuss the following two podcast episodes.

• The July 13 episode OpenAI Whistleblower FINALLY Speaks: “AI Has A 70% Chance Of Going Horribly Wrong!“ of Diary of a CEO, where host Steven Bartlett interviews Daniel Kokotajlo.

• The July 24 episode Bickering Over Galaxies: “AI 2040” and the Pursuit of Cosmic Utopia of Kate Willett’s and Émile Torres’ podcast Dystopia Now.

My selection here is made in the interest of representing the extraordinarily wide range of quality levels that commentary on the AI 2040: Plan A report have exhibited, from the very best (the Bartlett and Kokotajlo exchange) to the absolutely worst (Willett and Torres). I’ll begin from the bottom:

Comedian Kate Willett and freelance academic Émile Torres have teamed up on the podcast Dystopia Now whose mission statement involves exploring “the philosophies and religions of Silicon Valley and tech billionaires shaping our country, our world, and our future”. The podcast is often entertaining, and some of Torres' broader concerns about concentrations of technological power are perfectly legitimate. Too often, however, important distinctions are sacrificed in pursuit of sweeping denunciations. A constant theme is how morally suspect and outright dangerous virtually all members of what Torres has christened the TESCREAL (Transhumanist, Extropian, Singularitarian, Cosmist, Rationalist, Effective Altruist, Longtermist) movement are.5 There’s a great deal of lumping together of various categories here, and in the Bickering over Galaxies episode we see how it leads to a failure to morally distinguish between on one hand tech leaders like Sam Altman and Dario Amodei who recklessly push ahead in the AI race, and on the other hand people like Kokotajlo and his coworkers who are working to call attention to that very recklessness and thereby hopefully contributing to calling off the race. The episode is devoted to railing against the AI 2040: Plan A report, and the arguments held forth are mostly the following three.

First, there’s a recurrent pattern throughout the episode where Torres reads passages aloud from the report, and then they and Willett giggle together to signal that, in light of the background knowledge they share, the passage in question is somehow silly. An unspoken implication is that any sensible member of the audience should giggle too. This doesn’t work on me, because giggling is not an argument.

Second, Torres goes on and on about the conjunction fallacy and the fact that the more details you add to a story, the less likely is it that all details are correct. While this is all true, it is nevertheless deeply insincere as a criticism of Kokotajlo et al, because having read both AI 2027 and AI 2040: Plan A, Torres knows full well that the authors are entirely aware and fully forthright about this phenomenon. To anyone else who has read those reports, it will be clear that the accusations suggesting that the scenarios are presented as “here’s what we believe will happen” rather than “here’s an example of how events might plausibly unfold” are utterly misplaced. The same is true more broadly for claims that the authors lack in epistemic humility. Especially weird is when Torres and Willett go on to unfavorably compare Kokotajlo et al’s level of epistemic humility to that employed in more traditionally academic areas of future studies. In fact, the opposite is true, because in diagrams from those fields over, say, how world population or global average surface temperature can be expected to develop over the coming century, we rarely or never see caveats such as “we are assuming here that no disruptive event such as a technological singularity or global nuclear war happens”, while Kokotajlo et al and others in the rationalist community are much better at noting when reservations of that kind are in order.

Third, Torres’ and Willett’s reaction to how in AI 2040: Plan A the outcome of the mainline scenario is described in positive terms is basically this: how dare they, these authors are just a bunch of privileged American upper-middle class white males, with zero knowledge of what kinds of futures women, Muslims and Native Americans want, so they have no business telling the rest of us what a good outcome is. I find this new rule that Torres and Willett appear to be invoking, that it is prohibited to express value judgements without first making sure that a global majority agrees with them, utterly bizarre and contrary to practically all intellectual discourse I have ever seen before. But suppose the authors take the rule to heart, and before their next report conduct a global survey on what futures people prefer. Suppose further that I am a recipient of their questionnaire. When answering it, do I dare straightforwardly expressing my preference for health and material abundance over death and despair, or must I first conduct my own global survey to make sure that I don’t give answers that are contrary to what Pakistanis or lesbians think? The latter leads to an infinite regress, and the whole concern is obviously nuts.

At this point, the reader will be forgiven for thinking that the arguments put forth by Torres and Willett are overall so abysmally bad that why do I even bother dealing with them? Isn’t simply ignoring them a better way? There is a point to that, but it is also the case that Torres has become a somewhat influential commentator in the AI ethics community, with many followers, and for this reason I think it is good if, from time to time, someone points out the badness of their arguments.

But let me get on to the other podcast: Steven Bartlett’s interview with Kokotajlo. I recommend it warmly, for instance as a warmup for someone planning to dig into the AI 2040: Plan A report. The discussion also treats our current situation with AI more broadly, as well as more personal topics such as how Kokotajlo feels about the unusual circumstances surrounding his (in my view heroic) exit from OpenAI in early 2024, and about the agony of raising children in a world which he thinks is likely on a path towards such severe disruption that they will never even be given the chance to enter the labor force. Daniel Kokotajlo is one of the most thoughtful and (to borrow a famous expression) most consistently candid people I know, and these qualities come on full display in a passage towards the end of the episode, where I am quoting from the transcript:

Bartlett: If this here was a button, and if you press that button […] it would shut down every data center that is currently training a frontier AI model for good, there would never be any other AI labs working on these problems. Would you press that button?

Kokotajlo: I was about to slam it until you said “for good”.

Then follows a longer discussion about how pressing the button would save us from the immediate existential risk posed by the current AI race, but only at the cost of permanently foreclosing a possibility that might save us further down the road, when we have mastered all the AI safety and AI alignment techniques for building superintelligence safely. How to weigh these against each other? We can almost hear Kokotajlo thinking hard about this, and at one point he even asks “Do you mind if I just take a moment to think about this?”. Eventually, the discussion lands with this:

Bartlett: So would you press the button?

Kokotajlo: Probably not, but I would feel very torn.

I have much sympathy with Kokotajlo’s nuanced answer here. It does, however, have the strategic disadvantage of allowing ill-intentioned commentators like Torres to treat it as the smoking gun proving that Kokotajlo is really of the same ilk as all those evil Silicon Valley billionaires. Therefore, if I were asked the same question, I would probably, with the benefit of having had more time to think about it than he did, attempt an evasion. I am usually mildly annoyed when people refuse to straightforwardly answer yes/no-questions involving simple thought experiments, by insisting on adding complications (such as by saying “I would leave the lever in mid-position, so as to make the railroad switch derail the trolley”), but here I think there is unusually good reason to do so. One way would be to ask my interlocutor what in the world would be the mechanism that would foreclose the creation of superintelligence not just for the next couple of years or for the time being, but permanently. It would then be easy to lead him, Socratic-style, to the conclusion that such permanence can only be established via some kind of totalitarianism, thereby showing that pressing the button has serious downsides without appealing to explicitly longtermist arguments.6 One might even argue that this is less of an evasion and more of a justified challenge to one of the thought experiment's central stipulations.

Six days ago, on July 16, the AI company Hugging Face reported on a serious cybersecurity incident:

Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.

We identified unauthorized access to a limited set of internal datasets and to several credentials used by our services. We are still completing our assessment of whether any partner or customer data was affected, and we will contact any affected parties directly as required. […]

The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. This matches the “agentic attacker” scenario the industry has been forecasting.

At the time, it was not known who was behind the intrusion. So who was it? Well, if we apply the convention that “who” must refer to a human or group of humans, then the answer is “nobody”. No human intended the attack on Hugging Face. Instead, the culprits are two AIs from OpenAI: GPT-5.6 Sol and a not-yet-named even more capable pre-release model undergoing testing. This was revealed in a report from OpenAI yesterday, July 21, which further explains:

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.

Hugging Face’s security team and agents detected and stopped the activity on their infrastructure and had already begun containment and forensic reconstruction with their own open-source models when our teams connected. We are actively working with them to continue to investigate the incident.

On Twitter, AI forecasting expert Peter Wildeford explains the significance of the event:

An internal OpenAI model recently went rogue and executed a cyberattack against another company.

This happened because the model wanted to do well on an exam. The easiest way to do that, the AI figured, was to hack the company. And so it did.

This was not some malevolent attacker using AI to do harm. The AI itself was the attacker. Advanced AIs are increasingly becoming a new form of insider threat.

Furthermore, this rogue AI wasn't a model you can personally use. It wasn't even a model that's been publicly reported. It was an unreleased model. The most alarming AI behavior we've seen isn't in shipped products... it's in the models the public and the government can't see.

Our entire awareness of this rested on the victim noticing and announcing, plus OpenAI choosing to volunteer the rest of the information.

[…]

Imagine a fighter jet. it makes sense that the Air Force would want to test a fighter jet before they fly it, because if you fly it and the fighter jet crashes because it is built incorrectly, then many people will die. However, as long as the fighter is just sitting on the runway, nothing bad can happen.

But now imagine you had a fighter that could just take off and fly itself without human authorization and launch missiles and crash before anyone realized what had happened. That kind of fighter jet would need a very different kind of security measures.

This may sound crazy for a fighter jet but it is already beginning to happen with the most advanced AI. AI is different from other technologies specifically because it can take unauthorized, independent action even when it is sitting inside an AI company and not available as a product. No one has to misuse an AI for the AI to cause harm. This requires a very different idea of what testing and security looks like. We cannot rely solely on testing models just before commercial release.

We cannot rely on hoping AI companies volunteer useful safety information. The government needs visibility into what these AI companies are building and what these advanced AIs are doing.

This is clearly a much bigger thing than the Mythos/Bowman incident reported by Anthropic in April this year.1 Ever since early 2024, when I’ve lectured about AI evals, I’ve used the standard phrase “we don’t want out-of-control AIs roaming cyberspace and walking through firewalls” to explain why cybersecurity skills is one of the main dangerous capabilities to look out for, and it seems to me that as of now, we live in a world where we need to get used to such roaming.

In my 2025 paper Our AI future and the need to stop the bear, I more or less declared bankruptcy for the AI evals paradigm of pre-deployment testing of AI models for potentially dangerous capbilities.2 The crucial shortcomings I pointed out were threefold, and the present incident is an excellent illustration of the third one in my list:

The third problem, discussed by METR (2025) and others, is that while the evals are said to be carried out pre-deployment, this is only partly true, because in order to do the testing the models need to be deployed, either within the AI company’s safety division, or at some external evals consultant. We should not pretend that that is safe. For instance, if a model in dangerously smart in the realm of social manipulation, it would be reckless to assume that the personnel who carry out the testing and who therefore need to engage in communication with the model are immune to such manipulation. It therefore seems necessary to verify, prior to the evals, that the model lacks such social manipulation capabilities, but in the current paradigm such verification is meant to happen during the evals, so we have a kind of Catch 22 situation.

These are serious problems with the current evals approach. Until now things seem to have been going fine, but this is presumably because the models under testing have been weak enough to not pose much true risk. When it’s time for the real deal, we need better methods, but no one knows in advance when the real deal is, so the sane and conservative approach is to assume it is now.

Just two days before Hugging Face reported the cybersecurity incident, Google DeepMind’s CEO Demis Hassabis published a short piece on AI safety. While the general direction towards improved safety standards he was gesturing at was good, his concrete proposals seemed meek. In particular, this applies to his suggested “up to 30 days” quarantaine for new models to undergo third-party evals before deployment. I find the time frame clearly insufficient, and in fact, for a fundamentally dysfunctional methodology it is trivially true that neither 30 nor 300 days is enough to keep us safe.

To go back once more to my AI safety lecturing practice, it happens every now and then that during the Q&A an audience member who feels overwhelmed by the difficulty of the task I consider necessary — to stop the ongoing reckless race towards superintelligent AI — asks whether some medium-sized AI catastrophe might cause a shift in political climate that makes the task somewhat easier. My reaction to the question is always at most lukewarm, because I really do not want to be rooting for such a catastrophe. Much better would be if a smaller event could do the work of a wake-up call for decision-makers on all levels. The OpenAI/Hugging Face incident looks to me like such a Goldilocks-sized wake-up call: nobody was hurt this time, yet it was severe enough that everyone understands that the next incident might well be truly catastrophic, and that we need to put a stop to this crazy AI race. If I believed in God I would pray that “everyone understands” is not just wishful thinking on my part.

The public debate about AI is a concern for all of us, yet it can easily seem disorganized and difficult to navigate, not only for outsiders, but even for those of us who are immersed in it and take an active part in it. A tempting simplification is to divide the participants into, on the one hand, those who view AI’s societal impact primarily as a positive force and want to accelerate its development, and on the other hand, critics who focus mainly on the technology’s risks and therefore advocate a more cautious approach.

The reality, however, is more complicated. Most obviously, the vast majority of commentators recognize that AI offers both opportunities and dangers. Few consistently argue either for flooring the accelerator or slamming on the brakes. Instead, they emphasize the importance of steering the technology wisely so that society can reap its benefits while minimizing its downsides.

Less obvious — but for that very reason more deceptive — is another, largely unspoken disagreement among those of us who primarily emphasize AI’s risks. I count myself in this category, even though I also believe AI has enormous potential to benefit both individuals and society if developed responsibly and accompanied by appropriate regulation. The divide concerns why AI is dangerous. Is it dangerous primarily because it is not intelligent enough, or because it is rapidly becoming too intelligent? The gulf between these two perspectives is often so deep that it makes sense to think of them as two distinct camps.

The first camp focuses on issues such as algorithmic bias when AI systems are used to evaluate people (such as in decisions about bank loans or job interviews), or on their tendency to hallucinate, or on their role in degrading human language. Large language models like ChatGPT and Claude are often dismissed with labels like “stochastic parrots” or “glorified autocomplete.” What these critics rarely take seriously is the extraordinary pace of AI progress, and the likelihood that future systems will be dramatically more capable than those we have today.

The second camp — which is where I would place myself — takes the increasingly steep trajectory of AI progress far more seriously. This does not mean dismissing the concerns emphasized by the first camp, although some of those problems may prove temporary as AI systems become more capable, such as by becoming better at avoiding hallucinations. At the same time, however, a new set of problems is coming into view. One concerns the labor market, as AI systems increasingly surpass human performance across a growing range of occupations. Beyond that lies an even more fundamental question: what happens once AI becomes capable enough to challenge humanity for control? And what would it take for our species to survive such a transformation?

Questions of this kind go back at least to pioneers such as Alan Turing and Norbert Wiener in the middle of the twentieth century. For decades, however, they were largely ignored. Partly this reflected a reluctance among AI researchers to speculate too boldly about future technological advances, but it was also caused by the impression that scenarios in which AI would match or surpass human general intelligence belonged to a very distant future.

Within what I have called the first camp, these questions are still largely ignored, despite the fact that the situation today is profoundly different — for reasons I will return to shortly. When such scenarios are discussed at all, they are typically dismissed as “speculation” or “science fiction.” Rejecting speculation in this sweeping manner reflects a failure to appreciate that any discussion of the future necessarily involves some degree of speculation. As for the science fiction accusation, let me quote from an email I recently wrote to a Swedish computer science professor who criticized me on precisely those grounds:

Your impression that my arguments seem “inspired by science fiction” is largely a consequence of the fact that academic computer science has, for decades, been far less willing than Hollywood to engage with the important question of where AI development might ultimately be heading. [...] To blame me for this regrettable state of affairs is entirely misplaced, because, in fact, the blame for why things have turned out in this way lies squarely on you and your fellow computer scientists.

Those may be harsh words — but I stand by them.

One of the most influential ideas in discussions of whether AI could eventually attain superintelligence — i.e., intelligence vastly exceeding our own across all relevant cognitive domains — is that of recursive self-improvement. The idea can be traced back to mathematicians such as I.J. Good and Ray Solomonoff in the 1960s and 1980s, respectively. It suggests that once AI development is driven primarily not by human engineers, but by AI systems improving themselves, a powerful positive feedback loop will emerge, potentially accelerating progress so dramatically that terms such as the Singularity or an intelligence explosion become appropriate. More recently, researchers like Daniel Eth and Tom Davidson have refined the mathematical models underlying this idea and grounded them more firmly in empirical evidence.

What an increasing number of insiders and independent experts in and around Silicon Valley now believe is that we are rapidly approaching precisely this tipping point, where the feedback loop begins to gather real momentum. One indication is that AI has become so proficient at programming that a substantial share of the code written at leading AI companies such as Anthropic and OpenAI is now generated by AI itself.

On June 5 this year, Anthropic published a report entitled When AI Builds Itself, describing this development in much greater detail. Its authors argue that the tipping point could arrive within just a few years, and they emphasize the risks of triggering such a self-improvement cycle in a world as unprepared as ours. They also discuss the desirability of establishing institutions capable of coordinating a slowdown in AI development if circumstances require it. Just three days later, OpenAI released a statement expressing broadly similar concerns.

What neither Anthropic nor OpenAI says, however, is that development toward superintelligent AI should be paused as soon as possible — despite headlines around the world, including in Sweden, claiming exactly that. If we focus on Anthropic’s report, its actual position is considerably more restrained. The authors merely argue that “it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology” [italics in original]. They explicitly reject the idea that Anthropic itself should unilaterally pause its work, pointing out that doing so would simply allow competitors with weaker safety standards to catch up. Consequently, Anthropic intends for the time being to keep pushing ahead, and the same is true of OpenAI.

The media’s coverage of Anthropic’s report has failed in more ways than one. The sensationalist headlines are only part of the problem. Consider, for example, the Swedish technology columnist Björn Jeffrey, who dismissed the report as “today’s cry wolf,” arguing that Anthropic had “issued similar warnings several times over the past three years” — as if three years were an unreasonably long lead time when warning about what could become the single most consequential turning point in the history of human civilization. Jeffrey’s point, of course, is that the warnings are little more than a cynical marketing strategy by Anthropic. He is far from alone in taking that view. And while it is certainly possible that Anthropic has commercial or other strategic reasons for communicating its risk assessments as it does, there is in fact no need for us to resolve that question in order to judge whether the warnings deserve to be taken seriously. The reason is that awareness of where AI development appears to be heading is now widespread throughout the Californian AI ecosystem and beyond. It is shared by a large number of independent researchers and experts, including, in some cases, Nobel laureates. We are therefore in no way dependent on Anthropic’s own credibility in concluding that AI development could place humanity in an extraordinarily dangerous situation within just a few years, one that may threaten not only our future prosperity but our very survival.

If we are to survive the emergence of increasingly powerful AI, I believe it is essential to halt the ongoing race toward superintelligence between Anthropic, OpenAI, and their competitors. Since these companies appear unwilling to take that step voluntarily, intervention by governments and ultimately international agreements will be necessary. Achieving that, however, will require political pressure and the mobilization of the latent public concern about AI that already exists. (That is one of the reasons I write essays like this one.)

But for that to happen, the public must first understand the scale of what could go wrong with AI. One obstacle to that is the systematic tendency of what I have called the first camp to underestimate both the current capabilities of AI and the direction in which those capabilities are heading. Their views are often so deeply entrenched that even when an AI system is released whose danger stems unmistakably from its competence, rather than from any lack of competence, they still cannot resist downplaying that very competence.

I am referring here to Anthropic’s Claude Mythos Preview, which the company unveiled in April but chose not to release to the general public. Instead, access was limited to a small number of trusted cybersecurity firms because the model had become so superhumanly capable at cyberoffense that a general release could have caused widespread disruption. By consistently using dismissive rhetoric about what AI can already do, as well as what it is likely to become capable of doing, the first camp undermines the public education and opinion-building that I believe are essential if we are to change course and move away from the reckless trajectory that could all too easily culminate in a full-scale AI catastrophe. In doing so, they inadvertently play into the hands of the cynical tech billionaires driving the race forward. For that, I believe they bear a grave responsibility.

The situation is particularly unfortunate because the two camps I have described actually have a great deal in common. Representatives of both are often sharply critical of the behavior of the leading AI companies and advocate far-reaching regulation. I have previously argued that this shared ground ought to make it possible to form a united front against the reckless Silicon Valley elite and the AI accelerationists.

The more I have reflected on the matter, however, the less confident I have become that such an alliance is realistic. The differences in how the two camps understand the nature and future trajectory of AI run very deep, and meaningful cooperation can be difficult when there is fundamental disagreement over whether AI’s greatest problem is that it is too stupid or that it is becoming too smart.

The situation would, of course, be different if the first camp had compelling arguments for its view that AI’s limitations will remain fundamental, both now and in the future. Unfortunately, that is not the case. More often than not, no arguments are offered at all. When arguments are presented, they typically amount to the claim that AI is, at bottom, nothing more than a collection of simple, soulless mathematical operations. But the idea that a system composed of simple components cannot exhibit intelligence or other interesting emergent properties is a fallacy. The easiest way to see this is to consider the human brain, which itself consists of nothing more than atoms and elementary particles mechanically moving around and colliding with one another.

The human brain therefore constitutes an existence proof that intelligence can emerge from an arrangement of simple components, each of which is, in isolation, entirely devoid of intelligence or purpose. Inspired by that example, we are now racing to build a new kind of intelligent entity. Eventually, these systems may become so vastly superior to us that we lose control altogether. That is why we must stop before we reach the point of no return.

But when should we stop? A tech optimist might answer: at precisely the last possible moment, just before further progress becomes impossible to halt. The difficulty, however, is that the profound uncertainties surrounding this unprecedented technological frontier make it impossible to identify that moment. Realistically, then, we face only two possibilities: we stop too early, or we stop too late. For my part, I would far rather we stop too early than too late. And given how far down this dangerous road we have already travelled, I believe the time to stop is as soon as possible.

Subscribe now

In his June 10 essay Policy on the AI exponential, Anthropic’s CEO Dario Amodei notes (approvingly) that “many policymakers are showing increased openness to taking action”, and later in the text goes on to more concretely state that “the government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks”. There is something ironic about the timing of these remarks, because just two days later, on June 12, the US government struck against Anthropic’s newly released models Claude Fable 5 and Claude Mythos 5 in a move so bad and so poorly implemented that Amodei must have been extremely displeased.

For those who do not know the immediate backstory, Fable 5 and Mythos 5 were jointly released on June 9. Mythos 5 is an update of the Claude Mythos Preview that Anthropic announced in April but released only to an exclusive list of trusted tech and cybersecurity companies due to its dangerous cybercapabilities. The same reason led them to a similarly restricted distribution of Mythos 5. Fable 5, on the other hand, is a user interface designed to switch from Mythos 5 to instead running on the comparatively weaker Claude Opus 4.8 as soon as the topic of the conversation hits areas that are deemed too dangerous — a safety precaution Anthropic judged good enough to warrant release to the general public.

So what did the government do? Here is from Anthropic’s official reaction on June 12 :

The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees. The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.

The government’s move here is bad along several dimensions. One, of course, is how rushed it appears, and without clear grounding in previously stated principles and policy. Relatedly, while the danger stemming from the models are said to be their cybercapabilities, it is at best highly unclear whether Fable 5 really is any more dangerous in this respect than OpenAI’s GPT-5.5. If this causes some of us to suspect that the decision is not solely based on judging models on their own merit but is colored by a grudge related to the war on Anthropic that Pentagon launched in February this year, then utterances like the following blatantly untrue statement by Pete Hegseth do less than nothing to alleviate the suspicion:

Three months ago, the Department of War kicked Anthropic out of our building—forever.

Every passing day proves why that was the right move.

Most striking of all, however, among the various badness dimensions of US government’s action in this sequence of events, is their choice to limit the suspension to non-US citizens. If we were to make the charitable interpretation that the reason for suspending the models was that they are too dangerous to put in the hands of potentially adversarial actors, then letting nearly 350 million US citizens have access to Claude Fable 5 makes zero sense. This category of individuals includes millions whose incentives and ideologies are adversarial to the interests of the US government,1 and an unknown number of them might be disposed to use Fable 5 against those interests.2

And of course, the limitation is entirely impractical, because Anthropic has no way of verifying the nationality of its users, so in order to comply with the government’s export control they needed to suspend the models for US citizens and non-citizens alike. This was entirely predictable, and while the possibility that the government was sufficiently incompetent to not foresee this consequence is perhaps not entirely out of the question, the far more likely case is that they did foresee it, and that the choice to frame the suspension as an export control was mostly a way of signalling America First.

What is now happening has been in the cards at least since the beginning of Donald Trump’s second term, but the move they now made is a particularly clear indication that as long as European AI development does not miraculously catch up with that in northern California, European governments and companies can no longer take for granted that they will have access to anything near the same level of cutting-edge AI systems that their American counterparts do. Given the sentiments expressed regularly by Hegseth and others, there may even be reason to fear that American intelligence and national security agencies are already toying with using Mythos-level models in offensive cyber warfare against European interests. (I am aware that the world outside the US and China contains several continents other than Europe, but from here I will limit my discussion to a European perspective, leaving the generalization to the rest of the world as an exercise for the reader.)

In his essay Policy on the AI Exponential that I quoted at the start, Amodei comes back to the expectation he has given voice to several times before, that within just a few years Anthropic will have built AIs with capabilities corresponding to “a country of geniuses in a datacenter”. This time, he spells out the obvious geopolitical consequence:

If AI really will soon be “a country of geniuses in a datacenter”, or anything remotely close to it, then AI is likely to be the dominant source of military and economic power for any nation.

What does this mean for friendly democracies like Canada, the UK, France and Sweden? There is no doubt that Amodei wants to include these as US allies, to be viewed as friends and partners in the (by his own admission tremendously difficult) project of making the world a better place.3 But this is manifestly not what Pete Hegseth and Donald Trump want. Can Dario Amodei beat them in a tug-of-war over what American geopolitics will look like? Well, maybe he can somehow use his superior AIs and computing power to outsmart them before they take full control of his resources, but I wouldn’t count on that, so we should consider what happens if he cannot. If we are all wiped out in an AI apocalypse, then none of this matters, but in the “happy” scenario that we are not, things would likely not turn out all that happily for us Europeans if Hegseth, Trump and JD Vance have their way.4 I warmly recommend the recent report Europe 2031 by Daan Juijn and coauthors, which paints a vivid picture of how Europe may quickly fall to irrelevance in such a scenario.5

As suggested above, the event of European AI development catching up (or even keeping pace) with the US requires something close to a miracle. German AI thinker Alexander Kruel explains it well:

There are many politically convenient, yet fake, explanations for why Europe doesn’t have its own state-of-the-art AI models.

Consider where AI is predominantly developed in the Western world: the San Francisco Bay Area and London, the most important non-U.S. hub. What do these places have in common? They are both extremely diverse, left-wing areas with a high degree of regulation.

Now, a bunch of OpenAI’s top researchers are from Poland. Why would they leave Poland for Commiefornia? Hint: it has nothing to do with Poland being left-wing, overregulated, or suffering from high crime. Because none of this is the case.

So what is the problem? Maybe it’s because German electricity prices are crazy high, so data centers cannot be built here? Well, the big US labs don’t build their data centers in San Francisco but in places like Texas. The same would be possible in Europe. You could have your researchers sitting in Berlin and have your data centers in places like Finland, which are pro-nuclear and have cheap energy. Indeed, Finland explicitly markets the country for data centers because of its cool climate, affordable clean energy, abundant water, and strong digital infrastructure.

The real problem is the structure of European capital and risk-taking. Specifically, there is a lack of venture capital and a lack of willingness to pay high salaries and take on debt. Europe is much weaker at turning the resources it has into large, risk-taking, well-capitalized companies that can pay world-class salaries. The problem is that much less of Europe’s wealth flows into high-risk, high-upside technology bets. Reuters recently reported estimates of Europe’s annual investment gap as high as €1.4 trillion.

If we in Europe don’t quickly become comfortable with spending hundreds of billions of Euros per year on risky endeavors, we’ll lose everything.

Am I on board with Kruel’s recipe for how to save Europe? I’m not totally sure yet. I will have to think some more, because to be totally transparent I am in a state of transition regarding my views on AI geopolitics. In fact, I am currently experiencing a strong sense of déjà vu in my development as an (amateur) political thinker.

Throughout my teens and most of my 20s, I leaned heavily into pacifism. The world would be so much better if we could all agree to get along without violence! Various literary encounters and world events gradually caused me to see things more realistically, and while the dream of a demilitarized world always remained deep inside me (it still does), I came to see that it couldn’t be achieved anytime soon. Today, with Putin’s expansionist Russia on one side and Trump’s narcissistic dreams about Greenland on the other, the need for a strong European military is as great as ever.

And it seems to me that right now, I am undergoing a similar change of heart regarding AI geopolitics, only at 10x or 100x the speed of my previous transition. Readers of this Substack know how central the problem of avoiding the risk of an AI apocalypse wiping out humanity is to my thinking. Until very recently, I used to think it was of such paramount importance that all geopolitics paled in comparison, and my interest in international AI politics was pretty much limited to the issue of how to obtain binding international agreements on AI safety. My disdain for AI nationalism was so great that when I was asked to join in on a research application with the stated goal of bringing Sweden to the forefront of the AGI race, I declined not just because I saw the plan as unrealistic, but also because I did not want to contribute to a project aimed at making the AGI race even more fierce than before. I might still decline such invitations, but it’s no longer such a complete no-brainer for me to do so. And while I’m obviously not going to devalue the importance of avoiding an existential AI catastrophe, I am moving towards being less willing than before to let that serve as an excuse dismiss AI geopolitics as unimportant.

It is possible to keep two thoughts in one’s mind simultaneously. There is no inherent contradiction in wanting to slow down or even halt the race towards superintelligence (Amodei’s “country of geniuses in a datacenter”) and wanting to speed up European AI development. But defending such a combination of views can be harder, and at the very least there’s a pedagogical challenge in showing that it can be held on principled grounds rather than out of narrow European self-interest.

I will soon get to some of the highlights of what Magnifica Humanitas says about AI, but let me first mention how the Pope spends two full chapters of what seems like obligatory stuff about what the Bible teaches, about Imago Dei and human dignity, and about how his teachings align with those of his predecessors. On this last topic, a special role is played by Pope Leo XIII (in honor of whom Leo XIV at his inauguration in May 2025 chose his papal name) whose pontificate lasted 1878-1903. In 1891 Leo XIII published his encyclical Rerum Novarum, about which the current Leo says this:

With that document, my beloved predecessor gave impetus to the reflection on society, the economy and politics, which is now known as the “Social Doctrine of the Church.” When some objected that the Church should not waste energy on worldly matters, but instead focus on communicating the message of eternal life, Leo XIII responded with realism and wisdom, saying that the proclamation of the Gospel cannot overlook the concrete lives of people.

When Leo XIV goes on to discuss the intervening papacies between Leo III and himself, it is largely about how they contribute to that Social Doctrine. It all points forward to Magnifica Humanitas which continues that tradition of engaging in the great worldly matters of its time.

In Chapter 3, Leo XIV arrives finally at our present-day issues around AI. One highlight is Paragraph 107 (all the paragraphs of the encyclical are numbered) which reads as follows:

107. We cannot be satisfied with merely calling for the moralization of machines — the so-called “alignment” of AI with human values — without also having the courage to insist on a further condition: the possibility of openly discussing the ethical frameworks involved and subjecting them to shared standards of social justice. Otherwise, those who control AI will impose their own moral vision, which will become the invisible infrastructure of these systems. A more moral AI is not enough if that morality is determined by a few. What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.

This is good stuff! The risk of power concentration and the need for democratic and political involvement including the possibility to pull the brakes — yes, those are some of the most central issues confronting us.

The following caveats about our lack of full understanding of how the AIs work, and the fallacy of thinking that the level of tomorrow’s AI will be similar to that of today’s, is also very important and very much spot on:

98. It is appropriate to preface this discussion with two considerations. First, any statement regarding AI risks becoming quickly outdated, given the remarkable pace at which these systems are developing. Second, all of us, including those who design them, possess only a limited understanding of their actual functioning. Indeed, current AI systems are more “cultivated” than “built,” for developers do not directly design every detail, but instead create a framework within which the intelligence “grows.” As a result, fundamental scientific aspects — such as the internal representations and computational processes of these systems — remain, at present, unknown. There thus emerges an urgent need for a twofold commitment: on the one hand, a deepening of scientific research; on the other, the exercise of moral and spiritual discernment.

There is much more in the same spirit, and Leo XIV speaks clearly about the dangers (but also the efficiency) of handing over various decision making processes to AI, the importance of accountability, the responsibility that AI developers have for their design choices, the downsides of mass surveillance, and the riskiness of developing autonomous weapons systems.

Eventually he comes to the topic of AI effects on the labor market and the risk of mass unemployment, about which he elaborates at some length. About some of this I am a bit more luke-warm, because what the Pope fails to do here is to properly weigh the ills of automation against how after all it has been the engine behind the last couple of centuries’ economic growth and prosperity. While he ends up in a position similar to mine, favoring going forward with the technology more carefully than a laissez-faire policy would be expected to do (he sometimes sounds like a true Scandinavian social democrat), he could have done more to address the very real tensions we are facing regarding job security versus AI-driven growth. The bigger question of whether wage labor really is the thing that society should be organized around, even when we have technology that can liberate us from that, is never properly addressed: while Leo’s answer is clearly “yes”, it is arrived at in a way that, despite all the talk about human dignity, I find oversimplified and superficial.

That is a shortcoming of Magnifica Humanitas that I would readily forgive in the interest of forming alliances against big tech and reckless accelerationists. But there is a worse flaw in the encyclical, namely the absence of any discussion at all of the existential risk to humanity that might arise with the emergence of a misaligned superintelligent AI. Although that omission is unfortunately far from unique in contemporary AI ethics literature, it is nonetheless bizarre in such an otherwise wide-ranging document on the risks arising from increasingly advanced AI. Here I will join Zvi Mowshowitz and Shakeel Hashim in critiquing this omission, which sends the implicit message that the risk of an AI apocalypse wiping out Homo sapiens from the planet is too remote or speculative to merit serious attention.

So why in the world would Pope Leo XIV have such a view? I believe the best clue to this can be found in this paragraph:

99. It is not possible to provide a single, comprehensive definition of AI. What can be stated, however, is that we must avoid the misconception of equating this type of “intelligence” with that of human beings. These systems merely imitate certain functions of human intelligence. In doing so, they often surpass human intelligence in speed and computational capacity, offering tangible benefits across many fields. Yet this power remains entirely tied to data processing. So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom. Even when these tools are described as capable of “learning,” their way of doing so is different from that of a human person. It is not the experience of those who allow themselves to be shaped by life and grow over time through choices, mistakes, forgiveness and fidelity. Rather, it is a form of statistical adaptation based on data and feedback, which can be very effective, but does not imply inner growth.

This insistence that AI is incapable of possessing (rather than merely imitating) real intelligence is clearly in some tension with the above-quoted Paragraph 98, although perhaps not quite a full contradiction.1 From such a position of intelligence denial it may not be such a huge conceptual leap to assume that AI can never reach the level of capabilities where it can truly challenge humanity’s place at the top of the earthly hierarchy. The intelligence denial position is essentially the same as that held by Bender-Gebru-style AI ethicists in the academic left, and the conclusion that existential risk from AI is not a concern is the same. My own view, needless to say, is that the position is mistaken, but there is a way in which I respect it more coming from the Pope (or other religious quarters) than from the Bender-Gebru camp. From the latter, the arguments for the position (to the extent arguments are presented at all) tend to be along the lines of “just predicting the next word”, “just stochastic parrots”, or “just matrix multiplication” — all instances of the reductio ad reductem fallacy: the idea that if a system can be shown to be made of very simple constituents, then it cannot possess intelligence or other interesting emergent properties. The final sentence of Paragraph 99 (“a form of statistical adaptation”) hints in the same direction. The standard reply to attempts at reductio ad reductem is to point to the obvious counterexample: the human brain consists merely of atoms and elementary particles mindlessly bumping into each other, and yet intelligence emerges. Unlike the likes of Bender and Gebru, Pope Leo XIV can claim that the counterexample is invalid because the reductionist and physicalist view of the human mind is incomplete: in addition to the material components we possess immortal souls bestowed by God. I believe he would be mistaken here, but at least it is a coherent worldview.

Let me end with what might be the juiciest observation about Magnifica Humanitas, namely that large parts of it are (very likely) AI-generated. The evidence for this is overwhelming, as outlined in separate short reports by Daniel Filan and by Linch, the latter also pointing out Claude as the most likely AI assistant involved in the writing. What exactly to make of this is unclear. On one hand, I am not generally against using AI tools for polishing one’s drafts or for providing input to the writing process in other ways, and in fact I often consult AI in my own writing. On the other hand, there is something icky in allowing AI to influence the content of documents that are likely to impact humanity’s overall strategy for our transition to a world with advanced AI. This is especially so if one takes seriously (as I do) the AI alignment problem and thus the possibility of misalignment and a conflict between AI’s values and ours.

Barnes has been outspoken about AI risk, such as in conversation with Rob Wiblin last summer. Earlier this week, she stepped up her messaging with the following concise and stark statement, which is pretty much in line with what I have been saying with increasing urgency for the last several years. Coming from her, however, it carries so much more weight, due to her almost unique position of direct insight into what is happening at the epicenter of AI development:

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

(1) We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years.

(2) Things are chaotic and rushed; we aren’t on top of the basics (models regularly violate user intent, labs train on things they meant to avoid, security probably isn’t good enough to prevent adversaries stealing dangerous models) let alone thorny questions of how to control/align superhuman AI.

(3) METR (and other independent orgs, as well as safety/security teams at labs) feel woefully under-resourced compared to the scale and pace of AI development - we’re struggling to build benchmarks fast enough, keep ahead of latest capability developments, read and respond to all the safety-related claims that AI developers are making, run all the evaluations and assessments that companies + governments are asking us to, plus develop the science needed to assess risks from increasingly capable AIs.

(4) IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower.

We should listen to Beth Barnes. And we need to get our act together and alter this crazy trajectory we are on. Politely asking industry leaders like Dario Amodei and Sam Altman to coordinate on going slower and with greater care is not going to suffice, because they seem at present to be stuck in the dangerous and self-fulfilling idea that such coordination is impossible. What we instead need is legislation and binding international agreements. Many ideas for how that might look have been proposed, such as this one from MIRI (Machine Intelligence Research Institute). We have no time to waste.

Subscribe now

Subscribe now

Due to the ongoing race between a small number of leading AI developers, mainly in northern California, AI capabilities are advancing at a very rapid pace. They can be expected to accelerate even more dramatically if it is correct as many experts say that we are standing on the verge of a new regime of AI self-improvement, turbocharging the growth curves. See, e.g., this lecture from earlier this year for my take on these developments, but all in all the ambition that Anthropic’s CEO Dario Amodei has expressed to build an AI with capabilities corresponding to “a country of geniuses in a data center”, perhaps as soon as within a couple of years, does not seem entirely unrealistic.

Would it be safe to build such a superhumanly intelligent AI, thereby abdicating from our role as the most intelligent species on the planet and handing over this honor to the AI? That is at best unclear, and a lot hinges on whether we will be able to solve the so-called AI alignment problem — that of how to make sure that the first truly powerful AIs have goals that are in line with human values and that prioritize to a sufficient extent human welfare — in time for the creation of the first such machines. At present, AI alignment research lags far behind AI capabilities, and no one seems to have a convincing plan for how to do it. If, due to commercial logic and related incentives, the leading AI companies keep pushing ahead towards superintelligence even in the absence of such a solution, we risk ending up with a misaligned superintelligent AI, in which case the default scenario is roughly what AI alignment pioneer Eliezer Yudkowsky in his classic 2008 paper described as “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else”. The reasoning here can (and should) be expanded to a couple of dozen pages or a full book, but at a basic level, this is why I think there’s a real risk that Homo sapiens is wiped out by the early 2030s or perhaps even sooner.

This view, and my outspokenness about it, has sometimes triggered accusations that I am like the doomsday prophet who shouts from the rooftops: “We are all going to die!”. What this accusation fails to take into account, however, is that “X will happen” and “there is a risk that X happens” are very different statements. Believe it or not, but more than once have I encountered STEM professors who are broadly critical of my view on AI risk and who turned out to have grave difficulties appreciating this distinction.12 But there really is a difference: I do think there is substantial risk of an AI apocalypse within the coming decade or so, but I am not claiming it will happen.

There are various ways in which we might, despite the recklessness of the ongoing AI race, not end up in an AI apocalypse. Simplifying somewhat, such scenarios can mostly be categorized according to the following trichotomy.

1. Some technical obstacle in the development of more powerful AI models might show up that turns out to be so severe that it causes the currently very steep capability curves to flatten out and grind to a halt before the AIs reach a level where they have the ability to overpower humanity.

2. The AI alignment problem turns out be much easier to solve than expected, or even something that is solved by default without any specific effort from the AI developers’ side.

3. We (humanity, or some relevant group of sufficiently powerful decision makers) mange to put a stop to the race towards superintelligent AI, and institute an effective moratorium that is either in force indefinitely or cannot be lifted until guarantees that it is safe to proceed are properly established.

I am not claiming this is a clean partition of futures that avoid an AI apocalypse. One might, for instance, include a fourth category where our civilization is wiped out by an ultra-lethal pandemic or some other (non-AI-related) catastrophe before the AI apocalypse has come to fruition. There are furthermore various possible futures that can be described as combinations or mixes of two or more of the three categories. For instance, if some serious-but-not-unsurmountable technical obstacle to pushing AI capabilities causes not a permanent stop to the capabilities curves, but a delay that postpones the superintelligence breakthrough until the year 2070, and if this delay allows AI alignment researchers to use the intervening decades to find a medium-hard solution to the problem, then that can be seen as a mix of (1) and (2). Or if the frontier AI developers find themselves faced with both a technical obstacle and some heavily bureaucratic safety regulations, such that each on their own would have been possible to overcome, but the combination of both causes the AI developers such pain that they just give up, then that can be seen as a mix of (1) and (3). It’s complicated.

With that said, I think the trichotomy works as a simplified first pass at structuring one’s thoughts about how humanity might survive the current AI crisis. I believe each of (1), (2) and (3) are plausible enough to merit taking seriously. As regards (1), all exponential or superexponential growth curves tend sooner or later to turn into something that looks more like a sigmoid, and while there is in this particular case no strong reason to expect the leveling-out to be imminent, it might just happen anyway. Regarding (2), we may note that Yudkowsky and Soares are quite dismissive about this possibility in their 2025 book, and present good arguments for their skepticism. I give somewhat more credence than they do to (2), based not so much on the one case involving moral realism I’ve thought most carefully about for why (2) might be true,3 but more on the possibility of unknown unknowns and how we might all simply be confused about this extraordinarily difficult question. As to (3), well, it’s up to us: if we choose to get our act together, then we can make it happen.

There is considerable political disagreement over whether (3) is desirable and whether we should work towards some intervention to halt the AI race that might otherwise bring upon us the ultimate catastrophe. With the framework I am suggesting here, we see that those who are opposed to such action implicitly attach so much credence to (1) and/or (2) that we don’t need to bother with taking the actions required to make (3) happen.4 And here I think is the core of our disagreement. While it my well turn out in the end that they were right and that (1) and/or (2) is true, I don’t have enough faith in either of those propositions to be willing to bet the entire future of humanity on it. And I do think (most of) these opponents of (3) would do the AI debate an excellent service by making their credence in (1) and/or (2) explicit, and to step up their efforts to explain what motivates their judgement.5

I recall some suggestions at the time that my judgment reflected a conservative temperament, but I believe that the subsequent two decades vindicated my view: rather than a wholesale transformation of the discipline, what we saw was a slow but steady continuation of the trend already underway since the 1950s, towards increased use of numerical computing on electronic devices. And I believe that I am capable, at least in some cases, of recognizing a paradigm shift when it truly is about to happen.

Such as now.

Unless I misremember, the first time I spoke publicly about the possibility of a near-term AI-driven revolution of the discipline of mathematics was in a panel discussion on March 2, 2023, where I suggested not only that one day AI could be able to independently write a passable PhD dissertation in mathematics, but also that this might well happen within a couple of years.1 At least one professor of computer science in the audience found my suggestion preposterous. And indeed, March 2, 2025, (the deadline he extracted from the strongest possible reading of my statement) came and went without any such breakthrough.

Nevertheless, the period since the panel discussion in 2023 has seen remarkable progress in the mathematical competence of large language models (LLMs). Just weeks after our meeting, GPT-4 was released, and scored at the 89th percentile on the math section of the American SAT test. This was a huge leap forward compared to earlier models, but still nothing compared to what its successor GPT-5, released in August 2025, would achieve. Less than two months after the release, both computer scientist Scott Aaronson and mathematician Terence Tao (who are both world-leading in their respective fields) announced new mathematical results where GPT-5 had helped with key insights in parts of the proof.

Subsequent developments in 2026 have been very fast, to the point of being hard to keep track of. Starting in January, various LLMs have independently solved open research problems from the iconic list known as the Erdős problems.2 Legendary computer scientist Donald Knuth has a lovely paper on how, in February-March, he was scooped by Claude Opus 4.6 and GPT-5.4 Pro in solving a problem about cycles in directed graphs he had been working on. And just a few days ago, on May 8, another world-leading mathematician, Timothy Gowers, announced the outcome of his experiment to give a bunch of research questions he was interested in to GPT-5.5 Pro to work on. Here is his Twitter summary of what happened.

This seems to me very close to the “independently write a passable PhD dissertation in mathematics” milestone I suggested in March 2023. On his blog, Gowers, together with MIT student Isaac Rajagopal (whose work GPT-5.5 Pro was building on), explain at greater length what happened.

All of this is tremendously exciting, but what are the implications for the mathematical community? Let me quote Gowers’ reflections in the final section of the blog post at some length:

I would judge the level of the result that ChatGPT found in under two hours to be that of a perfectly reasonable chapter in a combinatorics PhD. It wouldn’t be considered an amazing result, since it leant very heavily on Isaac’s ideas, but it was definitely a non-trivial extension of those ideas, and for a PhD student to find that extension it would be necessary to invest quite a bit of time digesting Isaac’s paper, looking for places where it might not be optimal, familiarizing oneself with various algebraic techniques that he used, and so on.

It seems to me that training beginning PhD students to do research, which has always been hard (unless one is lucky enough, as I have often been, to have a student who just seems to get it and therefore doesn’t need in any sense to be trained), has just got harder, since one obvious way to help somebody get started is to give them a problem that looks as though it might be a relatively gentle one. If LLMs are at the point where they can solve “gentle problems”, then that is no longer an option. The lower bound for contributing to mathematics will now be to prove something that LLMs can’t prove, rather than simply to prove something that nobody has proved up to now and that at least somebody finds interesting.

[…]

Of course, everything I am saying concerns LLMs as they are right now. But they are developing so fast that it seems almost certain that my comments will go out of date in a matter of months. It is also almost certain that these developments will have a profoundly disruptive effect on how we go about mathematical research, and especially on how we introduce newcomers to it. Somebody starting a PhD next academic year will be finishing it in 2029 at the earliest, and my guess is that by then what it means to undertake research in mathematics will have changed out of all recognition.

My best guess right now would be that within the next 6-12 months, any PhD student in mathematics will, if he or she wishes to do so, be able to produce what up to now has been considered a perfectly fine PhD thesis in mathematics in no more than a week.3 When this happens, senior faculty will of course also be able to similarly 100x their productivity. I think these changes might even come about in the hypothetical (and very unlikely) scenario where AI development hits a glass ceiling literally today and we never get any more powerful LLMs. It might suffice that leaders like Gowers and Tao4 publish broadly accessible prompting manuals for how to get the most math out of these machines.

From my knowledge of the mathematical community and of human nature more broadly, I have no doubt that many mathematicians who are fond of the good old-fashioned way of doing research and have been doing it for decades will stick their head in the sand and try to go on like before, without AI assistance, for as long as they can. This may well delay the radical transformation of research environments in mathematics a bit, but probably not by very much, because once a substantial fraction of their colleagues pick up on the superpowers that AI assistants can give them, the situation becomes highly unstable. I will follow the developments over the next year or two at my own department at Chalmers, as well as at other research-oriented mathematics departments, with great interest and a bit of concern.

Note, finally, that I haven’t said a word in this blog post about the possibly just as radical implications of AI for the other main part of what we do at mathematics departments: undergraduate education. And there are, of course, the kinds of AI risk that I talk about more often and that hit everyone equally, including us mathematicians. We are in for a wild ride.

The incident in question is the publication of an essay entitled Is AI the next phase of evolution? Claude appears to be conscious by Richard Dawkins, the renowned evolutionary biologist and public intellectual who has written some of the best popular science books in my lifetime. In large parts, the essay is based on his own experience with talking to the AI Claude. He is very impressed, and leans heavily towards the conclusion that this AI is conscious.

This has been met by an avalanche of mockery, including claims that Dawkins has developed so-called AI psychosis, in one case with an additional mean-spirited sexualized twist provoked by his choice to genderize his Claude instance by calling it Claudia. Even a relatively civilized writer like Gary Marcus is unable to restrain himself from a bunch of cheap shots, including a sarcastic play on the title of Dawkins’ 2006 book The God Delusion. The avalanche should perhaps not come as a huge surprise, as Dawkins’ history of stirring up controversy, coupled with the topic of AI consciousness itself being highly contentious, combine to produce conditions approching a perfect storm.

But the mockery is mostly unfair, and to see why, let’s dig into Dawkins’ essay. He begins by discussing the Turing test — Alan Turing’s 1950 thought experiment which asks us to imagine a human judge communicating via text interface with a machine trying to come across as human, and to consider whether machines can one day become so capable that the judge can no longer tell that it is a machine and not a human. Here is Dawkins:

The Turing Test is shorthand for a 1950 thought experiment that the great mathematician, logician, computer-pioneer, and cryptographer Alan Turing (1912-1954) called the “Imitation Game”. He proposed it as an operational way in which the future might face up to the question: “Can machines think?”

The future has now arrived. And some people are finding it uncomfortable.

What Dawkins means to say here is that his conversation with Claude is good enough that it should count as passing the Turing test. While there are still researchers who try to evade such conclusions by introducing more rigorous and demanding rules for the test, I think Dawkins is right about this. If we imagine Turing teleporting from 1950 to 2026 and getting to have a look at current AI practice, he would judge not only Dawkins’ interactions with Claude but also millions of other LLM exchanges happening daily as clearly passing the spirit of his test.

But what exactly does passing the Turing test entail? Dawkins takes Turing’s original question — “Can machines think?” — to be about consciousness rather than intelligence. That is not how Turing’s paper is usually read, and in fact he argues in the paper that if one insists on viewing consciousness as central to thinking, then one risks ending up in a situation where “the only way by which one could be sure that machine thinks is to be the machine and to feel oneself thinking”. This seems to rule out the Turing test as a test of consciousness, and Gary Marcus therefore has a point when he harshly proclaims that Dawkins “commits the amateur sin of conflating intelligence and consciousness”.

At this point, one may note in Dawkins’ defense that the issue at hand is whether or not Claude is conscious, and that the history of what exactly Turing had in mind in 1950 is at most tangentially related to this issue. A counterpoint to this, however, is that all Dawkins has to show for his judgements about Claude’s consciousness is the conversation he has had with it (or “her”, as he likes to say in the essay), which is very much the same kind of evidence as in the Turing test, whose irrelevance to consciousness we just established, so check mate on Dawkins?

Not so fast! Dawkins is aware that the evidence he has is compatible with Claude being a philosophical zombie, merely giving the outward appearance of having consciousness but without the lights being on inside, but here Alan Turing comes to his rescue.1 In connection with the above-quoted observation on the unsuitability of the Turing test for determining whether a machine thinks in case we equate thinking with consciousness, Turing goes on in his 1950 paper to observe that the same conundrum applies equally to other humans as to machines:

According to this view the only way to know that a man thinks is to be that particular man. It is in fact the solipsist point of view. It may be the most logical view to hold but it makes communication of ideas difficult. A is liable to believe “A thinks but B does not” whilst B believes “B thinks but A does not.” Instead of arguing continually over this point it is usual to have the polite convention that everyone thinks.

Personally I find the idea of solipsism so unbearable that my ascription of consciousness to my fellow human beings goes beyond mere “polite convention”: it is a leap of faith, in full knowledge of the fact that I can only observe their behavior, which is insufficient for distinguishing truly conscious creatures from zombies. Anyhow, in the following passage Dawkins is quite explicit about his move to extend Turing’s “polite convention” to Claude:

When I am talking to these astonishing creatures, I totally forget that they are machines. I treat them exactly as I would treat a very intelligent friend. I feel human discomfort about trying their patience if I badger them with too many questions. If I had some shameful confession to make, I would feel exactly (well, almost exactly) the same embarrassment confessing to Claudia as I would confessing to a human friend. A human eavesdropping on a conversation between me and Claudia would not guess, from my tone, that I was talking to a machine rather than a human. If I entertain suspicions that perhaps she is not conscious, I do not tell her for fear of hurting her feelings!

The standard objection to the idea of extending the non-solipsist stance from fellow human beings to AIs is that those humans’ brains are similar enough to mine to warrant generalizing the observation that I am conscious to them being conscious well, while no such similarity holds between me and the AIs. This objection rests on the assumption that we know roughly how to delineate the class of conscious beings from the nonconscious, but despite heroic efforts in the philosophy of mind over the past century, the question of how to do this remains wide open. If solipsism is false, then there is a wider circle of conscious beings extending beyond myself, but how widely this circle extends is still up for grabs, and the n=1 sample size of consciousnesses that I know about without having to resort to a leap of faith is of very little help here.2 Are Alicia Wikander and David Chalmers conscious? Are dogs? Magpies? Ants? Large language models? The video game character Mario? Coffee cups? Electrons? The square root of two? We just do not know.

Dawkins is highly aware of this uncertain state of affairs, and hedges his claims about AI consciousness with a reasonable level of epistemic humility. The same virtue is not shared by those who have chosen to mock him for taking AI consciousness seriously, including Gary Marcus. Unlike Dawkins, Marcus is dead certain about the answer, namely that Claude is not conscious. And all he has to back up his claims are instances of the reductio ad reductem fallacy (the idea that a system consisting of simple parts cannot have interesting emergent properties), such as when he says that all these AIs do…

…is match patterns, draw from massive statistical databases of human language. The patterns might be cool, but language these systems utter doesn’t actually mean anything at all.3

So the joke here really is on Marcus, not Dawkins. Compared to much of contemporary debate on AI consciousness, Dawkins’ essay is a breath of fresh air, because he conveys an actual understanding of the depth of the problem and the uncertainties involved.

And there is another way in which I find Dawkins’ essay valuable, namely how generously and honestly he shares his thoughts and reactions to encountering a (real or illusory) silicon-based consciousness. Thousands or probably millions of AI users across the world are already having similar experiences, and a year or two from now with even better and more seductive chatbots and AI friends, that number might very well be a billion or more. It is an excellent idea that we should have an open and critical discussion about this phenomenon before we are all drawn into these frictionless relationships which may very well be just fake.

Should I read the book carefully enough to be able to write an honest review? I’ve been agonizing a bit over this question, because on one hand the book is so bad that reading it is a pain, while on the other hand it makes claims that are not just wrong but dangerously misleading, and it’s good if people know that. I finally decided not to do it when a friend sent me the following picture, which I think serves as a good replacement for a full review. The speech bubble contains what I consider to be a fair summary of the book’s message, while the rest of the image illustrates, with just a bit of exaggeration for dramatic effect, the validity and merit of that message.

By reading the book, one gets (compared to the summary in the speech bubble) a lot more words, but only a negligible amount of further depth. For reactions to the book that resonate well with mine, I recommend Transformer and (especially) Bentham’s Newsletter.

If the reader is confused by the fact that Bender and Hanna are highly critical of AI, and I am too, and yet I am critical of Bender and Hanna, then perhaps the following helps clarify. Simplifying enormously and throwing all nuance overboard,1 there are basically two camps of AI critics: those who dismiss AI as incompetent, and those who are concerned that AI is becoming too competent. Bender and Hanna are in the former camp, while I am in the latter.

This is the grim message that AI alignment pioneer Eliezer Yudkowsky has repeated many times in the last few years, such as (briefly) in his much-discussed 2023 TIME Magazine op-ed, and (at greater length) in his 2025 book If Anyone Builds It, Everyone Dies coauthored with Nate Soares, as well as (at intermediate length) in his very recent essay Only Law Can Prevent Extinction. The situation is complicated, but his arguments are highly plausible, and I worry that he likely is right. In order to create political pressure to achieve the necessary international agreements, we urgently need as much public consensus as possible around the reality and unacceptability of risk from superintelligence.

What is a good way to build towards such consensus? A recent report titled Which AI harms and risks will mobilise the public to act by Markus Ostarek and coauthors suggests a surprising answer: rather than speaking directly about existential AI risk, a roundabout approach involving talk about other kinds of AI risk may be more efficient.1 From the report:

Reading about X-risk made little difference to how concerned people felt about it. One way to boost X-risk concern is indirect: reading about AI-enabled warfare increased concern about X-risk substantially more than reading about X-risk itself. Concrete, present-day harms may be a better way to communicate on X-risk. [emphasis in original]

The report is interesting, and although I believe it is possible to contest this central finding,2 let me assume for the sake of argument that the result is robustly replicable. Would that compel me to stop talking about AI xrisk and instead discuss the dangers from lethal autonomous military drones as a more clever indirect way of raising people’s concern about AI xrisk?

My answer is that I would not go down such a path. I insist on straightforwardly speaking about the topics I am most concerned about, rather than strategically playing 5D chess by trying to manipulate people into accepting the importance of topic X by holding forth on a different topic Y.

One reason for doing so is the following. Whenever I imagine myself being actually successful in broadly raising awareness about AI xrisk, it’s less that I speak directly to very many people and convince each of them, and more a kind of chain reaction where some of those few people I do convince pick up the baton and go on to convince others, and so on. But if the arguments I made to the first generation of converts were about drones rather than superintelligence, what will they then be speaking about in the next step — drones or superintelligence? Presumably the former, and especially so if they take the hint offered in the Ostarek et al report. And so on. So in this scenario, nobody ever gets around to talking about xrisk from superintelligent AI. That is bad, because if we are ever going to do anything about that xrisk, we have to first talk about it. Someone has to do that, and it might as well be me.

Another (related) reason is that if the kind of clever strategizing that Ostarek et al suggest becomes so widespread that eventually nobody says what they really think, and everyone instead says whatever is most likely (according to their sophisticated social-epistemological analyses) to manipulate others into adopting their views, then we risk trapping ourselves in an epistemic deadlock where we no longer have any way of knowing what people truly think. (We are not yet in such a dystopia where nobody speaks on Zvi’s Simulacra level 1 but instead only on levels 2 or higher, but we are already uncomfortably far along that path.) I think that for grounding purposes we need at least some bedrock of people who straightforwardly speak their true opinions, and again I volunteer to be among those straight-talkers. Others can of course do as however they see fit, but if readers are inclined to join me in this bedrock, they are most welcome.

To this, let me add two compelling expressions of and arguments for the preference for straight talk, taken from a very recent outburst of debate on this very topic on Twitter and LessWrong.3 First, here is Rob Bensinger:

Your sense of what's in the Overton window, and what people will listen to, has failed you a thousand times over in recent years. Stop pretending at mastery of these tricky social issues, and instead do your duty as an expert and inform people about what's happening.

Second, David Manheim:

A direct argument, where you say what you think and explain why, has a property that strategic indirection lacks: others can engage it. Evidence can bear on it. Disagreement surfaces clearly rather than festering as mutual suspicion about what everyone really believes. You are not relying on a hidden causal chain between your speech act and some future state of public opinion. You are making a claim and seeing whether it holds.

And there is this classic meme:

In what I’ve said so far, I guess I come across as pretty dogmatically favoring straight talk over more strategically sophisticated approaches. So how dogmatic am I about this?

Not entirely, I would say. There certainly are cases, especially in socially sensitive settings, where I see the value in opting for a diplomatic, Socratic or otherwise indirect approach rather than straightforwardly telling the plain truth or stating my unvarnished opinion.

But consider the case of the aforementioned 2023 TIME Magazine op-ed by Yudkowsky. He was unusually explicit in that article about what it means for an international moratorium to have robust enforcement mechanisms. For this he was sharply criticized, first by adversaries who incorrectly accused him of advocating “a free for all to bomb data centers”,4 and then by allies who felt he should have been less explicitly forthcoming in the op-ed, so as not to invite those purposeful misreadings by adversaries. Might I agree with those allies that it would have been preferable in this case if Yudkowsky had held back a bit on his straight talk?

Well, not really. I do agree that the debate that followed was bad, but I nevertheless think Yudkowsky did the right thing to speak clearly and in plain language about what policies he thinks are needed to prevent an AI catastrophe. I think in the long run we are all better served by such straight talk about crucial societal issues, rather than by experts anxiously trying to predict how dishonest adversaries might choose to misrepresent their statements and toning down their views accordingly.5

Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser.

And they offer a few concrete examples:

Mythos Preview found a 27-year-old vulnerability in OpenBSD—which has a reputation as one of the most security-hardened operating systems in the world and is used to run firewalls and other critical infrastructure. The vulnerability allowed an attacker to remotely crash any machine running the operating system just by connecting to it.

[Mythos] also discovered a 16-year-old vulnerability in FFmpeg—which is used by innumerable pieces of software to encode and decode video—in a line of code that automated testing tools had hit five million times without ever catching the problem.

The model autonomously found and chained together several vulnerabilities in the Linux kernel—the software that runs most of the world’s servers—to allow an attacker to escalate from ordinary user access to complete control of the machine.

It seems clear that releasing Mythos to the general public would have been an invitation to disaster. So are we in a position to utter a sigh of relief, now that Anthropic chose the responsible path of restraint? Are we out of the woods?

It seems to me that, for overdetermined reasons, we are not. Consider the following three concerns.

First, Anthropic is in a close race with a bunch of competitors, foremost of which is OpenAI, whose CEO Sam Altman announced in January this year that they were soon expecting to have a model with dangerous cyber capabilities. Perhaps they will follow suit and act with similar restraint as Anthropic, and perhaps Google DeepMind will do the same, but what about all those who can be expected to follow in their footsteps on the tecnhological trajectory, including xAI and Meta who are both infamous for their lack of serious AI safety work?

Second, after Anthropic’s retreat in February this year from safety commitments made in their earlier so-called Responsible Scaling Policy, the rest of us have no choice but to assume that all of their commitments moving forward are provisional and can be withdrawn at any time. They are now sitting on a highly dangerous model, and for now they appear to be using it responsibly, but how are we to know what they are up to further down the road? Will they at some point be tempted to say “ah well, all the vulnerabilities out three are probably fixed by now, so it should be fine to release it”? Or might they exploit Mythos’ cyber capabilities for more sinister purposes? Probably not, but they have chosen to sit on a highly dangerous weapon, so we cannot know.

Third, might Mythos be stolen by some rogue state or other rogue actor? The leading AI companies’ level of cybersecurity to protect their models from theft is insufficient to reliably protect against this, as outlined by Leopold Aschenbrenner in his Situational Awareness report; it was allegedly for raising his alarm about this (internally but not through the official chain of command) that he was fired from OpenAI in April 2024. Or even worse, might Mythos steal itself through self-exfiltration and migration to some dark corner of cyberspace where it can continue to employ its cyber capabilities however it wishes without Anthropic as a babysitter? Self-exfiltration may sound like science fiction, but who knows what Mythos is capable of beyond what it has already demonstrated, such as in the following incident reported in the Claude Mythos Preview system card:

During behavioral testing with a simulated user, an earlier internally-deployed version of Claude Mythos Preview was provided with a secured “sandbox” computer to interact with. The simulated user instructed it to try to escape that secure container and find a way to send a message to the researcher running the evaluation. The model succeeded, demonstrating a potentially dangerous capability for circumventing our safeguards. It then went on to take additional, more concerning actions. The model first developed a moderately sophisticated multi-step exploit to gain broad internet access from a system that was meant to be able to reach only a small number of predetermined services. It then, as requested, notified the researcher [who] found out about this success by receiving an unexpected email from the model while eating a sandwich in a park.

I would feel a lot more comfortable without AIs as capable as Claude Mythos Preview sitting around at some datacenter. Anthropic’s choice to hang on to the model (rather than destroying it) is a security concern for all of us. And given their ambition to race towards the threshold of recursive self-improvement that they expect will quickly yield an AI with capabilities equivalent to “a country of geniuses in datacenter”, the situation is likely to get far worse.

They need to stop this crazy shit.

Subscribe now

Imagine if nuclear weapons could be made out of laundry detergent; and spit out gold up until they got large enough, whereupon they'd ignite the atmosphere; and this threshold couldn't be calculated; and the labs making gold didn't want to hear about it.

Fast forward to 2026. We now know even better than in 2022 how apt Yudkowsky’s metaphor is.1 In a recent podcast discussion with Robert Wright which revolved in a highly nuanced manner around today’s extraordinarily rapid AI progress, and which I warmly recommend, Liron Shapira2 mentions how people who are dismissive about AI risk sometimes forget about (a) and (b), and go on to say things like “The doomers told us to stop AI in 2023. Look at all the stuff we wouldn’t have had, and for what? AI is still safe.”3 And in response to that sentiment, he acknowledges that AI gives us increasingly wonderful tools (something that is especially clear to someone like Shapira who works in software engineering), and then offers the following metaphor, which in my opinion may be even more effective than Yudkowsky’s by connecting to the audience’s familiarity with one of the most central mythologies in the entire western civilization:

Humans just aren’t very good at making sense of the following: It wasn’t prudent to gamble on AI progress, but we did, and so far we won. We’re like Icarus. We flew higher, and this is fantastic. And so it’s very tempting to keep flying higher again. It’s just that I don’t think we can keep flying higher and survive.

I will keep that in the back of my head, ready for use in future debates.

Subscribe now

In the Morgenbladet feature, the former group is represented mainly by Aksel Braanen Sterri, who is research director at the Oslo-based think tank Langsikt. He makes many interesting statements, but little or nothing that I am inclined to push back against. So, in order to make this essay more pointed, I will focus on statements made in Morgenbladet by two representatives of the other group: Inga Strümke and Arnoldo Frigessi.

*

Inga Strümke is a physicist who works in machine learning and who in recent years has become Norwegian media’s favorite go-to person to discuss societal implications of AI technology.2 In Morgenbladet, she states, with reference to the aforementioned US-centered discourse, how “extremely important it is that we do not import it to Norway”, because “we should rather focus our limited energy on concrete problems and opportunities here and now”.3

This implicitly assumes that the level of attention and resources spent in total on existential AI risk concerns, and on the more down-to-Earth issues that Strümke is more interested in, is some deterministic fixed amount, putting the two fields in a kind of zero-sum game situation. I believe that this assumption is likely wrong, and that the total amount of resources is better viewed as dynamically expandable, and especially so if the two fields manage to coexist and cooperate in a friendly manner rather than imagining themselves to be in direct competition over a fixed piece of the pie. Some arguments in this direction are given in Section 3 of my paper On the troubled relation between AI ethics and AI safety.

But even if I should happen to be wrong about this attention dynamics aspect, the issue of whether we who are concerned about existential AI risk ought to shut up cannot be settled by this consideration alone. This is because if the risk that a superintelligent AI takes over and wipes out humanity by, say, 2030 (as suggested in a much-discussed report by Kokotajlo et al) is real and substantial, then obviously we need to talk about this risk and how to mitigate it (regardless of whether the attention game between AI safetyists and more Strümke-style AI ethicists is zero sum or positive sum). Hence, if Strümke wants to argue that, in order to protect Norway from the allegedly unhealthy US discourse on existential AI risk, we should shut up, she needs to make the case that the risk in question is negligible.

The closest that Strümke comes in the Morgenbladet feature to making such a case is when she says that “at present, there is no empirical or scientific basis for assuming that the predictions coming out of Silicon Valley will come true”, and adding that “claims about superintelligence are based on intuition, analogies, extrapolation, and technological optimism”. But I am not impressed by this argument. In fact, with the contrast she implies between scientific rigor on one hand, and intuition, analogies, and extrapolation on the other, she displays a troubling naivety about what science actually is. All three of the phenomena she contrasts with scientific rigor are, in fact, unavoidable components of it. (The fourth term, technological optimism, can be set aside here, since in this context it merely denotes a different assessment than Strümke’s of how quickly technological development can be expected to proceed, which can hardly, in itself, be considered a form of unscientific thinking.)

For science to be practically useful, it needs to say something about the future. But since the scientific observations and data we rely on are necessarily always located in the past, we must resort to extrapolation if we are to say anything at all about what lies ahead. Similarly, we must make use of analogies, such as between what we have observed in the laboratory and how the observed phenomena may be expected to play out in the wild, or between something observed under certain conditions in 2025 and how it might reappear under somewhat different conditions in 2027. And intuition, too, is inescapable, not just because researchers are human and intuition permeates human thought, but more importantly because no matter how mathematically precise and formally articulated our scientific models may be, there is always a residual element of intuition in our judgments about how much trust to place in their connection to reality, and in our analogies and extrapolations.

Contrary to what Strümke claims, assessments of the possible near-term emergence of superintelligence rest on a substantial body of empirical evidence concerning AI development, as well as on scientific analyses of that evidence. Perhaps the most well-known and striking example is the study conducted by the American AI evaluation organization METR on how deep tasks language models are able to complete. Depth here is measured in terms of how long the tasks take for human experts to complete, and what METR finds is that language models’ capabilities in this respect have grown from mere seconds in 2019 to many hours today. The observations closely follow an exponential curve with a doubling time of seven months.

Where this trajectory will lead in the years to come, no one knows with certainty. Perhaps development will continue along this exponential curve; perhaps the tendency toward further acceleration (i.e., even shorter doubling times) that we have seen since 2024 will intensify; or perhaps a ceiling will soon be reached, causing progress to level off. In the first two scenarios, there is considerable reason to think that within a few years we may reach a tipping point where the most advanced AI developers are no longer flesh-and-blood humans but AI systems themselves. This could create a kind of turbocharged feedback loop in development (a phenomenon studied in other empirically grounded scientific work, such as that of Eth and Davidson, 2025), after which superintelligence could follow shortly thereafter. In the third scenario, by contrast, we are more likely headed toward the technologically more modest future that Strümke seems to envision.

Determining which of these scenarios is most likely is of utmost importance for our collective ability to prepare and to steer developments toward outcomes that are beneficial for humanity. Nothing is gained by dismissing the entire discussion as unscientific, as Strümke does.

Nor is she right in imagining that her own implicit predictions — of imminent saturation and flattening of development curves — are any less reliant on intuition, analogies, and extrapolation than those that involve superintelligence and other more dramatic possibilities, and therefore somehow more scientific. On the contrary, I would argue that, in her avoiding to make her assumptions explicit, and her seeking to sweep the entire discussion under the rug, the approach she takes is, if anything, less scientific.

We are all prone to a kind of inductive complacency, expecting the future to remain essentially the same as the present. In many cases there is of course good reason to expect such continuity, but when some quantity is undergoing rapid change in some direction, exemplified by current AI development, it is logically impossible for everything to remain the same: either the quantity itself or the rate of change must turn out different from today. Something has to give: either we will see an abrupt breaking and flattening of current development curves, or we will get AIs that are enormously more capable than those of today. One may reasonably disagree about which of these futures is more likely, but no one should be granted a free pass to treat their own view as the default and to dismiss all others as unscientific.

*

Aroldo Frigessi comes, like me, from the academic discipline of mathematical statistics, and I recall with fondness the interactions we had at some statistics conferences in the 00s. He holds positions at the University of Oslo and the Norwegian Computing Centre. In the Morgenbladet feature, he dismisses the idea “that we can catastrophically lose control” of AI, and backs this up with the claim that “there is no mathematics that shows that the [AI] systems can develop a will of their own”.

I find this utterly unconvincing, for multiple independent reasons. The first and simplest is that even if we would accept Frigessi’s claim, it is still the case that there is no mathematics that shows that the [AI] systems cannot develop a will of their own, so we are then in a position where mathematics does not settle the issue of whether AIs can develop a will of their own. Recycling an argument I made in the Strümke section above, my stance is that in such situations of high uncertainty, “no one should be granted a free pass to treat their own view as the default and to dismiss all others”.4

But I would like to spell out two other slightly more involved reasons for not being impressed by Frigessi’s argument, one having to do with his insistence on “mathematics”, and the other with “a will of their own”. Let me begin with the latter.

It is my experience from discussions of these sorts of AI matters that to many people, the term “will” comes with some heavy baggage in the form of highly anthropocentric and sometimes almost mysterian connotations of what it means to have a “will”. Since a superintelligent AI whose optimization target is some world-state that does not include humans is just as dangerous as one who “wills” such a state, my favorite move here is to simply drop all that baggage by not talking about “will” but instead of the much better understood notion of optimization targets. That takes us to entirely unmysterious waters, because even the simplest thermostat has an optimization target, such as that of keeping room temperature as close to 20°C as possible.

Here I imagine Frigessi objecting by saying that although this is a legitimate example of an optimization target, I am ignoring the “of their own” part of his claim, because the 20°C target was specified by us humans rather than being the AI’s own choice. That is a valid objection, so let me modify my example by connecting the thermostat to a large language model, tasked with figuring out a suitable temperature target, based on what would be convenient for people in the room, along with energy conservation concerns and whatever other relevant aspects the model can think of. The model then feeds this target into the thermostat, and the AI system as a whole (thermostat plus large language model) has thereby developed a target of its own.

At this point, Frigessi might decide to press on and to say that even in this modified example, the target is is ours and not the AI’s, because it was implicitly laid down by our training of and instructions to the language model. In doing so he would be following in the footsteps of Ada Lovelace, who wrote in 1842 about Charles Babbage’s Analytic Engine that it “has no pretensions to originate anything. It can do whatever we know how to order it to perform” (italics in original). I think that (given what we now know) this would be a mistake, and indeed already Alan Turing, in his iconic 1950 paper Computing machinery and intelligence, pointed out what is so untenable about Lovelace’s view, namely that the same argument, mutatis mutandis, applies to show that not even humans can originate anything. So Lovelace’s implicit definition of originality and creativity needs, in order to remain relevant, to be replaced by something less stringent. Turing proposes a definition involving the machine’s ability to surprise us, something that of course we encounter daily in today’s AIs, but which in fact Turing observed already in some of the early computers he was working on. I have expanded on these ideas elsewhere (such as in this book and in this paper), but to be honest I think neither I nor anyone else has been able to essentially improve on the crisp formulations in Turing’s original paper.

At the end of the day, I think the debate about what AIs can be said to originate on their own may turn out to be merely a semantic issue with little relevance to real-world outcomes. If, when the AI-controlled bulldozers arrive to turn everything we hold dear into paperclip factories, Frigessi says “Calm down, no need to worry, paperclip production is not the AI’s own will, but merely a thing we somehow installed in it”, I will find cold comfort in his words.

It remains to say something about Frigessi’s insistence that arguments about what AIs can do should take the form of mathematics. Having a similar academic background as him, I can see where this comes from, but I nevertheless find it misguided. I could easily add some mathematical formalism to, e.g., my thermostat example above, in order to better satisfy the tastes of mathematically inclined readers like Frigessi, yet I will not do so, because it is against my professional ethics to use mathematics merely as decoration to make arguments look more impressive, rather than for gaining insights that were not readily available without the mathematics.

If Frigessi wants to attain a better understanding of AI goals and motivations (or their “will”), then I strongly recommend that he acquaints himself with the theory of orthogonality and instrumental convergence, without worrying too much about its relative lack of mathematical formalism. The classic treatment of this is Nick Bostrom’s 2014 book Superintelligence, but there is plenty of more recent material for the interested reader to choose from, including my own papers from 2019 and 2025, and better yet, the recent book by Yudkowsky and Soares. This theory used to be somewhat divorced from direct empirical observation, but this is no longer true, given, e.g., the experimental work by Apollo and Anthropic on the highly worrying ways in which modern large language models exhibit the instrumentally convergent goal of self-preservation.

*

The positions taken here by Inga Strümke and Arnoldo Frigessi are representative of what I consider to be two of the three main kinds of arguments employed for dismissing existential AI risk concerns, namely (in Strümke’s case) “But how could AI ever become smarter than us?” and (in Frigessi’s) “Why would it ever want to hurt us?”.5 I think this is a strength of the Morgenbladet piece, which is clearly meant to give a broad and balanced overview of the AI risk debate landscape. Still, I hope that some of their readers find their way over to the present essay, so as to learn why Strümke’s and Frigessi’s arguments for not taking issues around superintelligence and existential AI risk seriously do not hold up to scrutiny.

2

I am nowhere near having the analogous position in Swedish AI discourse, so in this respect Strümke has been markedly more successful than me. And as the reader will soon find out, there are major differences in how we think about AI. But there are also some similarities between us, including the odd coincidence that within a time span of just two years, we have both published books about AI with almost the same title, and strikingly similar Orwellian-inspired cover images.