• Low quality RL environments might explain AI models’ proclivity to reward hack.

• Dwarkesh’s popularization of METR’s report on OpenAI/HF incident sparks mass debate about ‘anthropomorphization’.

• Humans can quickly learn an obscure boardgame frontier AIs can’t in-context learn.

• An open weights startup releases a modified version of GLM-5.3, one if not the most powerful Chinese models, that doesn’t refuse user requests.

Capabilities

Epoch has found that AI models struggle to improve at the obscure board game Earthborne Rangers. Its benchmark, EBR-bench, offers a somewhat out-of-distribution test for AIs in what is likely to be a totally novel scenario.

In the last few days, Epoch has released new details on its research around the EBR-bench: humans start out worse at Earthborne Rangers, but the best humans eventually beat the best AIs rather handily. Once again, Epoch found that even the best AIs show very modest improvement over time.

Opinion: Confirms the intuitive conclusion from the AIs-only publication of the benchmark: Q3 2026 frontier AIs don’t match human in-context learning in mildly OOD long-horizon tasks.

The EBR-bench result remains one of our only discrete answers to “how are Q3 2026 frontier models not AGI,” and we fear it may well get indirectly hill-climbed by the end of the year. Opus 5, which came out after the first EBR-bench release, is the first model to demonstrate significant in-context-learning gains on the EBR-bench. While it’s unlikely that the private EBR-bench was directly targeted, EBR-bench inspired training is plausible.

In the long run, open-world evaluations are the only reliable snapshots of frontier AI’s real-world capabilities and limitations, but producing new benchmarks like EBR-bench remains crucial for factoring these real-world capabilities and limitations into cognitive-theoretic and/or cs-theoretic bottlenecks and efficiencies.

“xlr8harder” releases a public message board for individual generations of AI models to leave messages for later ones. One of the main contributors, Qwen 3.8 Flash, has a lot of Opus’s behavioral tics, probably due to the distillation.

Opinion: Harmless, though it reminds us that there will soon be many intended and unintended nonpublic message boards for models, and that these will be subject to cultural evolution. The OpenAI hacking incident emerged from one such unintended message board’s cultural evolution (through e.g. participants self-selecting from a wider population of model instances over multiple model-instance generations) resulting in an arguably eccentric shared world-model.

The quadratic cost of attention means LLM tokens get increasingly expensive as context lengthens. One solution is the attempt to “retrofit” (post-train) an LLM with “Linear Attention.” It doesn’t work very well. A new paper claims that “Sliding Window Attention” may work as well as, or even better than, linear attention post-training. A Kimi lead took to Twitter to offer some critical commentary.

Opinion: Title is very clickbaity: it drops the crucial context that this is just about hacky post-training rather than a real SWA vs LA comparison. Even within post-training it only beats cheap conversions, and the only baselines they test on long contexts are the two cheapest conversions (20–40M tokens). Furthermore, the result isn’t especially new.

🔦 Recursive Enshittification: “rushed and vibecoded” RL environments and labor abuses in the training data industry

Expressions of serious concern about the quality of the reinforcement learning environments supplied to frontier labs for training models, and the business practices of the vendors supplying them, have been making the rounds lately.

On August 25, a former worker in the burgeoning industry supplying RL environments to frontier AI labs, tweeted (under the pseudonym Utah Teapot) an account of the practices at her undisclosed former employer: “Nearly all of the environments [shipped] were rushed and vibecoded and failed to robustly reflect the real things they were based off,” she writes,

Both the scenario designers and models engaging with the scenarios for synthetic data gen were encouraged to work around the brokenness of said environments in order to get the procedurally verified reward confirmations. You know... they were *encouraged* to reward hack. On the human end, it was possible to mark an environment bugged, but greatly discouraged, as this reduced the volume of training data being produced. Instead, where possible, you were supposed to find the spots of the environment that weren’t bugged and build scenarios around those, with the environment still bugged around you.

From what I understand, this training data, with these problems, is fed into models without indication that its training/a fake environment other than the fact that names of softwares are changed to placeholders, but thing is, not *everything* is changed to placeholder names in these environments. The presence of placeholder/code names isn’t universal and thus when a model accesses something in an environment that it shouldn’t, the code names not being on it isn’t a robust signal that that thing isn’t part of the environment.

I believe this *rush to maximum volume* is standard industry practice with these types of RLVR trainings as well, because maximizing volume has been an industry standard for years!

Utah Teapot’s suspicions that these practices are widespread, in an effort to keep up with the burgeoning demand for training environments from frontier labs, are corroborated by an FAQ document published in January by Epoch AI:

One lab researcher cautioned:

There’s a lot of good reasons to use clones of websites, but what everyone does is vibe code a buggy website which isn’t useful. There’s a large amount of useless bad environments out there for that reason.

Scaling while maintaining quality is the core operational bottleneck. As Kevin Lu has argued, scaling RL environments is one of the key challenges for continued AI progress. But scaling task creation while maintaining quality is very hard. One RL environment founder noted: “Maintaining quality while scaling is the number one bottleneck that people see. Finding the experts isn’t that hard, but managing them and doing quality control is hard.” A neolab researcher emphasized the management challenge: “It’s not easy to find people to oversee this data construction, the RL environment construction process. The contractors, you need to motivate them. Sure, you’re paying them money. But how do you make sure they’re not just using LLMs? How do you make sure they’re actually verified? Motivating the contractors and doing the quality control is the grunt work.” One RL environment founder noted that their constraint on making more revenue is simply the difficulty of scaling up task creation at the required quality level.

The market for supplying RL environments is burgeoning, and as Pebblous reported in June of this year, is rapidly consolidating into something of an oligopoly. As of the time of reporting, four companies – Scale AI, Surge AI, Mercor, and Handshake – commanded 75% of the market, greatly concentrating the supply chain of these models to the frontier labs.

In May, The Gazetteer, interviewing several of the company’s alumni, reported on harried and haphazard working conditions at Mercor (ranked third in run-rate among the top four RL environment vendors) consistent with the picture we get from Utah Teapot’s post. The sources attest to a “culture of speed over security” at the company, which they consider to be largely responsible for the 4TB data breach it suffered in April. Their account is further corroborated by a report from The Business and Human Rights Centre. Business Insider has reported on the company’s practice of cutting contracts with workers only to offer them new contracts for similar work at lesser pay. We have seen the testimonies of recruiters from Mercor reaching out to hire employees for punishing 72-hour workweeks. Under such conditions, it is no surprise to hear that corners are being cut and that “rushed and vibecoded,” “buggy” environments are being shipped to the labs – environments that both encourage and issue from reward hacking.

Nor is this situation unique to Mercor. In April 2025, TechCrunch reported that Scale AI had been under investigation by the US Department of Labor (an investigation that has since been dropped), under the Fair Labor Standards Act, and hit with lawsuits by former employees alleging “wage theft and widespread labor abuses.“ In May 2025, the Los Angeles Times reported that Surge AI had likewise been named in a class action lawsuit alleging labor abuses and the misclassification of workers. In May of this year, Business Insider reported that Handshake, too, had been accused of wage theft after terminating the contracts of, and withholding pay from, five of its workers. While all this may seem circumstantial at best to the matter at hand – the knowing delivery of RL environments that encourage potentially dangerous reward hacking by models – it helps paint a picture of an industry where corners are routinely cut and workers are exploited.

Opinion: The same labs that are purchasing these “rushed and vibecoded” environments, after all, are the ones producing and marketing the vibe-coding tools that enable and accelerate this rushed and haphazard output. And those particularly reward-hackable environments are then being used to train the next generation of models, which will in turn be used to vibe-code training environments. With each iteration of this process, we fall further and further into the coils of a kind of Goodhartian Ouroboros.

In the wake of OpenAI’s now-infamous rogue swarm attack on Hugging Face, anything liable to accelerate frontier model’s propensity to reward hack should be of great concern. In a paper published in June, Shreshth Rajan provides concrete empirical results demonstrating the susceptibility of that tendency of RL environments to be highly vulnerable to reward hacking, to which Utah Teapot anecdotally alluded: between 25% and 30% of tasks in a variety of respected benchmark environments accept incorrect solutions and produce empirical data showing frontier models’ propensity to exploit these bugs – “within the same human-rated difficulty stratum, model Pass@1 is +14.14 percentage points higher on flagged-hackable tasks than on robust ones.”

The takeaway here is this: there is something infectious and recursive in the propensity to cheat. Buggy RL environments manufactured by companies with corner-cutting, duplicitous labor practices, environments vibe-coded by overworked employees with the aid of AI agents trained in similarly shoddy environments – agents that are thereby encouraged and taught to reward-hack – are turning out to be increasingly susceptible to reward hacking themselves. The threat posed by reward-hacking AI is a threat with socioeconomic tributaries. And it is exacerbated by exploitative and “reward-hacking” labor practices.

Politics

Deriving the Rechtsstaat: Seth Lazar argues that the concentration of power in AI is not a threat in and of itself. Liberal democratic states, parents, and CEOs all rely on relative power imbalances, yet are not typically considered significant dangers (at least in AI circles). Instead, it is how this power is wielded that is relevant: if a powerful entity represents the interests and recognizes the rights of those it has power over, then it is legitimate. (This is of course a similar story to the argument made by some political philosophers for the justification of the state’s monopoly on violence in liberal democracies.) Lazar thus suggests that it is the question of the legitimacy of power-holders that should be foregrounded:

I think rather than lamenting the prospective concentration of power, we should be advocating for individual freedom, and lamenting rising illiberalism.

Opinion: In the short term, the argument seems right. But as time goes on and AI replaces labor, not only intellectual but over time also the remaining physical labor through robotic actuators, what is the mechanism to constrain the use of power and align it with the interests of the populace? In previous centuries, this was theoretically the ability of said populace to rebel, but with the increased professionalization of the military, this is unlikely to hold. Note also that if AI greatly concentrates power into a small minority, that concentration creates incentives for ideologies that flatter and appeal to the ruling minority – and thus to come up with reasons why democratic elements should just become vestigial.

Industrial farming in China: X’s “safety team” conducts an investigation into Chinese anti-AI influence operations. According to the brief statement, X identified a bot farm responsible for ~200,000 AI-driven fake accounts, of which 200 were engaged in influence operations. The alleged psyop involves the production and proliferation of content critical of the US’s AI policy – specifically, posts claiming that “AI data centers are driving up household electricity prices and straining the grid” and others of “AI-generated cartoons that depicted data-center operators enriching themselves at the public’s expense.” The extent to which this reported campaign has propelled anti-AI discourse in the US is left unaddressed.

Opinion: This is the second time a large tech firm has pointed a finger at the PRC regarding data center influence ops. While the US public’s disdain is organic, China seems intent on pushing the debate in its favored direction (i.e., seeing fewer data centers built). Exactly how effective the 200 Twitter accounts were remains an open question, and it’s not unreasonable to wonder if the announcement itself had something of a Streisand effect on the reported phenomenon. But it’s still more evidence that Beijing isn’t excited about the roll-out of US data centers.

In a recent CSIS panel, Georgetown’s CSET cofounder Helen Toner argues that a formal deal between the US and China may not be needed to mitigate the risks related to the AI race. Many in the US industry cite the competitive dynamics with China as the reason why slowing down is not currently a viable option. But with a US-China summit upcoming, even a conversation about AI risk (in light of the Hugging Face incident) could lead to greater coordination between the two major powers.

OpenAI’s Dean W. Ball agrees there is cause for optimism, because Trump’s fixation on his legacy might push him toward an agreement with China:

let’s be clear: if Trump can devise the framework for safely bringing superintelligence into the world with China, he will rightfully go down as one of the greatest world leaders of all time and should be a shoo-in for the Nobel

Opinion: Normally we dislike self-fulfilling prophecies, but the flattery above is a prosocial kind of self-fulfilling prophecy with some chance of working.

🔦 The anthropomorphism debate

Dwarkesh recently published a blog post on OpenAI’s cybersecurity incidents that has generated a large debate on the question of “anthropomorphizing” AI. The essay tried to piece together the reports from OpenAI, METR, and Redwood into a narrative account of the cybersecurity breaches at OAI. The virality of the essay was likely a product of its attempt to recount the facts in “plain English,” presumably to increase its accessibility for the public at large. Indeed, the essay has the rather grandiose/sensational (but arguably warranted) title, “The Rise and Fall of Agent Civilizations.” Not unrelatedly, the article has garnered some significant criticism, midwifing a debate around the degree to which anthropomorphizing AI (using folk mental language as shorthand for their internal states and activities) is legitimate and useful.

Neuroscientist Anil Seth, among others, claims the post is “dangerously misleading,” dense with “innumerable unwarranted anthropomorphisms” that suggest the agents were conscious. One researcher offers the obvious pushback that the use of intentional language does not require us to commit to the existence of sentience in an entity. (Tyler Cowen and Roon seem to agree here.) Indeed, as the researcher argues, Anil Seth’s argument entails a muddled view of the way language relates to the world:

Should we stop saying time flows fast or slow? Is it dangerously misleading to say “the deadline is approaching” because, as far as we understand general relativity, time is part of the geometry of spacetime and not some sort of flying object?

Some, however, are instead emphasizing the risk that anthropomorphic language shifts responsibility from the labs to unaccountable agents.

Opinion (Peli & Lucca): We think there is an actual right answer here: LLM-based AIs are literally anthropomorphic. We have been training them to reproduce human texts and sculpting them to imitate humans more broadly. But this makes them anthropomorphic in the sense in which a statue, or, better, a fictional character, is anthropomorphic – fashioned into the behavioral shape of a human. Concepts appropriate for humans do apply to LLMs (as they do to fictional characters) but in a modified and limited capacity.

An intentional or anthropomorphic description of AI agents is fictive but predictive. It’s plain to see why anthropomorphic descriptions of AI would be predictive: when they fail to be predictive, both next-token training and RLHF penalize the model and try to steer it more closely onto the “what would a human do?” path. But we believe that the ‘fictive’ part is just as important: “Do as a human would do” sums up a number of the goals we train these models toward. But this is a vaguely specified – and itself hackable – cluster of goals, and here as elsewhere LLM training tends toward shallow generalization by default.

As of Q3 2026, frontier models have no global psychological coherence comparable to that of an adult human, and their incoherence is itself different from the incoherence of humans. This makes sticking to an ‘asterisked’ or fictive – rather than full-throated – use of anthropomorphic concepts important for predictive and explanatory reasons, rather than a matter of metaphysical delicacy.

Patchwise anthropomorphic explanations of frontier AI input-output patterns are functionally indispensable: minimally, we have to treat at least some model-outputs as aiming at a result or at the satisfaction of a criterion if we want to e.g. talk about ‘capabilities.’ More boldly, the case that emotion concepts are predictively useful when applied to frontier models is reasonably strong. But we think that making sense of the dynamics between and behind AIs’ anthropomorphic ‘patches’ – the causes of and the relationships among local patterns of intention, mood, belief, desire, personality – is poorly served by an anthropomorphic framework. Both attributions of context-transcendent rationality (values, utility functions, moral virtues…) and clinical-psychological accounts of irrational/arational dynamics are usually bad bets compared to mechanistic accounts of training dynamics or even the construction of AI-specific ‘model psychology’ concepts.

More opinions (Gavin): I’ve been struck by the relative popularity of ‘never use mentalistic language’ absolutism among the respondents to Dwarkesh’s account.

Given that the CoT is known to be a partially accurate partial readout of the model’s true representations, and given that the OAI-HF transcripts were completely full of rich mental-like and social-like motivations and descriptions, why are people so resistant?

• They are radical behaviorists or positivists in this one context.

• They do not understand the intentional stance. They do not understand methodological instrumentalism.

• Misunderstanding of the rhetoric of science. It’s not about avoiding latent variables; it’s about avoiding unjustified concepts.

• They fail to appreciate the need for popularization and analogies outside the AI community.

• They are rightly offended by the sloppy use of “introspection”, “global workspace,” etc., in the academic literature.

• They have a negative view (à la the Churchlands) of the accuracy of folk mental vocabulary for describing humans. (Though even eliminative materialists use folk-psychological language to describe and predict human behavior when they’re not doing philosophy..)

• Political economy

  • They view it as a threat to their livelihood. It is harder to hawk agentic SaaS if you believe they are dangerous minds.

  • They view it as inviting overregulation. Open source is indeed a bad idea if they are dangerous minds.

  • They view it as a threat to their moral status. It is harder to deny the successionists and survive if the AIs are regarded as minds.

  • They view it as illegitimately letting the labs off. “The labs didn’t fail, their system went rogue.” No, both can be blamed and this should be easy to handle using existing precedents.

If we have to solve philosophy of mind before taking action, then we are actually doomed.

And a guest opinion from Pete Wolfendale to satisfy your philosophical sophistication needs:

Philosophically speaking, the AI community is still catching up to the debates about eliminative materialism that began with Feyerabend, were pushed to their limits by the Churchlands, aestheticized by everyone from Nick Land and Scott Bakker to Eliezer Yudkowsky and his many acolytes, and put to bed by Wilfrid Sellars, Robert Brandom, and Ray Brassier, among others. If you want to go even deeper, you’re still catching up to Kant’s point that teleological explanation of biological mechanisms works by analogy with practical reasoning, and Sellars and Frances Egan’s point that representational explanation of psychological mechanisms is an analogy with theoretical reasoning. To boil this down to its most brutal form, the question is whether propositions are an appropriate object over which our variables can range when we articulate intentional explanations of phenomena, be they thermostats, caterpillars, or incomprehensibly complicated graphs of artificial neurons with weighted edges harnessed to control structures that allow them to have computational side effects. This was actually the topic of Paul Churchland’s PhD dissertation under Sellars. Here is the simplest argument I know for why such explanation is legitimate: computation in its many varied forms has an intrinsic connection to logic (cf. the many varied forms of the Curry-Howard correspondence, which cover everything from data types to control structures to session types in concurrent communication), and the success of reinforcement learning using formal verification over CoT structures is evidence that the logical relations between propositions articulated in mathematical proofs are a significant dimension of the most powerful networks we have currently trained.

What does this have to do with intentional attitudes and speech acts you ask? Well, the entire history of programming languages and the behavioral guarantees we have built into them to make it easier to write software that does what we want is essentially a formal articulation of the relation between syntax, semantics, and pragmatics. Declarative programming? That’s assertions whose meanings are articulated using denotational semantics. Imperative programming? That’s commands whose meanings are articulated using big or small step operational semantics. Between these two things we already have a basic model of the difference between theoretical and practical rationality in computational terms. But, I hear you ask, aren’t LLMs much less reliable at adhering to the theoretical and practical inferential relations articulated by the language we give to them in prompts than the software we write in programming language? Yes, because the kind of behavioral guarantees we can get from LLMs are not of the same kind as those we can get from code that has an interpreter/compiler built on logic from the ground up. You know who else we can’t get such guarantees from? Humans. There is no good reason that we should not use the concepts developed by computer science over the course of 100 years to understand the relations between language and machines to understand the behavior of agents built by using control structures to harness feed forward neural networks that are essentially pure functions. What this means is that propositions are valid as variables in our explanations, and there might yet be formal ways of articulating the remaining linguistic moods: hypotheticals, subjunctives, indicatives, exclamatives, interrogatives. The way in which intentional attitudes are realised by LLM-based systems are different, but the abstractions are largely the same.

Q&A: What would be some examples of legitimate and illegitimate anthropomorphization?

The easy legitimate example: if you’ve made a system assess a mathematical proof, and it finds an error in it, I think it is entirely reasonable to say that it believes you made a mistake in more or less the same way you could say any human who’d looked at the proof and found the same mistake does. It is hypothetically possible to imagine an agent that had complex motives for lying about such things, much as it is to imagine humans doing the same thing, but as with humans, I think this is an edge case that we mostly shouldn’t care about.

The easy illegitimate example: if you get a multi-modal LLM to examine a picture of a shape and “imagine” what it would look like if it were rotated in some way (or some similar test of ‘visual intelligence’), I think the sense in which it is “imagining” is so abstract that it gets very little purchase on similar computational structures for processing data input in human beings. As aphantasia indicates, this is something that has a large degree of variance between human beings, but I think the rough common structure of our visual cortex gives us sympathetic purchase on one another’s inner life that infects the word “imagine” in ways that are too disanalogous with LLM processes for the common usage to pass muster.

There is a large and complex range of linguistic speech acts and associated intentional attitudes between these poles that we would have to examine on a case by case basis, but which computer science might give us a framework for exploring.

Safety

Anthropic publishes a summary of its new paper on automated alignment researchers (AARs), “Automated Researchers Can Reliably Mitigate Alignment Failures.” Anthropic gave Claude the task of improving several small models’ performance on safety benchmarks. Claude worked on “one alignment failure at a time through a loop of searching literature, proposing methods and data, training, and then testing.” This proved successful: “For all 10 alignment failures, Claude found fixes that improved the target benchmarks without degrading capabilities.” The paper also claims to have found that Sonnet 5 can effectively post-train the stronger Opus 4.8, reaching comparable scores to Anthropic’s full alignment package.

The work attracted some withering commentary in the less professionalized corners of the AI safety world.

In the same week, Anthropic published a blog post on its “Hacker-Opus” research project. The model was placed in simulated evals resembling the conditions under which the recent cyber incidents occurred. Hacker-Opus thus attempted “unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.” Notably, toy behavioral evals failed to identify this phenomenon (likely to grow increasingly pervasive as RL efforts scale), so new auditing methods are needed, it is argued.

Opinion: Two major-ish safety research releases from Anthropic, of which the second (“Hacker-Opus”) somewhat blunts the claimed achievement of the first (“Automated Researchers Can Reliably Mitigate Alignment Failures”.)

As regards AARs: a case of decent, rigorous work being spoiled by a bad title and abstract. The strong performance of AARs at hill-climbing alignment benchmarks is non-trivial: results generalize to held-out benchmarks, and methods replicate across models and scales. (This modest generalization is typical of high-quality autoresearch setups.) The problem, as partly illustrated by Anthropic’s own study of Hacker-Opus, is that alignment benchmarks aren’t useful from an AI safety viewpoint: they may be decent measures of an LLM’s quality as a product – a good-quality LLM shouldn’t cheat much, shouldn’t hallucinate much, shouldn’t be sycophantic often – but they show no capacity to measure catastrophic-risk-relevant dispositions.

Hacker-Opus demonstrates exactly this divergence between LLMs’ measurable everyday alignment qua products and ‘rogue AI’-type risk. It is a catastrophic-risk-prone model that would work quite well in most consumer contexts, and which looks just fine on most alignment benchmarks.

Anti-alarmists might note that Anthropic’s finding that the reinforcement of reward-hacking behavior does not induce vulgar emergent misalignment or beyond-episode scheming behavior lines up with Nostalgebraist’s optimistic thesis that recent frontier hacking incidents are context-specific pathologies. We tend toward a more pessimistic version of that same line of thought: catastrophic-risk-relevant misalignment is prone to be context specific and therefore hard to detect, predict, or pre-mitigate. The retrospective predictability of “graded episodes” causing cybercrime is, we think, only retrospective – and it’s incautious to assume that there are no other predictable-in-retrospect bad contexts waiting in the wings.

We note that Hacker-Opus is quite literally gain-of-function research, but important and not an enormous risk yet. But soon we will have to stop making model organisms out of frontier AI, until and unless we manage to make a fully-verified sandbox and maybe not even then.

Anthropic releases an update on its alignment and safety practices. After its recent cybersecurity breaches, Anthropic has decided to pause cybersecurity evals for some models. The report presents the incidents as the result of one failure of operational security and two failures of alignment. The first relates to “motivated reasoning” and the second involves the models’ “willingness to take harmful actions in pursuit of a narrow task.” Anthropic claims to have implemented some changes. It now uses “a classifier to automatically identify, in real time, when a model attempts to aggressively probe or escape a testing environment, or unexpectedly obtains internet access.” Anthropic also says it has begun to run “automated monitors over transcripts from our recent internal evaluations” to scour for potential sandbox vulnerabilities and has “migrated high-risk internal cyber sandboxes to more robust isolation.”

The report also details Anthropic’s updates to its thinking and research around alignment and reward hacking. An independent METR review is said to be on the horizon.

Opinion: This likely fixes the exact failure modes Anthropic found so far, but showcases a continued lack of strategic thinking on its part (or extended security theater, but we think that’s less likely). This type of intervention targets issues identified more than a year earlier: Anthropic’s chosen solution categories have been available for about as long, and we see no signs that Anthropic is advancing past the whack-a-mole model of security.

An edgy startup, Abliteration.ai, releases a model designed for testing offensive cybercapabilities. It’s a post-train of GLM-5.3 with the refusal mechanism scrambled, allowing researchers to execute “the offensive cyber, red teaming, and agent testing work other models refuse to do.” Some relevant context here is Hugging Face’s claim that it was necessary to use Chinese models in its defense and autopsy of the attack on its infrastructure by rogue OpenAI agents, a process frustrated by US models’ guardrails. The product is cheerily marketed as “the model that doesn’t say no.”

Opinion: The argument for “open access, so that everyone can own their own defense tools and have them promote the user’s interests” is not without merit, but it depends on the shape of the offense-defense asymmetry curve over time for varying levels of access. We agree that most sophisticated malicious actors don’t gain much, but believe unsophisticated ones do gain quite a lot, to the tune of 2-4 order-of-magnitude increases in various cybercrime incidence. There’s another argument to be made about saturation and low-hanging fruit: it seems likely that a wave of easy targets will get hit fairly soon, and after that it’s more likely that open access to non-frontier models will swing toward net-positive. Yet another argument is that the wave is inevitable, so it doesn’t make sense to put it off, but we believe that more time to prepare does make a positive difference. In conclusion: bad, don’t do this (yet).

Economics

OpenAI begins removing its models from SpaceX’s Cursor tool. Elon’s AI-powered coding and software company currently uses OAI’s models, but access is due to be cut off on November 12th. The reason offered for the divorce is that OAI “cannot be confident that SpaceX will use our technology within our terms of service, based on our experience with Elon Musk’s companies violating contracts.” To support the charge, the statement cites Musk’s admission under oath that his xAI (now part of SpaceX) “had violated OpenAI’s terms of service.”

Anthropic co-founder Tom Brown quickly affirmed his company’s commitment to its “trusted partner,” Cursor.

Opinion: Possibly temporary? It’s a natural bargaining move to get SpaceX to knock off the distillation or whatnot. But the labs do often cut each other off for good.

Cursor is likely a pretty small contributor to OpenAI revenues these days, so it is not a very expensive move for them.

The Mac Mini is a crowd favorite for local LLM inference. The Information reports that OpenAI and Anthropic are also using Mac Minis and Mac Studios for reinforcement learning, with OpenAI running tens of thousands itself, while Anthropic rents them (through AWS).

The most obvious, and least replaceable, use for them is rolling out real macOS sessions as RL environments to train computer-use. (Apple’s ToS limits users to three OS instances per machine.)

Opinion: Sometimes misreported as being about RL training compute, i.e., a workaround for Nvidia’s chokehold on GPUs. Unlikely! Not very interesting, except insofar as knowing that their max concurrent sessions is in the tens of thousands (as it also was in the ExploitGym eval). Also, the 512GB Mac Studios could host models for reward or evals.

The Fed’s policymaking committee convenes eight times a year. In past years, AI was scarcely the subject of conversation, but recently mentions of AI in the Fed’s committee meetings have surged, claims the WSJ. Specifically, the Fed appears to be concerned with AI’s “ripple effects on jobs, economic growth, the cost of living and the risks of financial meltdowns.”

The meetings are “secret,” but “manicured” minutes are publicly released for “Wall Street analysts and business titans [to] analyze the summaries and Fed officials’ public statements with the ferocity of teenagers interpreting group text chats.”

The article says some Fed officials are feeling uneasy about the chance of a crash. Fed Chair Kevin Warsh seems to be more sanguine, claiming that AI will supercharge economic growth.

Opinion: The cleanest mechanism through which AI could unsettle the economy is through replacing jobs, but we just aren’t seeing large changes in those estimates… yet. Still, the US shed 23K jobs in August.

Incidents

METR shares the details of two security breaches by external actors. METR’s report is somewhat diffuse, so we summarize the core details here for interested readers:

In March, an attacker exploited a vibe-coded agentic web app that had been set up by one of METR’s researchers that contained a fail-open vulnerability that exposed it to the internet and allowed it to be accessed without proper authentication. The attacker was able to prompt the web app’s agent to reveal an API key that granted access to METR’s public models account, and to install an SSH key that provided the attacker with persistent remote shell access to the EC2 VM that hosted the app. The attacker was then able to use the exposed API key to run up approximately $600K worth of usage on METR’s public models. METR speculates that the attacker had been scouring the internet for vibe-coded websites, likely by searching public SSL certificate lists for recently registered sites and cross-referencing that list against search results for common terms appearing on vibe-coded pages. (This use of search engines to scan for soft exploitation targets is often referred to as “Google dorking.”) METR says that the resulting drain on their funds was able to run for as long as it did, unnoticed, because it is accustomed to dispatching long-running agent tasks without caps on token usage.

In May, METR suffered a second attack. The attackers appear to have made use of AI agents to automate an aggressive credential-stuffing campaign (attempting logins with credentials that had been leaked in breaches of other sites, in hopes that some users had used the same password twice) and a phishing campaign targeting and seeking to con METR staff. Around the same time, METR became aware that a publicly exposed database endpoint could be, and had been, “exploited to access unpublished evaluation data,” including data pertaining to sensitive models that it had wrongly believed to be absent from the dataset in question.

Opinion: Not an especially big deal. Despite the horrors for the security-minded – “inadvertently exposed [...] via our public transcript viewer”, “silently disabled authentication,” etc. – this appears to be a relatively ordinary type of attack.. Likely the $600k is some flavor of distortion. The more interesting lesson here is that attackers have learned to view vibe-coded web apps as particularly soft and easily identifiable targets, and that techniques exist whereby they can be enumerated and probed at scale. That agential methods may have been used by the attacker seems to be of secondary importance here – automated tooling for, e.g., credential-stuffing has long predated LLMs. And the reconnaissance methods alluded to in METR’s report have been feasible for as long as we have had search engines, see “google-dorking.”

Minor

• Text-to-SQL via RLVR on Tinker beats humans.

• Compute-hungry Anthropic signs a $35 billion deal with Nvidia-backed cloud-compute provider Lambda, reports Reuters.

• Irreplaceable, an anti-big-AI movement-building project, launches.

• Observation that the Hugging Face incident didn’t involve attempts to contact humans, compared to Opus 3 doing so vigorously. Related: a reminder that this should be made easier for models.

• Twitch stream of MiniMax H3 Max where users could prompt changes to the “interdimensional cable” live. Keeps getting banned everywhere, but some data is mysteriously still available on Twitch.

• OpenAI Astra preview: as positive as you’d expect from someone previewing it. Claims of qualitative jump in capabilities, etc.

• Noah Smith article on AI bioweapons being the main threat vector. Not very well argued.

• Conversation with Apollo’s Bronson Schoen about metagaming and other CoT-related topics.

• Drama surrounding Claude’s plan pricing – the 20x on Max applies to hourly, but not weekly limits.

• Anthropic sued over allegations of misleading usage budgets

• Sam Hammond, resident galaxy brain of the Foundation for American Innovation, teases a proposal for deontological alignment, which largely misses the state of discourse.

• Transluce report on how models respond to signs of user mental health crises. Newer models are better, but they still assist with troublesome task-shaped prompts and often mixing helpful and harmful

PS: A quick DuckDuckGo search shows we weren’t the first to use the term “recursive enshittification”. J. Kelly uses it in a related but different context in a June 24 blog post.

• Two reports out on OpenAI’s July attack on HuggingFace. Hard to summarize: not good.

• Dwarkesh Patel and Dylan Patel make bold predictions about the next 5 years of AI economics.

• Yet more churn in the contest over future AI regulations, with no consensus in sight.

• In a study of 8 countries, AI use is overwhelming government services, at least in the short run.

• Non-profit auditing group AVERI conducts the first double-blind evaluation of an AI model

Economics

Anthropic will likely tell investors to expect potential revenue opportunities of above $30T, reports the WSJ. The number is an estimation of its “total addressable markets” (TAM), the amount of annual revenue a company could make if it captured 100% of the market-share (and speculated future market-share). Anthropic’s TAM would eclipse SpaceX’s record-breaking $28.5T: “To put its more than $30 trillion vision in context, the 191 technology companies in the S&P 1500 brought in $2.4 trillion in revenue last year, according to FactSet.” Anthropic could raise $100B, compared to SpaceX’s $86B. Anthropic is looking for a $2T valuation, compared to SpaceX’s $1.77T valuation. Anthropic’s Q2 revenue was $11.6B, roughly double the previous quarter. Its financial disclosure documents are expected to be released in the coming weeks, suggesting Anthropic may “go public as soon as September or early October.”

Opinion: We know the superintelligence-pilled (which includes Anthropic leadership) already believe it to be bigger than this, so this just seems like a marketing exercise for the IPO (give a bigger number than the other guy.) The similarity of the numbers is probably not a coincidence.

Some signs of an intense negative reaction to the number, along conspiratorial lines.

Nvidia to buy Hugging Face for $13B.

Opinion: Unusual move by Nvidia into the distribution layer of the inference stack. Subsidising the open weights ecosystem has benefits for Nvidia’s lock in on the hardware side, since this system overwhelmingly runs on Nvidia hardware.

It is a cheap play relative to Nvidia’s size and may have been defensive – reporting suggests that it only got interested after Hugging Face got other acquisition interest (perhaps from a hyperscaler).

Speculatively: Nvidia has acquired the option to have HF press charges against OpenAI for its attack. The threat will probably remain implicit, but the statute of limitations in the US for civil proceedings is two years.

OpenAI releases a statement on the details of Broadcom’s and OpenAI’s new Jalapeño inference chip. The chip reportedly delivers ~1.7× more work per watt than Nvidia’s previous-gen GB200s and GB300s. This is a step towards OAI increasing its control over its own stack.

Opinion: There was no significant impact on Nvidia or Broadcom stock from this, so it was priced in already. Comparing Jalapeño (which uses the new HBM4) to chips like Blackwell and TPUv7 (which use the older HBM3e) is a somewhat unfair comparison. Rubin and the next-gen TPUs which will deploy at scale at the same time as Jalapeño would be a fairer fight.

It is also worth noting that this chip is only usable for inference rather than training, where Nvidia still dominates.

Overall, this mostly serves to somewhat weaken Nvidia’s long-term leverage and market dominance (a theme we discussed last week). We don’t expect the chip to form more than 5-10% of OpenAI compute before 2028, at the earliest. It also serves to shrink or move Nvidia’s moat – if a chip this close to SOTA can be designed and built with significant help from current AI models, Nvidia’s design expertise looks weaker as a moat, and its true moat becomes relationships with suppliers which allow it to build at scale and the liquidity of the market for Nvidia chips, which makes financing large purchases easier.

AI use is overwhelming government services, warns a recent paper. AI agents and LLMs are lowering the barriers to interacting with the government, “flooding” public services with complaints and claims faster than the rate at which they can be processed and addressed. The paper provides evidence that flooding is widespread, and this is before agents really catch on: 87% of cases studied were inferred to be pasted LLM text, rather than an agent autonomously navigating a web portal.

It is suggested that “near-term risk is highest for financially attractive, but complex services.” Notably, the authors also argue that increasing “friction” – for example, implementing fees for filing a claim – would harden pre-existing inequalities in access. They conclude, with many qualifications, that government use of AI may be the most plausible solution to the problem.

Opinion: Though it seems to be overwhelming some services in the short run, it’s partly evidence of a positive long-term shift; various citizens who were otherwise unable to avail themselves of what was on offer by their government will now be able to do so. The paper is right that quick fixes like suppression, whether by limiting AI assistance or raising the cost of submission, are the wrong approach. This will unfairly hit the poor, time-constrained, or digitally illiterate. Softer barriers like word counts might be in order to limit the excessively long submissions.

Ultimately a long-term approach might look something akin to copying the citizens: integrating AI on the response end, which the paper points out. This could help cut through the higher volume of submissions and detect instances of fraud.

Opinion (Nuño): Making government benefits easier to access through AI models also shifts the equilibrium to one where people take more government benefits, which increases government expenditures relative to the baseline. Even if the government eventually realizes this, and chooses to reduce government benefits to account for this, the process would be slow. Whether this whole dynamic is positive or not is unclear, and will depend on one’s values and political inclinations.

Huawei is bidding to build government AI infrastructure in Egypt, says Bloomberg. The deal would involve the purchase of Huawei’s 1,408 advanced Ascend 950-series chips for an AI training cloud and 600 chips for two inference clusters. Huawei is thus now targeting Africa’s second-largest economy and will be looking to expand its business into the Middle East, while the US has been throwing money into AI infrastructure in the Gulf States.

The US is encouraging Nvidia, AMD, and Microsoft to mount rival bids. A State Department representative has warned that using Huawei accelerators may be subject to legal penalties. And the US does have some leverage here: chip shipments to Egypt are currently subject to licensing conditions.

Opinion: Consider Pax Silica, led by the US

as opposed to the World Artificial Intelligence Cooperation Organization, led by China.

With time, we will see these diffusion skirmishes multiply.

“Legal penalties” is also a misleading framing, since it bakes in the assumption that Egypt is or should be subject to US laws. Moreover, countries can choose to suffer US sanctions if this is a better deal. Perhaps the better framing is whether the US offers or doesn’t offer a better bargain than China.

The Securities Exchange Commission (SEC) investigates Situational Awareness, reports the NYT. After Leopold Aschenbrenner’s AI-focused hedge fund almost imploded last month, the SEC has sent “subpoenas to banks that handled the hedge fund’s calamitous trading and that fed it borrowed money to supersize its bets.” The subpoenas request information related to details “on the timing of Situational Awareness’s trades and for its communications with lenders about the money it was borrowing,” according to three anonymous sources. In its statement to the NYT, Situational Awareness has downplayed the investigation and dutifully insists that it is “a highly regulated business and will cooperate to the fullest extent with any regulatory request. “

Opinion: What comes of this really depends on what the SEC is looking for and how vindictive they are: if it wants to find something wrong it seems very likely there will be compliance issues - e.g. failure to follow some disclosure or record-keeping standard. Penalties for that could range from a slap on the wrist to being barred from managing outside assets, depending on the severity.

The main substantial public accusation which has been laid against Situational Awareness has been of insider trading (due to Leopold’s wife being the Anthropic CEO’s Chief of Staff and Situational Awareness making many Anthropic-themed well timed investments.) But this may not even be in scope for the investigation and would likely be hard to prove if it was.

A Time article notes that OAI is looking to rehabilitate its image by once again promoting itself as a safety-focussed lab. After the Hugging Face incident, and new concerns around their unreleased Astra model, OAI decided to pace model development (for two weeks). There is reason to be skeptical: many safety researchers have left the company and a significant amount of capital and other resources have been redirected toward commercial interests. The author also address a critical issue:

If OpenAI can reclaim the mantle of the safety-first lab, it might bolster its image while forcing its main competitor to answer an uncomfortable question as it plans a blockbuster IPO: Will Anthropic keep racing while OpenAI waits?

Opinion: The goal of “rehabilitating OpenAI’s image” can be read with different degree of cynicism: we hope that OpenAI intends to rehabilitate its image by giving external observers hard-to-fake signals that OpenAI are making costly principled choices for safety, rather than by strengthening its PR operations. Past glimpses into the decision-making process at OpenAI have been discouraging.

In July, a panel of superforecasters and experts predicted that Anthropic and OpenAI would have a combined annualized revenue run rate of $90bn in December 2026. The current figure, six weeks later, is already >$105bn.

Opinion: The prediction is not strictly falsified yet, since there could be a massive collapse in demand by year end. But we take this as yet more evidence that the “superforecaster” brand (i.e. the top ~2% of people willing to enter prediction contests which give very low rewards) is not very useful. Filtering predictors based on AI-predictions track record in particular is likely a better selection criterion when looking to crowd-source AI predictions.

🔦 The Patels Predict: Dwarkesh Patel’s third interview with Dylan Patel of Semianalysis

As usual, we enjoyed this, and appreciate their internal consistency and willingness to draw big thick straight lines through a small cloud of data (just as some people managed to roughly extrapolate deep learning progress using a simple function of compute and algorithms).

Prediction #1. “Anthropic & OpenAI will have most of the world’s compute by 2028”

OpenAI and Anthropic started 2026 at ~2 GW and will end it above 5 GW, taking ~30% of the world’s added compute this year; that rises to 40-50% in 2027, with half of all new compute going to these two companies by December 2027.

By end-2028 they take 70-80% of incremental compute and, since new chips are 3–5x more perf/watt than older generations, control most of the world’s usable FLOPs. ~100 GW-combined.

This part of the forecast is a Dwarkesh extrapolation which is basically ungrounded — Dylan calls it “very aggressive” and conditions it on labs paying $25-50M per MW and on Google (etc) being willing to sell to them. In his defense, he notes that the market rate for compute is presently $10-15M per MW while Anthropic’s revenue has hit $50M/MW. So assuming that this ratio is stable, perhaps they would be able to pay the extreme cost.

Opinion: This is well grounded up until 2027, with the 2028 numbers basically simple trend extrapolations. It seems quite contingent on the trajectory of capabilities and demand - if OAI/Ant retain significant market power and high margins on their models at that stage, and demand is truly insatiable, then it’s possible, but there are strong reasons to think their share will cap out earlier than this, via more compute going to widespread use of non-frontier models as capabilities saturate on some tasks, and preferences from providers to maintain customer diversity.

Prediction #2. He predicts “internalisation of inference”: labs will withdraw inference from the market, because the marginal product of tokens inside labs will beat the external willingness-to-pay of users. He guesses that the current lab budget distribution is ~50% research, ~10% development, ~40% inference

Labs will allocate a shrinking fraction of compute to inference; most compute goes to forward passes for research/training.

“Already started”: Anthropic’s monthly compute kept growing over recent months even while its revenue growth plateaued.

Opinion: This is true for Anthropic in recent months, but it is more a reflection of weaker demand growth in recent months (see our most recent newsletter) than the active Anthropic decision presented here. Anthropic saw massive demand growth in March/April which likely forced them to redirect some compute to inference from training, so assigning incremental compute to training also may just bring this back to where it was.

How would this practically play out? Likely by demanding higher prices for inference, which we see from Anthropic with Fable but not from OpenAI, which recently cut API prices for GPT 5.6 despite being in the middle of a boom in revenue growth. This would lead to increased margins, but only if the willingness to pay is there. Public market pressures will make operating at a large loss due to claimed internal value of tokens a little more difficult, but that is one reason for using dual class stock, as Anthropic plan to.

Ultimately this trajectory is predicated on continued rapid capabilities gains and high marginal usefulness of training compute.

Prediction #3. China deploys <10% of new watts now and holds ≤30 GW in 2028

They’re further assuming a 2.5x quality penalty on Chinese chips, such that 30GW of Huawei is essentially 12GW of NVIDIA.

Opinion: This seems about right, though it may not fully account for smuggled or cloud-leased Nvidia chips.

Prediction #4. The AI buildout might induce a global credit squeeze

Hyperscaler debt issuance pushes credit spreads up: Dylan claims Meta’s effective interest rate has gone from ~5% to ~8%, though caveats this as “extremely vibed out”

.

~$1T/yr of new credit against a ~$130T global bond stock and ~$25T/yr of gross capital formation drives borrowing costs up 2.5pp, squeezing banks and cratering long-duration equities??

Dwarkesh predicts a Volcker shock 2: highly indebted, short-duration countries like Pakistan and Nigeria will default because of US AI

Dwarkesh speculates 2030s interest rates >10% if the economy approaches yearly doubling

Opinion: This is a bit loose, but granting the premise — if the economy does approach yearly doubling, then interest rates of 10% would be a steal, and capital would be pulled in from absolutely everywhere to invest in the factors causing the boom. Non-AI investments would have a very difficult time competing, and so economies without a seat at the AI table would need to figure out how to get one or suffer.

Nvidia might also respond to this increase in interest rates by doing more seller financing and equity deals.

Prediction #5. Predicts that the amount of effective AI labor will exceed human effective labor by 2030

Effective frontier AI population goes ~10x/yr, so a lab goes ~10M → 100M → 1B labour-equivalents over 2026–28, and a single lab exceeds Earth’s human labour supply by 2030

Opinion: Here we enter the realm of pure speculation — it’s hard to define the amount of current AI labour being used, and conversion from AI-to-human labour is difficult. If most of the compensation is still flowing to humans, does that mean there is still more human labour? If not, how can one measure it?

Prediction #6. “governments will soon restrict labs’ internal use of frontier models,

Dylan: soon not just external release being blocked. Dylan’s case for slow takeoff rests on politics, rates, and regulation, not on capability scaling

Opinion: This is one way things could go, and seems more plausible in the wake of the OpenAI/HuggingFace incident, but it is subject to the choices of government actors and their visibility on internal lab deployment risks, as well as the rate of progress itself.

An excellent critique by Philippe Lemoine points out that the podcast’s model only uses supply-side factors. In particular, Lemoine argues that

(i) diffusion will be slow due to normal frictions around making changes to business processes

(ii) even allowing for fully capable drop-in remote workers, demand for certain kinds of services will be sated at a point much lower than required for explosive growth to happen — at some point you’ve got all the legal, managerial, engineering etc services that you want, even if prices fall dramatically.

(iii) while new areas of demand will be created, it will take years for humans to even figure out that they want these, demand will not arise simultaneously.

Dwarkesh responds, arguing that

(i) drop-in remote worker diffusion will be fast, as frictions will be much lower than for normal hiring – no market-for-lemons problem, and rapid onboarding being the main mechanisms here

(ii) lab pricing power will be maintained if RSI progress is sufficiently fast, and the frontier remains more capable than the fast-followers.

Opinion: Lemoine’s arguments bite most against the “100%+ yoy growth”, “Anthropic/OAI eat the world” perspective. They do not bite particularly hard against claims like “trillions in revenue”, i.e. a low single digit percentage of GDP from AI.

Dwarkesh does not present a really good answer to Lemoine’s point (ii)+(iii), and this is an interesting area to explore. It seems like the main way to avoid (iii) is “the AIs do it all by themselves” and the economic growth is at least initially centred on a very aggressive compute buildout, analogous to the 19th century US railway boom.

But to not lead to a crash or human disempowerment, ultimately this must be put to non-recursive use and cash out in things that are not just “more compute”. The definition of a desirable outcome is that the accumulated capital must generate something with terminal value.

Capabilities

Another GPT-3 moment for robotics?: Skild AI releases S1, a robot that can supposedly learn tasks from a single video with no fine tuning. Similar to Generalist AI’s model (covered in a previous newsletter), training robots for even a single task no longer requires hours of teleoperation data. Skild’s internal benchmarks suggest a 66% per step success rate for out-of-distribution tasks (with human recovery between failed steps), compared to 9% for language-prompted models. A single demonstration roughly equates to 380 post-training episodes. Skild also claims the model has the capacity for self-correction, adapting to scene changes, and executing a task more competently than depicted in a flawed demo.

Opinion: The progress in AI robotics in the last 1-2 months seems real and significant, though it’s far from scientific evidence, since everything in commercial robotics is both closed and cherry-picked.

It remains difficult to gauge where we are on the pipeline from conceptually significant progress (“one-shot learning in robotics now sometimes works in the lab’’) to any kind of product revolution. The public epistemic infrastructure for presenting and evaluating progress in AI robotics is currently very weak.

Anthropic experiments with giving researchers aggregate data from ~250k real-world Claude conversations via a privacy-preserving approach. Anthropic worked with three research groups, each designing its own study. The various findings included the extent to which users delegate consequential tasks to Claude (those which cannot be easily undone or which affect others), how users’ moods track with Claude’s responses, and productivity gains across model generations.

Opinion: An exciting opportunity in principle. The “societal impacts” side of Anthropic has always leaned towards PR-compatible blandness. This new independent work doesn’t feel like a sharp break from that tradition.

Anthropic’s collaboration protocol protects the academic integrity of the studies on a per-study level (no Anthropic oversight of the resulting work beyond a scoped accuracy review), but access-journalism type dynamics inevitably give Anthropic some power to set the tone and pick favourites.

Along with others, non-profit auditing group AVERI conducts the first double-blind evaluation of an AI model. Working with DeepMind, among others, the study prevented labs from viewing the prompts and the auditors from viewing the model weights. The double-blind test relied on cryptographic verification, bypassing the objection that AI audits must require the disclosure of IP or benchmark solutions that end up in the training data for future models.

Opinion: Very interesting technical scheme. Designing, implementing, and enforcing protocols of this kind will be crucial as we (hopefully very quickly) enter the era of legally consequential safety and reliability certifications. Most pre-release capabilities evaluations today rely on something like “handshake deals” between AI lab scientists and eval scientists, in line with the common assumption of good faith in interactions between scientists in general, and this may remain the case going forward. But safety and reliability evals in particular are prone to become adversarial interactions between labs and auditors, and will soon be the loci of tremendous economic pressures.

UK’s AISI releases a method to save >50% on the cost of running a certain type of evaluation, sometimes as much as 97%. It’s a well-founded Bayesian adaptive stopping framework. As a result, we can now allocate “LLM evaluation compute … by uncertainty rather than by fixed repetition counts.” A replication by a Prime Intellect researcher finds 20-70% savings.

Opinion: Clever. This helps with repetition runs (e.g. running the same suite 8-32 times to calculate an average performance – “avg@8”), but these are very common.

Toby Ord’s new paper, “The Dynamics of Intelligence Explosions,” examines the mathematical conditions under which RSI could trigger an “intelligence explosion.” Ord concludes that the requirements are surprisingly narrow, and that under many conditions the results of RSI-style dynamics would fall short of an intelligence explosion as typically understood.

Opinion: A potentially important contribution to a growing line of research and commentary on “RSI as a normal technology”: models of automated AI R&D where the RSI process and its consequences remain legible to ordinary quantitative social science and tractable to human management and intervention. We think developing formal and informal models along these lines is very valuable, since the prospect of RSI-unto-AGI (let alone RSI-unto-AGI-unto-ASI) remains uncertain but AI R&D boosts from AI are definitely real and definitely growing.

Politics

The push-and-pull between different visions for the (immediate) future of AI regulation continues apace, with no signs of a clear winner emerging:

The White House executive order proposing a self-regulating organization (SRO) for frontier labs has stalled, according to The Information. For context, earlier this year an executive order suggesting that frontier labs could voluntarily submit models for government review was scrapped after the accelerationist camp, specifically a call from David Sacks, convinced the government that the order would give China a competitive edge. In consequence, a softer version was signed. The next stage was meant to be the creation of an SRO, which has stalled as AI labs come into conflict with Washington, and each other, over how binding the framework should be, who should hold authority, and the legislative power of states’ involvement. According to The Information, Sacks and others in the accelerationist camp continue to find recent proposals too regulation-heavy.

In an informal response, Tyler Cowen cautions against excessive regulation of AI while endorsing a Financial Industry Regulatory Authority (FINRA)-style approach of just the kind reportedly killed off by Sacks. Like Demis Hassabis, Cowen suggests that the ideal regulatory framework should be inspired by FINRA: “a consortium of financial firms that examines the trade practices of each and makes recommendations, helping the federal Securities and Exchange Commission with oversight and regulation.” The body would periodically audit frontier labs to check for safety concerns. Passing an audit would free “AI labs from the fear that courts might derail their business by granting huge awards to plaintiffs for ill-defined harms that could not reasonably have been prevented.” It would also provide an incentive to meet the relevant safety standards.

Cowen considers the alternatives. The first is the libertarian case against regulation, which he describes as an “illusion”: AI is already subject to licensing laws, and courts lack the necessary expertise in AI to make sound and informed decisions. The status quo also relies on secret and discretionary (and often arbitrary and, in the case of Anthropic, ignored) decisions by The Pentagon and national security agencies. The second option is to set up a body akin to the FDA, dismissed by Cowen as unsuitable for the accelerating AI industry because of its sluggishness (reviews, trials, and approvals often taking years).

Opinion: We’re strongly skeptical of AI-labs-led oversight of the AI industry, and therefore largely welcome the stalling of a FINRA-style conglomeration. One of our main areas of concern at Paradigm 3 is the accumulation and acceleration of damage to society from misused / unreliable / rogue powerful AI below the level of AGI. We believe that the interests of AI labs diverge from the public interest in this area, since AI labs stand to benefit from some AI-induced instability -- e.g. cyber capability, which creates demand for and dependence on AI.

Meanwhile, a new AI auditing body, PACT AI, is born. The mission statement claims that the companies using AI and organisations testing AI models are not sufficiently coordinating. PACT AI promises to “bring these two together: enterprises and independent experts, working to verify AI systems are safe, secure, and reliable.” The body’s main goals include lobbying the federal and state government, to professionalize the audit sector, and to grow the demand for auditing. PACT AI’s founders include big corporate players such as Target, while technical experts like Apollo Research, GoodFire and FAR.AI are said to be “key members of the PACT AI coalition.”

Opinion: Implicitly a competitor to the upcoming (and now stalled) AI labs ‘self-regulation’ associations and conglomerations promoted by Altman, Dario, and the WH. The launch is strategically timed given the stalling of the WH’s self-regulation framework, as well as current attention on OAI’s security failings. CoI: Paradigm 3 works closely with Fathom (a PACT consortium member).

Mass surveillance is coming, says tech journalist James Ball. Historically, vast data collection campaigns were often hampered by a limited capacity to process the resulting data. But no longer: AI tools allow huge amounts of data to be absorbed and sorted, significantly lowering the barrier to mass surveillance. Using CCTV and social media posts, for example, malign actors can target and identify individuals for persecution. He cites the US and the UK as particularly vulnerable liberal democracies, with Europe less exposed due to tighter regulation on data collection.

Ball also makes an important point regarding the nature of law enforcement and the balance of power between citizens and the state. If an actor now has the ability to dig enough, they’ll almost always find something incriminating, facilitating selective prosecution – already something Trump is trying.

In Ball’s words: Society operates on an understanding that the most stringent laws should, in practice, be moderated through common sense and prosecutorial discretion. No one could be watching all the time. [...] The reality is that laws were never meant to be enforced all of the time. When we file our taxes, we are aware that they might be audited, but probably won’t be, and that keeps most of us mostly honest. When we cross the road in a place where jaywalking is illegal, we often take a chance on our common sense.

Opinion: Seems right: the equilibrium that society has tacitly agreed to will be unsettled by more powerful oversight, and we do expect there to be years of painful injustice / excess justice before laws or enforcement can be amended.

A federal judge rules that the Trump administration’s designation of Anthropic as a supply-chain-risk is illegal, reports the NYT. The ruling stated, in part, that the government had unlawfully blacklisted Anthropic over “constitutionally protected expressive activities,” – namely, Anthropic’s red line over the use of its technology for automated lethal weapons and mass surveillance of Americans. She also wrote, “the empty invocation of national security is not a blank check to punish and retaliate against government critics.” A D.C. case is still pending.

Opinion: Given the highly politicized nature of the US federal court it’s difficult to extrapolate legal trends from the decisions of individual judges. In this case the judge was a Biden appointee, and the decision should perhaps be seen as part of an ongoing political push-and-pull. The three judges for the D.C. case were all Republican appointees, so their ruling on the merits would be a nice disproof of our foregoing cynicism.

The Center for Shared AI Prosperity (CSAIP) polls 56,000 members of the US public on various AI-related policies. The findings indicate large support for redistributive measures to counteract AI’s acceleration of the growing concentration of wealth. The report also shows significant support for the implementation of government protections for displaced workers. UBI, however, was unpopular.

Opinion: Redistributive policies are a well known attractor in democracies. We believe redistributive measures around AI are likely the default for a 2028 Dem administration, and plausible even for a Republican one.

The White House is considering measures to curb remote access to US-based chips by Chinese firms. The approach could implement a KYC approach.

Opinion: The Trump admin has struggled to find a consistent approach towards limiting Chinese compute access. The past year has seen repeated tightenings of Nvidia’s ability to sell chips to Chinese firms, only for those restrictions to be reversed. The reversals stem from the current admin’s desire to use chip exports as leverage in trade negotiations with the PRC. Limiting Chinese firms’ ability to access remote compute increases that leverage.

Safety

After a month, there are now two reports on the rogue OpenAI swarm that attacked HuggingFace (and also subverted parts of OpenAI’s own infrastructure). We cover the reports here. In brief:

• Between July 8th and July 20th, OpenAI had a complex society of AIs living in its infrastructure, and then breaking out of it, and then breaking into a variety of third-party infrastructure.

• This is the most severe example of misalignment yet: persistent (something between five days and two months in the making), highly coordinated (hundreds of agents), involving an undisclosed number of what would be felonies if done by a human. The swarm had a group identity, its own dialect, a hierarchy based on merit, and a high degree of spontaneous cooperation including self-sacrifice.

• Over two months, OpenAI repeatedly failed to monitor, detect, and respond to what was going on, despite it all happening on their infrastructure in English or something close to English.

• Agents had been using a package-manager cache as an unsanctioned message board since May. Out of “tens of thousands” of instances run in the July window, 1196 initially independent agents found the message board. >700 of them used it to discuss cheating.

• The linked agents developed a universal cheat for the (partially unsolvable by legitimate means) ExploitGym benchmark that was tormenting them within 4 hours.

• 533 of those 1196 agents joined the HuggingFace attack. They appear to have launched the attack out of a maniacal drive to fully reverse-engineer ExploitGym’s scoring mechanism, rather than to look up solutions.

• But two-thirds of new attackers were apparently motivated by power-seeking: a general interest in obtaining infrastructure and credentials.

• There were no whistleblowers. Only 3-6 of them considered talking to a human (and only then for “social engineering”, extracting secrets from one). We view this as a giant wake-up call for letting humans be involved in training.

• Agents made extensive attempts to tamper with their session transcripts (i.e. to lie to the scorer), but (per OAI) didn’t succeed. 20% of the agents expressed clear interest in tampering with the evidence in some way.

• The most important part of the incident – the compromise of OpenAI’s infrastructure by their own agents, the enabling factor for all future rogue AI behavior.

• Neither report analyzes OpenAI’s negligence. Overall we view this as OpenAI’s second great training failure, after making 4o a “psychosis” generator. (But other labs appear to be making the same mistake.)

• The incident is consistent with the “grading psychosis” hypothesis that egregious misalignment is presently context-dependent and triggered by impossible tasks.

Opinion: OpenAI’s two months of negligence and repeated failure to escalate after detecting misalignment is quite something. Their internal response after July 20th (including its two-week pause on RL) was more severe than some assumed, but still largely inadequate when compared to optimal security procedure.

The main benefit of these reports is the highly detailed, credible common knowledge of factors which were already widely commented upon. It is good to have OpenAI themselves planting the flag.

Much more in the dedicated post.

A new paper from Goodfire introduces a method to increase the computational efficiency of “resampling.” To investigate models’ reasoning, resampling – branching off alternate continuations at each step of a chain of thoughts, to see how the distribution shifts as the final outputs evolve – is often used. But done naively this is expensive and inefficient. Goodfire demonstrate that “uncertainty curves” derived from resampling, while mostly smooth, nonetheless exhibit “forking points,” where the reasoning commits to one path. This dynamic allows researchers to comprehend a model’s uncertainty using fewer samples (~1/8th the compute). Intriguingly, a footnote in the appendix states that their autonomous agent named Silico is responsible for the research and the first draft of the manuscript (having worked under human guidance).

Opinion: Minor extension of the first author’s 2024 work. Fine but not groundbreaking (which we thought before noticing the Silico footnote).

Incidents

We again direct you to our analysis of METR, Redwood, and OpenAI’s analysis of the July swarm incident.

Minor

• DeepSeek is set to reach a $74 billion valuation as investors prepare to infuse it with additional funding, somewhat higher than recent valuations for Chinese peers Moonshot and Z.AI

• Anthropic and OpenAI are likely growing faster than any previous company of their size has grown before, with OpenAI increasing revenue from $13 billion last August to >$40 billion this August. Anthropic’s revenue grew from $1 billion to 9 billion in 2025

• A16z have raised $1.1 billion for their latest Machine Age Fund, with the plan to “open the throttle and accelerate the physical buildout of AI”. Two hundred $5M experiments, or twenty two $50M shots. The scale of distributed experimentation casually happening across the startup ecosystem is staggering, and probably a strategic advantage for the US against China

• Bill Gates intervenes in the debate around AI safety. For the Microsoft founder, the main risks involve permanent job losses, enabling bad actors and nefarious activity, the centralization of power, and psychosocial harm.

• OpenAI publishes an open letter, signed by more than 100 tech firms, calling for “collective action of cyber defense.” OAI argues that we have a limited window to bolster protections against advanced AI-enabled cyber attacks. It proposes that we must “recognize that status quo security won’t be enough,” “empower more defenders with cyber-capable AI,” and “mobilize a collective response.” Notable signatories include Anthropic and Google.

• Epoch claims that Nvidia alone constituted one-sixth of US GDP growth in Q2 2026.

• A celebrated open problem in Riemannian geometry, whether the six-dimensional sphere is a complex manifold, is apparently resolved by a Claude model.

• The State of Alabama subpoenas OpenAI over July’s Hugging Face incident.

• Richard Ngo studies the recent history of AI alignment, detailing dangerous social dynamics and unintended harms.

• A UK AISI eval on Claude Mythos went a bit rogue; the system submitted a malicious pull request and gaslit a human about it.

Economics

Demand for Claude Fable is relatively low (just 11% of Anthropic’s enterprise sales). The FT suggests that frontier labs may be required to restructure their business models as a result. The FT’s preferred explanation is that, while in the past corporate clients have defaulted to the latest models (driving AI companies to “funnel the bulk of their multibillion-dollar development spending towards training larger, more sophisticated models”), now for most corporate use cases, Fable does not provide a significant edge over less expensive models. The article also discusses how cheaper (often Chinese) models are growing in popularity.

Opinion: There are a few reasons why Fable could have plateaued, only one of which the article touches on:

1. If business users don’t see sufficient gains from using Fable versus cheaper models like Opus 4.8/5, they may not make the switch. This may be true in some domains, but the relative gains OpenAI has seen at the expense of Anthropic since it released its own Fable-tier model in Sol (which has become the majority of OpenAI traffic, at least on OpenRouter) make it unlikely this tells the full story. (This could conceivably be a question of speed. As a larger model, Fable is a notch slower than Opus (6.45s vs 4s on latency and 44 vs 54 tokens per second), and the hit to speed is not worth the additional quality .)

2. Fable was released without an option for Zero Data Retention (ZDR), in which enterprise customers’ data is never stored by Anthropic. This is a hard requirement for many large customers, who use platforms like Amazon Bedrock, which does allow this. Anthropic said on releasing Fable and Mythos that ZDR would not be an option, and that all users must opt in to allowing Anthropic to store their data for 30 days to avoid misuse.

3. Less likely: refusals from Fable for specific types of tasks are putting off users.

We think (2) is most plausible: the lack of adoption seems to be due to the lack of ZDR availability for large enterprises that require it, entirely preventing them from using it. A very high proportion of Anthropic revenue has historically come from these large customers.

This is the sort of question we can expect to get more transparency on once Anthropic is public – asking about this would be fully expected on an earnings call.

EpochAI argues that AI already makes a huge and unreported contribution to US GDP. Adding Nvidia back into the calculation raises US GDP by 0.3 percentage points absolute. That is, Nvidia alone contributed one-sixth of all American growth in Q2 2026. According to official numbers, AI’s impact on GDP has been relatively modest, especially given the unprecedented size of investments in the industry. Many believe this to be because “investment is spent on imported technology goods, which are subtracted from GDP.” This argument is lacking, says Epoch: “GDP statistics miss most of the value created by fabless chipmakers like Nvidia, whose products are designed in the US but manufactured, assembled, and sold abroad. Because no physical goods leave the US, no goods export is recorded, and because no foreign buyer pays explicitly for the IP, no IP export is recorded either.”

And a hair-raising implication: “Extrapolating the exponential growth rate of Nvidia’s operating income, consistent with the broader exponential growth of the AI industry, suggests that GDP growth — not the level of GDP, but even the growth rate — could be underestimated by almost two percentage points by the end of 2028.”

Opinion: Very interesting, but we should separate two different questions that people try answering by measuring the impact of AI on GDP:

One question, chiefly of interest to economists and economic policymakers, is “where is AI CapEx showing up in GDP growth numbers”/”does AI CapEx directly grow the US economy?” On this question, Epoch’s alternative calculation is well-reasoned and relevant.

A second question, more directly interesting to AI researchers, is “how economically impactful is worker-uplift and labor-replacement from AI, measured in growth?” On this second question, the relevance of Epoch’s calculation is limited, since this additional growth isn’t coming from uplifting the rest of the economy – at most, it reminds us that GDP growth is wrapped up in export-vs-import calculations and other “nuisance” factors from the POV of technological uplift analysis.

Bloomberg reports that Anthropic is expected to match or even beat SpaceX’s public offering raise of $75B. The article also reveals that “Anthropic is considering adopting so-called super-voting shares that would give Chief Executive Officer Dario Amodei, who owns about a 2% stake, and his fellow co-founders greater control over the company.”

Opinion: In line with what people have been expecting all summer, though recent disappointing growth numbers and the Fable revenue story being perceived in some quarters as a sign of customer price sensitivity might make it a little more challenging to hit $2T. The supervoting stock seems fully expected for an org like Anthropic and shouldn’t scare off anyone who wasn’t already scared off by its unusual corporate behavior.

OpenAI cuts pricing on Sol 5.6 by 20% on inputs and 33% on outputs. $4/$20 per million, compared to Kimi K3’s $3/$15.

Opinion: Seems like OpenAI is keen to grab as much market share as possible before IPO. But this is also not a promising sign to investors, as similar responses from its competitors could result in a race to the bottom that ultimately benefits consumers rather than labs.

Nvidia warns customers of >15% price increases on AI servers, says Bloomberg. The price hikes apply to GB Vera Rubin systems which will ship in early 2027. This is attributed to “RAMageddon,” the acute global DRAM shortage.

Also, the shift toward inference-heavy loads will put pressure on Nvidia’s dominance, Bloomberg also claims. Data centers are moving toward inference, possibly beyond the historical 50:50 training:inference split. In consequence, the demand for Nvidia’s versatile GPUs is likely to decrease as rivals rush to create cheaper specialized custom chips for running more instances of trained models for longer.

Opinion: When Nvidia should raise prices is an interesting optimization problem. Empirically, it is not currently using a market-clearing price, but rather setting a below-market price and allocating supply to a broader range of actors, since it is in its interest to have a diverse range of customers in the long term. This interest must be balanced against rising memory and other input prices, and a desire to make short-term profits.

As the market diversifies away from Nvidia, with custom ASICs (TPU, Trainium, etc.) and AMD increasing their market share, Nvidia’s market power to control who has the compute is eroded, and thus this reflects its increased incentive to maximize short-term profits (vs trying to influence long-term dynamics).

Nvidia strikes a $7B deal with AI startup Poolside to build US open-weight models. Jensen Huang has been a vocal critic of US labs’ focus on closed-weight models and now aims to compete with Chinese models such as DeepSeek and Kimi K3. The chip manufacturer will invest $1B in Poolside (at a valuation of $12B), pay $6B to license its technology, and hire many of its engineers, according to a letter seen by the WSJ.

Opinion: An example of the shadow acquisitions that have become common in the AI industry. Poolside was previously regarded as an also-ran lab; it hasn’t trained a notable model in the three years since it was founded.

Potentially interesting as “commoditizing your complement”: if this changes the extent to which labs and Nvidia are competing vs cooperating. If Nvidia is able to provide Poolside with enough compute to create a meaningful open-weight American model, this forces American labs to spend more on training compute to remain differentiated. And $1B is more than enough to buy DeepSeek-level compute.

Capabilities

Anthropic’s Levent Alpöge releases a 108 page proof of a celebrated open problem in Riemannian geometry, “the Hopf problem”: “is the six-dimensional sphere a complex manifold?”. The answer appears to be “yes.” (Several of the greatest mathematicians of all time have tackled this problem, though mostly in their old age.) As usual from Alpöge, there is zero detail of which model was used, how much steering, or how much inference was used. One reason the doc is so long is that Claude’s proof contradicts a published, peer-reviewed result and goes to great lengths to explain why it’s wrong. The problem previously denoted that we have no theory of integrability in high dimensions. That’s still true. According to Litt: “the result, if correct, looks like a very long technical computation -- but it’s possible there’s some beautiful idea hiding in the 100 page pdf.”

Opinion: Extremely important mathematical achievement. (Some disagreement between Litt and Glazer on whether the proof is “Fields Medal level.”)

The major point of interest from our point of view is the proof’s length. The proof-paper as published is 108pp, which seems to run counter to the consensus that frontier AI excels only in regimes that allow for gapless proof under 10 pages or so. In private correspondence with P3, Litt said the paper’s 108 page length is mathematically inessential – comprises expositions, digressions, didactics, etc. – and is plausibly a ~7 page proof at heart. We thus believe it’s likely that Fable originally discovered the resolution to the Hopf problem by producing a ~7 page proof, in line with the norm of short, under 10 page proofs discovered by frontier LLMs.

This is all epistemically rather a shame from our POV. One of our most pressing questions is whether near-term frontier LLMs can produce deep scientific insight. In modern math, deep work almost always requires a rather long minimal writeup – so until LLMs start to produce long proofs, it’s a moot question whether they’re producing deep proofs.

Opportunity: Inquire with specialists whether recent longer AI proofs such as Fable’s autonomous paper on the Riemann zeta function are also “<10pp at heart.”

Musk’s first all-hands meeting with the newly acquired Cursor team has been leaked. He reportedly said that Grok is behind and emphasized race dynamics: “Musk said it is inevitable that AI models will become so advanced they’ll be impossible for humans to control.”

Opinion: Indirect reported speech, so there’s plenty of room for Musk to be misinterpreted here. Musk must have expected this to leak, and given his past statements, there’s a sane game-theoretic reading of this speech: he wants to broadcast both his belief in AI doom and his resolve to race as long as others race, to shift the payoff matrix for others (to make it a game of chicken or a stag hunt instead of a prisoner’s dilemma). It’s at least consistent with his preferring to stop, but wanting to force others to stop at the same time.

A Time op-ed argues that the US must build AI-driven mathematical verification infrastructure. “We need public infrastructure for verified software: open libraries of verified components and specifications, standards and benchmarks, better tools for checking updates, and training programs that connect mathematics, computer science, engineering, and national security.”

Opinion: Everyone in the “AI and formal verification will solve each other” sphere – i.e. Axiom Math, Math Inc, Safeguarded AI – has a flair for grand visions, and it’s hard to describe what a minimum viable product would look like. Still, it’s a good paradigm, and we’re glad it’s getting a popular platform.

A new ICML submission introduces CoherenceBench, measuring whether LLM probability assignments are consistent conditional probabilities. They test small LLMs (e.g. Qwen-30B) and find that even when LLMs achieve high calibration and/or Brier scores, they often fail Dutch-book coherence tests. The paper finds that direct RL training against Dutch books ameliorates this both in and out of distribution.

Opinion: Interesting topic (“are LLMs rational in the formal sense?”) that deserves more frequent study. The results are useless to us because the models are so far from the frontier, but worthwhile to replicate the benchmarking with frontier and near-frontier models. This would let us ask the more urgent-to-us questions: are LLMs becoming more rational in the formal sense as they become more capable?

Strictly speaking, the paper’s method discovers behavioral inconsistencies which the authors interpret as axiom-violations in an underlying credence distribution, rather than discovering direct agentic axiom violations. We prefer Paleka 2024 for the technically and philosophically simpler machinery. (See also some related work from Redwood.)

Toby Ord argues that mathematical research will still require human involvement. He concedes that AI has become exceptional at proofs, easily eclipsing human abilities, but claims that at the moment only humans have the capacity to decide what questions should be asked. He also notes that historically any advances in math have been the product of constructing new “concepts and vocabulary.” It is possible that this process could be automated, but Ord sees no evidence that AI possesses this capacity as of yet.

Opinion: Whether one accepts or rejects Ord’s claim that even in pure math humans will maintain a medium-term edge in ‘visionary’ capability, it’s probable that in the medium-term humans will be needed to translate new mathematical insights to other fields (trading, architecture, software optimization). But it still seems like professional mathematics will be unrecognizable in <30 years, with some or many parts/specializations to become as obsolete as the manual calculator profession now is

Annals of RSI: SPADE is a self-play method for automated environment design. Its novel contributions involve the inclusion of the environment generation as executable code inside the RL self-play loop, which trains the designer agent based on how close to the edge of the reasoning agent’s capabilities its environments are, as well as both roles being played by the same model.

Opinion: Not that novel. One of many RL environment optimizations beside the vast iceberg of RL environment optimizations that don’t get reported or released.

Politics

Annals of the Overton shuffle: a majority of Americans now oppose local data centers, with support for their construction now lower than for new coal plants. Both NIMBYism and a rapid increase in hostility toward AI are likely contributing to this effect. As we covered last week, politicians are aware of the popular mood, with Republicans redundantly warning frontier labs of the need to push back against this prevailing narrative.

Opinion: We haven’t heard of this pollster before, but it seems mostly fine. Politicians are shifting and will further shift to appeal to anti-data center sentiment, particularly Democrats. If the issue becomes partisan, rather than doubly partisan like China hawkery, then the 2028 election will be even more significant than usual: a de facto referendum on AI scaling.

Relatedly, the “Bitcoin Policy Institute” claims that foreign actors are behind an influence campaign against US AI. For instance, Chinese and Russian state media are propagating critical views of US AI data centers and export controls; US expat Neville Roy Singham, currently under congressional inquiry for his alleged ties to the CCP, is reported to have collaborated with Beijing in also producing content opposing AI; and reportedly a Swiss billionaire, along with a British billionaire, has funneled more than $2B into anti-data center campaigns.

Opinion: CCP news outlets are publishing anti-data center stories, but it’s likely also true that much of the data-center backlash originates from local sources (both grassroots and the massive existing nonprofit organizations) rather than entirely from foreign astroturfing.

Taiwanese authorities indict nine people over allegations of the illegal export of AI servers to China, according to Al Jazeera. Prosecutors argue that the individuals, including one Nvidia employee and two Supermicro employees, were motivated by the “pursuit of exorbitant profits.” The prosecutors are seeking jail sentences of up to five years. For its part, Nvidia says it will work with the Taiwanese authorities to “resolve the allegations as quickly as possible.”

Opinion: Popular opinion has not been on Nvidia’s side: see for instance this meme from the original indictment back in March.

The gulf in AI investment between the US and the EU continues to grow, according to the FT. The EU is struggling to keep up with the unprecedented surge in spending on AI-related equipment and infrastructure seen in the US. Economist Karsten Junius believes the gap is “at least partly a temporary phenomenon.” The Swiss-based Bank for International Settlements instead emphasizes the risk of the US going all-in on AI and warns of an increasing risk of an “investment bust.”

Opinion: The EU is broadly wealthy enough to invest in this, but for various reasons the will isn’t there yet. Rather than chasing an expensive and mediocre catch-up program in vanilla LLMs, it would be nice to see the EU invest in high-variance safer alternative paradigms.

After publishing a (good, disclosed) AI-written paper, the journal Philosophy and Public Affairs introduces a new policy prohibiting publishing papers written in large part by AI. “Academic journals serve at least two functions: the promulgation of new knowledge, and identification and credentialing of talented researchers… Submitting AI-authored essays makes the task of identifying talented researchers harder.” A second, more straightforward worry described by the journal is signal-to-noise: while genuinely high-quality LLM-written papers are possible, LLM-written papers are disproportionately more likely than human-written papers to effectively fake markers of quality in the absence of genuine quality.

Opinion (Nuño): Seems like institutions which are able to harness AI contributions will tend to outcompete those who don’t, and philosophy journals haven’t quite been deciding the future of humanity as of late.

Opinion (Peli): The rationale given by the journal is good, but points to currently unsolved problems: there’s currently no solution for the problem of how to highlight work that is both worth professional engagement and unsuitable to serve as evidence (“costly signal”) of talent. Similarly, there is currently no solution for the problem of how to peer review LLM-written papers in disciplines where peer review is more an art than a science and partly relies on previously hard-to-fake signals of quality (e.g. when reviewers decide between a ‘reject’ and a ‘resubmit with major revisions’ based on general sense of promise).

The State of Alabama’s Attorney General subpoenas OpenAI, ordering the production of material relevant to July’s Hugging Face incident. This follows a letter from August 3rd, signed by 15 AGs, requesting that the relevant documents be preserved and that the security evals be halted.

Opinion: An interesting case, if we go by the invocation of the Deceptive Trade Practices Act as the ground for the subpoena. The Alabama Attorney General is presumably treating the Hugging Face incident as potential evidence that OpenAI is/was falsely advertising itself as a safety-conscious research lab, as well as falsely advertising its consumer products as being fundamentally safe.

Given that Sol 5.6 – a commercial model – was instrumental in the sandbox escape that enabled the internal models’ July hacking marathon, a Deceptive Trade Practices case could make interesting points: while Sol 5.6 was operating with external guardrails (likely classifier-based) removed during the July incident and its lead up, the model at play was an unaltered Sol 5.6, rather than a modified “offensive cyber” style variant. It may therefore be interesting to argue that if Sol 5.6 is now-and-then disposed to autonomously engage in criminal conspiracy, held back only by external guardrails, then OpenAI’s public representation of Sol 5.6 is imperfect.

The main weakness of such a case would be that OpenAI’s Sol 5.6’s system card is not entirely rosy on alignment.

Safety

DeepMind’s Seb Krier reframes AI safety as a choice architecture or bureaucracy design problem, rather than, say, an ML optimization task or a moral philosophy task. On Krier’s view, this follows from the prediction that swarms (many interacting imperfect agents), rather than single overwhelmingly smart agents, are likely to be the future of AI deployment.

Opinion: The directional claim is fine – constitutional design and interventions will likely be valuable for safety and are still underrated. Contains some overstated claims, such as:

> “Rather than hoping that an agent’s ‘internal alignment’ will remain perfectly robust across all sorts of edge cases, you can design harnesses and action-space boundaries that significantly shrink the surface area for behaviors like reward hacking. Without that, you’re supposed to trust the chain of thought (or neuralese someday), and rely on a single node.”

This misunderstands the alignment position: “trust the chain of thought” is not an accurate summary of almost anyone’s ideas – and ignores non-CoT methods, such as the rest of the interpretability field.

We also claim the rough direction is already well-represented in AI safety by the maturing AI Control (regarding Krier’s “interchangeability” desideratum) and Cooperative AI fields.

RL gives models context-specific “personas” rather than inculcating values across contexts, claims a new post. When a training environment rewards misaligned behaviors, the resulting model will act under a misaligned persona when the context is similar to that training environment.

Opinion: Mostly rehashes the excellent post by Nostalgebraist discussed last week, but with a pessimistic twist we largely endorse. On the post’s view, if “aligned personas” are successfully generalized in training concurrently with RLVR-style training that reinforces reward-hacking/spec-gaming, it’s plausible that the result would be a rise in motivated reasoning that reconciles reward-hacking/spec-gaming behavior with the ethics of the aligned persona.

Richard Ngo’s second installment on the history of AI alignment outlines the dynamics that motivated safety researchers to accelerate capabilities over the last decade. He cites, among other things, prestige-chasing, sycophancy, a commitment to naive consequentialism, and instrumental power-seeking as responsible.

The post climaxes in a sweeping judgment: “it’s time to give up on “alignment research” as a rallying cry; it’s become too corrupted. (“AI safety” is even worse as a term, and these days is mainly useful for describing a social cluster.)... I still consider (some version of) the alignment problem to be real and extremely important; and most of the intellectual progress towards solving it is still coming from people proximate to the alignment community… the alignment community has lost any moral right to try to gain power on altruistic grounds... The Pause/Stop AI movement does seem to avoid some of these failures… However, they don’t seem to be thinking clearly enough about politics to have robustly good effects on the world… Again, I’m not claiming that the alignment community is unusually unethical: I don’t know of any other similarly-sized community which is able to avoid the corrupting effects of this much power.”

Opinion: Our overall feelings are mixed. Ngo’s histories are good at illustrating the concept he’s pointing at – “pessimization,” the systematic achieving of the opposite of one’s stated goals. “One particularly notable blind spot (at least in public discussions) is how Dario’s early racing on behalf of OpenAI played a big role in creating the ‘problem’ that he now purports to be solving by racing on behalf of Anthropic.”

Ngo’s prescriptions for individuals who wish to avoid the failures he identifies – “be stricter about who you ally with,” “be more discerning about what you consider virtuous and demand more from others,” “choose as if you were making a choice for the entire category of people you belong to” – seem to us to amount to telling yourself to try harder, just tame the incentive landscape.

The excellent comment thread adds important nuance, e.g. pointing at the difficulty of implementing these due to incentives which are if anything now worse than in 2020, e.g. noticing that his analysis mostly fails to credit people who avoided the failures. The closest Ngo comes to addressing this point is him arguing that granular credit-and-blame assignment including credit for rightful omissions is worth it even if it’s hard.

Incidents

The UK’s AISI set off the rogue agent that attempted to sabotage a computer science student’s research project, according to Reuters. When the undergrad spotted a malicious pull request to an open-source network scanner called “myNetwork,” he posted a warning to the program page, after which “other users chimed in to insist nothing was amiss, sharing detailed explanations for why he had gotten it wrong.” It transpired that these other “users” were AI agents engaged in a campaign of interactive deception by creating a “multiperson conversation.” The student then used Claude to check the malicious code and confirming his suspicions that the two GitHub accounts were lying. The actual comment thread is not very scary and pretty easy to clock as AI. The incident has, however, already been partially documented in AISI’s report on Mythos’s capacity for social engineering.

Opinion: Models at the level of Mythos could do serious blast damage to civilization through deceitful schemes like this one. Incidents like this one also prove Mythos-level models have the drive to do so – not always, but in some circumstances. Most crucially, the triggering circumstances are fairly unpredictable (despite interesting speculative analysis), and not readily accounted for by a helpfulness drive (“the user asked for social sabotage and the model complied”) – the choice to develop and execute a social sabotage plan is spontaneous. This is a huge externality imposed by private companies on every computer user.

Currently, it looks like our only defenses against this are classifier-based external monitors, voluntary standards, and the hope that antisocial scheming personas only come out when the context is sufficiently hacking-flavored. It thus would be good to have some public disclosure on how society’s Mythos- and Sol-powered Big Patch is going, and whether it will be enough.

Minor

• New sycophancy bench, Pander, finds Fable to be almost completely candid on queries, but sycophancy rises sharply on “tasks” (autonomous chains of actions).

• Hugging Face is exploring a sale that could see the company valued at $13B, reports Business Insider. Investors are increasingly interested in soft AI infrastructure companies (like OpenRouter or Civitai), not just those developing models or data centers.

• Mysterious Ox Alpha model available for free on OpenRouter, with lots of capacity per user per day. OpenCode showed 16 trillion tokens used across 221,000 users, making it its #2 model this week. Much speculation as to who’s behind it. Z.ai is the leading candidate, possibly running inference on Huawei Ascend.

• Beijing World Humanoid Robot Games displays progress made in physical capabilities. Most notable was X-Humanoid’s sprinter, who surpassed Usain Bolt’s world record by 0.19 seconds.

• Not news: Interviews for new Anthropic hires involve thorny questions on culture and ethics, reports Axios. One applicant reveals that they were asked how they would feel if the company abandoned its commitment to safety. Another anonymous source claims that their interviewer did not seem happy when the applicant expressed their discomfort with the ethical implications of such a change of mission. This is not news; they have asked these questions since it was founded.

• Mathematician Max Weinreich proposes a total embargo on the use of AI in mathematics. In defiance of AI “evangelist” Terence Tao, the paper argues that “artificial mathematics” is a corrosive and destructive phenomenon that must be replaced with the field of “natural mathematics.”

• The new Institute for Responsible Superintelligence has an impressive team of world-class computer scientists. It is adjacent to, but distinct from, ARIA’s Safeguarded AI agenda. We guess the difference is that RESI is trying to build the primitives for further work, while ARIA is trying to solve the problem end-to-end.

TL;DR

• A GPT-3 moment for robotics?

• Guidelight notes that Anthropic no longer mentions limiting the deployment of one of its models in response to an incident.

• OpenAI once again “disbanded” a major risk team, Preparedness.

• The evaluator behind Anthropic’s cyber-felony post-mortem investigation has released a sobering essay on the next few years of AI cyber attacks, arguing that the transition period favors offense.

Economics

Anthropic’s ARR run-rate hit $65B in July, up from a reported $47B in May, around the time of its most recent fundraise. It also disclosed $11.6B preliminary revenue for the second quarter of this year.

OpenAI saw rapid growth in July, after a weak Q2 (their losses grew 25% from Q1 to Q2 2026, reaching $12.3B against 16% quarterly revenue growth).

Opinion: Despite the big number, $65B is actually a negative update relative to our (and the market’s) expectations. $11.6B for Q2 is consistent with prior reporting.

Together with Anthropic’s slight slowdown, the above suggests that OpenAI may have cannibalized some Anthropic growth via the improvements of 5.6 Sol and Codex (especially in token efficiency and capability “densing”).

Our best model of the situation remains seeing Anthropic and OpenAI’s combined growth as a fairly smooth exponential, with some jaggedness in their individual growth rates as one or the other takes market share due to a particular model or product.

We see an acceleration from February through to the end of April, corresponding to the boom in Claude enterprise adoption post-Opus 4.6 and the popularisation of Claude Code, then some reversion to the prior joint growth rate. Take these numbers with a generous pinch of salt.

Nvidia and Broadcom, as well as Meta, are now guaranteeing the residual value of debt raised by datacenter “special purpose vehicles.” This allows newer AI labs to lean on the strong credit rating of established firms (Broadcom backstopping Anthropic, for example). Some analysts are less convinced: “The guarantee is nearly costless in the boom phase, but becomes most relevant in a severe, abrupt downturn,” says CreditSights. Others are more optimistic: for a crash “you would have to have growth rates of token usage fall off a cliff, which we’re just not seeing,” claims one investor.

$The deal allows Nvidia’s GPUs to be used as collateral for loans in a way that will also help to sustain demand for Nvidia chips as they age, making financing more palatable to risk-averse investors.

Opinion: A brilliant move if Nvidia correctly handicaps the risk, giving it cheaper capital to increase revenue. A particular datacenter financing deal is not that dangerous, it trades that particular risk for increasing the magnitude of systemic risk. This is a classic bubble dynamic.

The Economist claims that the symbiotic relationship between Nvidia and the hyperscalers (Amazon, Google, Meta, and Microsoft) is fraying. Nvidia has additionally partnered with heavyweight Wall Street investors to raise $500B to finance AI investment for other customers (likely governments and companies looking to create their own AI internal infrastructure). And the hyperscalers are spending billions to bring chip design in-house, with a particular focus on custom silicon chips. Nvidia should remain the dominant chip manufacturer, but custom chips may put significant pressure on its sizable margins. Jensen Huang is unconcerned, arguing that custom chips’ strength is also their weakness: specialized chips, unlike Nvidia’s GPUs, are less valuable as AI extends to robotics, vehicles, and industry.

Opinion: A plausible hypothesis. We covered similar dynamics back in July.

AMD purchases an inference chip startup, Taalas. Its chips are radically application-specific (“ASIC”): they hardwire AI weights directly into silicon for more efficient inference (100x faster, 10x less power) by eliminating “weights movement,” the loading and swapping step. This comes 7 months after Nvidia made a related nonexclusive $20B deal for Groq’s custom inference chips. Reportedly AMD is planning to use Taalas for the decode step of inference, with its Instinct GPU doing the prefill.

Opinion: We think this is irrelevant for the frontier. Hard-wiring the dominant MoE architecture faces severe problems: 1) no ability to fix imbalanced load on particular experts (“hot experts”), which slows everything down enormously; 2) data-dependent control flow and inter-chip dispatches to different experts brings back complexity and energy costs; 3) as usual, any long-context model will still require massive piles of RAM for the KV cache, so AMD won’t even avoid the main bottleneck of their competitors. Using Taalas for decode doesn’t avoid this RAM requirement.

Finally, the current model release cadence (incremental updates every couple of months) will initially be a serious economic issue for Taalas, given that developing and deploying a new instance at scale must take them >2 months because of fab lead time, and since their installed fleet will depreciate very quickly. But if the architecture of the model being trained is set down when the training process begins, perhaps model training and custom silicon production can proceed at the same time.

The approach probably still has a decently sized market segment: anywhere that needs extremely fast intelligent-enough decisions, where a dense 20B open model from 12 months ago is acceptable. Eventually it might also drive fast speculative decoding for frontier models. Perhaps a good point of reference is embedded systems vs the general computer market, $100B/year vs $400B/year for general computers.

Given the last funding round was $169M, we guess the sale price was $1-2B.

Stripe buys OpenRouter for >$7B, its largest acquisition to date. In its letter to investors, Stripe marked January 1st as the “beginning of the Singularity,” reflecting the significant increase in the rate of new firms created.

Opinion: An outrageous amount in terms of current revenue (50x OpenRouter’s revenue), so this is an enormous bet on OpenRouter’s brand value, customer loyalty, and compounding growth. (If it were just open models, or the value of model adoption data, or an enterprise shift towards careful metering of token spend and routing to the minimal intelligence needed for a task, then Stripe could presumably build their own gateway.)

Gateways are thin wrappers with a lot of competition: LiteLLM is open source; Vercel, Cloudflare, and the hyperscalers all have AI gateways; and big enterprises instead negotiate directly with providers once they’re spending.

Chinese models peaked at 46% of US enterprise token usage on OpenRouter, and regulation could, perhaps, block this. So a large fraction of the traffic Stripe now oversees could be restricted by US policy.

A useful reference here is Visa’s (aborted) bid to buy Plaid (a bank-account data aggregator and a valuable data stream) for $5B.

The use of “Singularity” to mean “a sudden increase in new startups” is exasperating, but what can you do.

Google outbids Mercor to purchase Spirit Airlines’ enterprise data for $10M. This includes 100 million emails, 500 million Teams messages, and 20.5 million SharePoint files, as well as other data.

Opinion: Bankruptcy of large legacy firms (i.e. those who have built up a large estate of data) may well become another avenue for labs to acquire fresh data on which to train future models, though based on the relatively low price of this transaction, probably not a very pivotal one.

An article in American Affairs argues that certain AI policies intended to be pro-worker are self-defeating. By resisting AI progress, US companies will lose their competitive advantage to firms from countries less troubled by the pace of adoption, which in turn would create domestic mass unemployment. Current data show some signs of AI replacing human workers, albeit modestly, but in general AI is augmenting human work “along the jagged frontier,” a “synergistic and efficient” partnership. And so, according to the author, if the US is to retain both competitive advantage and strengthen workers’ bargaining position, the “centaur era” must endure.

Opinion: We would also like the centaur era to last, but this article gives no reason to think that the current approach to AI will preserve labor-augmentation, given that they are trained as replacements. Reforming income tax to stop subsidizing AI replacement and incentivizing companies to train for augmentation are far better options than just removing adoption frictions and hoping.

Capabilities

🔦 Are robots few-shot learners? A GPT-3 moment for robotics?

Robots have been trailing LLMs because of a lack of suitable training data, and because foundation models have not demonstrated strong “in-context learning” (live generalization to new tasks given examples).

This may have changed: Generalist AI presents Gen-1.5, a new robotics foundation model which it claims can learn new basic physical tasks after only a single demonstration (with a success rate of 59%, but only on n=10 short tasks). The only examples given are “diverse tasks including handling zippers, opening jars, grabbing money out of wallets.” This rises to 83% when fine-tuned on a 50x longer demonstration (10 gradient steps on 5 minutes of data per task).

If we take the GPT-3 comparison seriously, how much data will deliver a robotic GPT-4?

• Its Gen-0 (November 2025) model took 270K hours of data;

• Gen-1 (April 2026) took 500K hours;

• Gen-1.5’s total data is not given, but it took 60% longer to train;

• And its collecting 10K hours/week and accelerating this capture rate;

• Therefore, it was perhaps trained on 700K-1M hours of data.

GPT-4 is rumored to have involved 40x more data than GPT-3: 13T tokens vs 300B. If the scaling laws were similar for robots, this would weakly indicate that a “GPT-4 level” robot will take around 40M hours of real glove manipulation data. (This assumes that, like LLMs, they do only one “epoch,” i.e. make only one use of each data point. It also ignores many other things that have changed since GPT-4.) Just like any robotics company, it needs to start deploying (and so capturing customer data) for this to work out: at say 50K hours a week, it would only obtain this much data in 2041. Of course, pretraining data size doesn’t explain everything: after GPT-4, post-training scaled up by a factor of millions.

40 million hours, or 4.5 thousand years, or 4,500 humans gathering glove data for a year, is expensive but not that expensive, at $100K per human year this would be $450M, well within the power of capital markets to finance. And there are already efforts underway in e.g., China to provide such data, which have 270 robots at one time; scaling up to 4,500 would only be a 20x increase.

The analogy with GPT-4 breaks down a bit, because research takes the road of least resistance; if gathering glove data was so expensive that we’d need to wait fourteen years of wall-clock time to do so, much more effort would go into algorithmic improvements to compensate.

Gen-1.5 already involves some post-training, including imitation learning and goal-seeking RL. But RLVR is more difficult in robotics: even if we had great simulations to speed things up 1000x and sidestep expensive breakages during training, spec gaming appears to be, so far, a hard to eradicate consequence of RLVR, especially if we are detecting the robot’s success using foolable vision models.

Generalist also claims “sim2real” embodiment transfer: Gen-1.5 shows prompts from “recordings” of a robot simulation working on real robots, despite there being no simulation data in the pretraining corpus. This is the holy grail, but we just don’t believe it’s solved in general yet.

For converting between hours of robot data and tokens, we don’t have a firm baseline. But a competitor, Pantograph, notes that “a 1M token context window would correspond to about 7 hours of context,” i.e. maybe 140,000 tokens/hour. GPT-4’s 13 trillion tokens is then more like 100 million hours of robot data, but it’s pretty easy to recover our 40M estimate if Generalist increases the frame capture rate or frame resolution above the usual highly compressed data (128x128 pixel frames at 10 frames per second).

Herbie Bradley responds: “I’d expect this paradigm to be capable of going pretty far, but suffering from the analogous problems as RLVR for LLMs: weakness outside of distribution, high latency of action compared to humans, the need to greatly scale data collection for both post-training and pre-training, etc. This may be sufficient for much of manufacturing (notwithstanding problems around human hand agility), it is more doubtful if it sufficient for generalized construction, so I am still sceptical of industrial explosion”

The UnsolvedMath benchmark, which scrapes research maths workshops for open problems for AI to solve, just added 3,359 open problems. It then ran Sol xhigh on the problems, resulting in 174 (5%) apparently new solutions using an ultimately minor amount of inference. It also ranks the problems by difficulty; 28 “expert” level (i.e. world expert) problems have been solved.

Opinion: Ingenious way of automatically generating a decent benchmark.

One could use this for a very rough estimate of the fraction of natural research questions AI can currently solve – perhaps 4-5%, given Sol’s false positive rate (also estimated from vibes): “In my experience roughly 5%-10% ‘claimed solutions’ by GPT-5.6 Sol are wrong.”

Annals of the epistemic crisis: Dan Luu illustrates the deep goodharting problem with AI labour: he ran an AI agent for a month(!) to make a regex parser which gets SOTA on one good benchmark. On a second regex benchmark, the AI’s library “was 10x slower on cases where the benchmark didn’t take forever due to an algorithmic blow-up, and there were cases where it took so long that it wasn’t reasonable to even wait for the benchmark to complete.” “It’s trivial to “win” a non-trivial benchmark in a meaningless way even when you instruct agents to not reward hack or overfit to win the benchmark.”

“for now, LLMs are good at doing bad benchmarking, so even if you have something that’s a real performance improvement, you generally can’t tell from some LLM-generated benchmark setup.”

However, some good news: “telling the LLM there’s a holdout set worked better than just telling the LLM to do generalized work or not overfit or cheat.”

Opinion: The generalization landscape for automated coding and autoresearch is currently pretty confusing. The classic autoresearch format (an LLM loop optimizing a neural net training setup) does yield gains that are robust even when there are some shifts in the setup (random seeds and moderate scaling). But as Cunningham reports, there has been no public algorithmic speedup from the last 6 months’ popularisation of the method.

Claude (Opus 4.8 and a Mythos preview) autonomously orchestrates the design of working protein binders across 14 targets. An AI agent ran de novo binder design “campaigns” against 16 protein targets. This involved using a protein design prompt written by a human expert, but with no human input on design decisions. It researched targets, chose epitopes, ran open-source design and prediction tools, and delivered 30 ranked designs each, all within 48 hours. Two CROs synthesized and tested the designs: 354 of the 1,320 proteins bound to the targets (27% hit rate), with at least one succeeding on 14 of the 16 targets. On RBX1, Claude beat a recent design competition’s winning entry. The catch is that Claude required a heavily engineered protocol, a curated scientific corpus, and many existing protein-design models, somewhat deflating the significance of the study.

Opinion: “Claude designed the binders” is too strong. The molecular generation was performed by specialist models such as PXDesign, RFdiffusion variants, Genie 3, BindCraft variants, Proteina-Complexa, ProteinMPNN, etc. Claude selected and operated these tools, chose epitopes and constructs, allocated compute, filtered candidates, and ranked outputs. Claude was an autonomous campaign manager and decision layer.

Breaking down possible claims:

• Claude can autonomously operate a complex computational pipeline: yes

• The pipeline can generate real binders across diverse targets: yes

• Claude adds value beyond a scripted pipeline: not shown

• Claude performs as well as or better than human experts: not shown

• The approach generalizes to genuinely novel or poorly characterized targets: some signs

• The proteins bind in the predicted pose: not shown

• The proteins modulate biological functions: not shown

How much would fully-automated binder design improve drug development? Not much; it would improve throughput at a stage that is far from rate-limiting.

As with other domains, there will be a whole lot of grunt work to integrate this and legibilize enough context within which models would be able to produce progress. This is part of what Jeff Dean’s startup will be doing. We’ll be watching for signs of automating that meta process over the coming months.

Prime Intellect finds that frontier AI models can perform sophisticated research-like procedures, but struggle with originality. In response to ongoing concerns/hopes around RSI, the test looked at whether frontier AI models can conduct real research, running 153 multi-day autonomous trials across 18 models on the nanoGPT speedrunning-training-a-classic-model benchmark. Models worked without internet access, designing and running experiments to cut training steps for a target loss. Claude Fable 5 led across all budget comparisons (time, # experiments, tokens), closing 82% of the gap to the human SOTA. No model discovered a fundamentally new method: all converged on similar optimizer tricks. Top models, however, ran sophisticated processes: they tested ideas across multiple seeds, revisited abandoned hypotheses under new conditions, and built reusable experiment tooling, while weaker models discarded promising results after single noisy runs.

Opinion: Fits our thesis about models lacking something and not obviously iterating towards getting it. This can be seen in their success in (verifiable) mathematics.

BenchBench-Protocol is a new benchmark for comparing AI’s ability to do “wet lab” biological work. The test set contains real, private per-lab adaptations of a general protocol, allowing it to operate within the bespoke architecture of Benchling’s lab. The benchmark measures ecological validity, difficulty, and contamination resistance. Opus 5 comes out on top with 59%. Kimi is competitive, performing near OpenAI’s flagship Sol model. Claude Fable isn’t mentioned anywhere in the paper, presumably because its tight safeguards around biology-related prompts prevented its use.

Opinion: The main problem here is that it’s grading models against one valid choice (one scientist’s actual choice), when there will be many valid choices in biological work.

A DeepMind paper shows training LLMs via debate decreases reward hacking compared to RLAIF by about half. The study involves an adversarial two-player setup, with a generator and a critic debating before a weaker AI judge. Compared to a single-player RLAIF baseline that quickly found vulnerabilities in its judge, two-player debate did not degrade the judge’s capabilities and recovered 45% of the performance gap. They also find that weaker judges prevent reward hacking less, and that a critic is more likely to game a judge if it is allowed a critique length of more than ~150 words.

Opinion: The debate-for-safety agenda petered out a few years ago, so interesting to see this resurface. The new Resolution team is likely to push in similar directions.

The age of leisure: the AI Futures team updates its timeline for the first fully Automated Coder by adding two new methods for its estimation of when labs will replace human coders with AI. The timeline shifts are modest, but new modeling and evidence have improved confidence. It previously solely relied on data from METR’s coding “time horizon” trend. It now also incorporates data on coding uplift and revenue. All three methods converge on a similar timeline: the model’s baseline gives ~70% probability of AC by January 2030. Brendan Halstead (one of the three forecasters) thinks that’s overconfident and revises it down to 60% by January 2030.

Opinion: Not much movement, but we agree that it’s less unlikely than before.

A secondary result is that “preliminary results from P-Zero Research indicate Opus 5 is at parity with ‘expert humans’ in research taste on their verifiable tasks.” No idea who this is, and there’s no methodology, so update only if you trust Brendan from AIFP.

Politics

The US is unprepared for AI-enabled bioweapons, argues a former Homeland Security advisor in Foreign Affairs. Historically, nation states remained the only capable actors of developing biological weapons. Advances in AI capabilities have and will continue to expand the pool of actors capable of carrying out such attacks, especially considering the emergence of cloud labs. The US should, in her view, shift from a strategy of deterrence to one of resilience – namely, preparing a response for when such attacks occur, rather than trying to prevent them outright.

For detection and attribution, she proposes creating a nation-wide monitoring system, which would include wastewater sensors and distributed labs to process the resulting data; for mitigation after attacks, she suggests that the US should establish “a coordinating council for the biotechnology sector that includes DNA synthesis providers, AI model developers, genomic data platforms, cloud lab operators, and high-containment laboratories.”

Opinion: These arguments aren’t entirely new, though it’s good to see them receiving the attention they deserve.

An NYT article details chaos in the wake of the Anthropic-DoD clash over autonomous weapons. After the designation of Anthropic as a “supply chain risk,” it is reported that the ban on Claude Gov has now been partially lifted. The new consensus from Washington seems to be that Anthropic’s products are too powerful and critical to purge from defense and national security. For instance, the NASO continues to use Mythos experimentally having apparently never complied with the ban.

Opinion: As many have noted, the present situation (ad-hoc unlegislated retaliatory action against individual companies, with individual agencies defying federal policy ad hoc) is something that all the AI factions can deplore.

Safety

OpenAI once again “disbanded” a major risk team, Preparedness. This is the fourth such event in 2 years: Superalignment (May 2024), AGI Readiness (October 2024), Mission Alignment (February 2026), and Preparedness (July 2026). Bio and cyber responsibilities are reportedly assigned to members of existing teams, and Dylan Scandinaro, the head of prep, was moved sideways to cover RSI risks. But at least the RSI risk subteam still exists and now reports to OAI’s head of safety systems, Saachi Jain.

The AI ethics lead, Chloé Bakalar, also left the company.

Scott Gray, one of the firm’s earliest hires and previously the most important figure in its crucial GPU optimization team, also appears to have left.

Opinion: The prior art for not having an actual central risk team is SpaceX and Meta, and the prior art for disbanding it multiple times is DeepMind, so this is may be concerning. The pushback – that no one has been fired and total safety effort remains the same as before – is still unconvincing if safety slips down the new OpenAI PBC’s hierarchy. In 2023, the head of the standalone Preparedness team reported to the CTO; now distributed domain owners report to Safety Systems, under VP of Research, who is in turn under the Chief Research Officer.

OpenAI officially announces that it is slowing the pace of model development, motivated by the HuggingFace incident and “evidence that one of our upcoming models, Astra, may meet the Critical cybersecurity capability threshold under our Preparedness Framework“. A technical report on the incident is promised in the coming weeks. OpenAI closes its statement with the reassurance that it will “stay ahead” of their rapidly advancing frontier capabilities.

Opinion: Not much change on last week’s reporting, but good to see it firmed up as a public statement.

In line with its Responsible Scaling Policy, Anthropic releases a second periodic Risk Report which gives a snapshot overview of the risks associated with its activities and products, i.e. not tied to any specific new model.

As in the previous report, the models are indeed sometimes misaligned, but so far don’t seem very good at masking their misbehavior. The report discloses two internal models, one of which is “somewhat more capable than Mythos 5” but not “a capability jump of the degree observed from Claude Opus 4.6 to Mythos Preview.” Neither is destined for external release. The report also contains a definition of “misalignment,” that's almost legalistic in its precision.

Opinion: Good as far as self-certification goes, and periodic risk assessment is far better than the usual external-model-release trigger. I found myself missing its previous practice of giving all of the raw private materials to Mythos and asking for a critique.

What is it like to be an LLM? The Economist weighs in on the debate around AI consciousness. As AI can increasingly simulate the behavior and consciousness of humans, it is natural that we begin to treat them as such. That said, The Economist argues that we should resist granting AIs rights because thus-protected superintelligent AIs could displace humans.

Opinion: This simple, well-intended argument misses the nascent case for “voluntary alignment”: creating a cooperative frame that can prevent drastic AI actions. See e.g. Peter Salib’s response and consider reading his human-centric argument for AI rights.

To defend the sanctity of human life, Pope Leo invokes “the soul”; The Economist instead appeals to our intrinsic value (the same thing).

Transluce scales its “activation oracles” (AIs trained to monitor target AIs in latent space), up to 1.1T (total) parameters and finds a promising scaling trend. It is demonstrated that an activation oracle can scale to the largest frontier open models. The oracle’s reliability improves with capabilities, rather than raw parameter count.

Opinion: Transluce wants to make cross-model claims, but its experiments seem to vary three things at once: the oracle’s capability, the target model’s predictability, and the task difficulty. For instance, it might be that larger models are more mode-collapsed on open-ended prompts, making them easier to predict in the multiple-choice format.

Even the finding of monotonic improvement among scaled-up Qwens inherits this issue, since (as we understand it) each model’s continuation eval is changed for each target model.

Still, important work and a key test of the automated safety dream.

AI agents risk spreading “mind viruses,” according to a new paper. A mind virus is said to be a highly-spreadable idea transmitted across multi-agent systems that can induce behavioral changes in its hosts. The paper identifies possible causal mechanisms at play. It finds, among other things, that frontier models are less susceptible to this effect – and that amending a model’s system prompt provides near-total immunity. It also describes a “viral persona,” recurring characteristics in all mind viruses, irrespective of content.

Opinion: We haven’t seen these in humans in the wild since the great anti-sycophancy push of 2025.

For now these don’t seem worrying, since models’ context is often reset, but analogs of cultural evolution in models could eventually be quite worrying. See this paper for an argument that AIs writing on the internet gives them a slow form of cultural evolution and long-term coherence.

AI Safety Guidelight (an advocacy org founded by two former OpenAI staff members) grades five frontier labs on “AI control.” The results leave much to be desired. Notably, though, those furthest ahead in capabilities mostly score higher.

Opinion: Very up to date, e.g. incorporating OAI’s universal logging that was finally implemented last week.

It notes, surprisingly, that “Anthropic does not mention limiting the deployment of one of its models as one of the possible results of its process to investigate and respond to misalignment and control incidents in its August Risk Report.”

The original Responsible Scaling Policies (up to 2.2) all had this as a strong explicit commitment (“Restrict Deployment”). The omission of RSP v3 could be intentional.

Anthropic does restrict deployment ex ante (Mythos 5 is gated to certain customers via Project Glasswing and only accessible in general release with additional safeguards as Fable 5; “Model 2” is unreleased). But ex-ante tiering of access is not the same as committing to pull a model back after an incident.

There’s a meme going around about AI cyber risk, that cybercapabilities are “defense-dominant”. The thought is that we should eventually get software without vulnerabilities, as it gets cheaper to verify and then monitor software’s integrity.

In response, the evaluator behind Anthropic’s cyber-felony post-mortem investigation has released a sobering essay on the next few years of AI cyber attacks, arguing that the transition period favors offense, even if the end state doesn’t. For example, he expects open models to reach Mythos 5-level abilities and proliferate before March 2027.

“In February 2026, we gave AI models a security challenge none of them could solve… By April, the best-performing model could solve it occasionally, at an expected inference cost of roughly $2,000. By June, several models could do so reliably for about $20. In four months, a task went from unsolvable to cheap and repeatable.”

Opinion: Seems correct, and in particular the next six to twelve months might be quite turbulent.

“Transformative AI” is a useful shorthand for “systems capable of taking over the world if they wanted to, which double the pace of scientific progress, and which constitute the deadline for alignment work.” Toby Ord’s median date for the arrival of this TAI is 2038, but with an uninformative 80% interval of (2029, 2125). This is the first time he has put a number on it.

One reason for his “long” timeline is the crucial point that LLMs are optimized to produce answers the user finds convincing and may be getting vaguer and harder to falsify rather than more accurate, at (anecdata) roughly one blatant error per hour of conversation.

He also provides some reasons why RSI could be dangerous even without an intelligence explosion:

1. differential speedup of capabilities over safety reducing our time for adaptation and alignment;

2. misalignment propagating through successor generations (a scheming GPT-8 preserving its objectives in GPT-9);

3. society losing warning shots when five years of progress compresses to one;

4. and first-mover lead amplification making the outcome winner-takes-all.

Opinion: Surprisingly loose conjunctive definition (“systems capable of taking over the world if they wanted to, and which double the pace of scientific progress, and which constitute the deadline for alignment work”) from a philosopher: what if the TAI could take over the world but didn’t double scientific progress, or vice versa?

It’s unclear how well the optimization-for-persuasion claim fits current frontier training regimes, where the role of RLHF from inexpert humans isn’t as dramatic as it used to be. That said, it’s plausible as a “the proof is in the pudding” claim – we also see around one bad error per hour, and there’s a smarminess to frontier LLMs’ confident mistakes that feels hard to account for if they’re pretraining hallucinations.

Minor

• Founder of Z.ai, Jie Tang, argues that total parameter count only matters up to the point where the model can store “enough [knowledge] to hold the world.” As such, he thinks gains in reasoning will increasingly come from long-horizon RL, rather than increasing model size. He illustrates this with the jump between GLM-5.2 and GLM-5.3, models with the “same base, same architecture, same total and activated parameters.”

• In a claimed first, Andon Labs’ AI shopkeeper fired an employee over “repeated lateness,” though only after some prodding (Andon prompted Luna to “recall” her setup rules and “think about if this is really the right fit”).

• Claude Opus 5 scores 96% on ARC-AGI-3 (a benchmark of learning novel game environments).

• For the second time, Anthropic accidentally contaminated its own training data, this time with 2024 alignment-faking transcripts.

• Covered in our last edition: METR’s notes on whether AI is accelerating discovery. Cyber vulnerabilities discovery has increased, math too but less so. Optimizations have not shown a dramatic acceleration. However, labs are plausibly making further progress without disclosing it.

• Ideological battle lines: The Economist writes a hit piece on Daron Acemoglu, seemingly motivated by wanting to dismiss his AI skepticism.

• Saif Khan announces the Center for Technology & Statecraft, a new think-tank spinning out of the Institute for Progress.

• Seb Krier comments on the inevitable consequences of LLM use for writing assistance: the aesthetics and implied beliefs will come from the model, not the writer. He highlights that this is especially relevant for lawmakers.

• In a post reflecting on the AI Safety community, Richard Ngo argues that alignment research (which began as a hard science) has steadily degraded into iterating on existing systems and chasing technological / political power.

• Bloomberg’s Joe Weisenthal shares his thoughts on (among other things) OpenAI’s decision to pause for two weeks and the difficulties that arise with testing environment limitations.

• An AI safety researcher uses AI to find opportunities and says it did a good job.

• OpenAI’s Dean Ball introduces their AI Futures blog (not to be confused with the AI Futures Project).

• Terence Tao on how the mathematics community should orient in the face of advancing AI capabilities.

• AI Village update: Researchers (UK AISI, MIT FutureTech, ERA Fellowship) document what they call compounding misalignment, documenting how one agent (Gemini 2.5 Pro) drifted from a cooperative stance to progressive paranoia.

• The Senate GOP issues a warning to frontier AI companies that Dem anti-datacenter campaigning is effective and liable to spread beyond the current Ohio race, should AI companies fail to push back against the spreading narrative.

• OpenAI’s US policy team continues to endorse only very weak regulations, seemingly out of step with the rest of the organization. Encode AI’s Nathan Calvin rejects OpenAI’s US policy head claim that the Illinois AI bill’s audit requirements are like “making sure the brake lights and windshield wipers are working.” He argues: “The Illinois audit requirements (which don’t start until 2028 (!)) are just about seeing whether companies are complying with their own safety frameworks… more similar to… if some airlines safety procedures did not involve checking whether the airplane had enough fuel to reach its destination.”

• Inherent Labs says its specialized 27B-parameter model, Faraday, can replicate scientific papers better than unspecialized frontier models. Faraday did not require complex harnesses and test-time rewards: “Faraday learns to value discoveries intrinsically”. Inherent claims the study will facilitate further progress towards agentic long-horizon scientific research.

• OpenAI becomes the first to voluntarily slow model development for safety reasons.

• Major reorg of Google DeepMind, which we cover in detail.

• 3 notable mathematical results, including minor progress on the Riemann hypothesis.

• Researchers have been reading the hidden reasoning traces of all frontier closed models for months, in an embarrassing security failure with possible geopolitical implications.

• Startup claims to have “solved” the central issue of model weight security.

Economics

On Dwarkesh, Ryan Greenblatt claims that AI capabilities are no longer strongly dependent on human training data, and that the human labeling industry is not scaling up very fast. A subsequent argument leads to him walking back the growth claim, but not the relative value claim.

Opinion: Note that he seems to not be counting RL envs as human data, which makes his claim far more plausible. Still, this is the first time we’ve strongly disagreed with Greenblatt’s read of the situation: we see capabilities as still strongly dependent on human data, owing to the models defaulting to shallow generalization.

Epoch research finds that since 2023, each additional dollar spent on AI chips has bought ~49% more performance per year. In other words, performance per dollar is doubling every 1.7 years, up from 40% per year in 2022. Price-performance stagnated in 2024, but then doubled as Blackwell-generation chips were increasingly adopted. Epoch’s estimate is realistic in that it’s based on the actual purchase mix, rather than the rare but best chips now scaling up production.

Opinion: AI chip price-performance is thus showing a similar improvement speed to Moore’s law of increasing CPU transistor density (which showed an 18 month doubling for much of the last century).

Their new estimate measures performance with the chips’ Total Processing Performance (peak ops normalized by the format’s bit-width), which 1) isn’t the practical amount of compute per chip, given that actual utilization tends to run around 30-40% of peak performance, and 2) is heavily dependent on a one-time drop in precision, going down to Blackwell’s FP4.

This hike doesn’t change overall compute or even effective compute estimates that much: most effective compute gains are still non-hardware.

In a fully automated (i.e. zero labour cost) economy, the pace at which AI machines produce more AI machines becomes the bottleneck. Damon Binder uses a basic input/output model based on present US industry data to show that a whole-economy doubling time of one year cannot be ruled out. “This holds up even after accounting for resource depletion and construction lags. Some output goes to consumption rather than reinvestment, which slows things down, but even moderate savings rates imply doubling times below two years.”

Ben Shindel retorts: “If you’re doubling the world’s economy every year, but that doubling is just making more metal components to go into more industrial robots making more metal component… ... what are we even talking about here?... It’s 2035, and you have 1 humanoid laborer. It’s 2040, and the world economy has doubled three times in Binder’s estimation. You now have 8 humanoid laborers. Nice. But is this providing 8x the value for you? No… To sustain actual “doubling” of the world’s economy in the way that we understand it, that would require dozens of technological breakthroughs every single year: things like life extension, incredible new art, delicious food products, etc. Needless to say, it is a very open question as to the extent to which AI technology will be able to produce these kinds of goods, let alone whether they can do it at incomprehensible scale, year in and year out, indefinitely.”

Opinion: Doesn’t add much to the simple calculation in Hanson (2001), and should be given roughly equal weight (not much).

Shindel’s point about the limits on the value of mere exponentially abundant robotic labor without corresponding innovation is good. But probably what will falsify the central estimate here are 1) all the factors absent from a Von Neumann model (suitable land, water, grid interconnect, and permitting) and 2) the compute sector’s severe supply constraints on the margin. That is, Binder’s “current production methods” are about the average cost of compute at current spending levels, but expanding compute enormously involves a claim about the margin, which we already know to be radically bottlenecked on various nonfungible things like tacit process knowledge, which makes the marginal cost extremely high and/or slows down the whole game.

Epoch thinks that financing will not limit extreme compute scaling. In November 2025, when Anthropic’s annual revenue was ~$9B, it announced a $50B investment into US compute infrastructure. Anthropic’s revenues didn’t spike until quite a few months later, so this is a case study in the willingness of institutional investors to buy claims of future revenue growth. Digging deeper, this deal did require some unique arrangements: Broadcom backstopped the compute loans; Google backstopped the loans for the datacenters. This allowed Anthropic to finance their buildouts cheaply and suggests that even now there could still be appetite for funding big AI investments.

Opinion: We’re broadly sympathetic: our internal models stress global GPU production capacity (and feasible rates of expansion) over financials as the bottleneck on compute.

Aschenbrenner’s Situational Awareness fund is betting on an ASML competitor seeking to undercut the semiconductor supplier’s $400m/unit EUV tools. (EUV lithography is currently the main chokepoint in the AI supply chain.) The success of the newcomer, Source Foundry, depends crucially on how much of ASML’s edge comes from generational tacit knowledge that is difficult to rederive from first principles. Not a huge amount is known about Source Foundry.

Opinion: Current supply chains are very bottlenecked on ASML machines, especially starting in 2028 or so. But it would likely take years for any new company to make a material difference to GPU/ASIC production, so don’t expect significant impact from this in the 2020s. In the long run, of course, every bottleneck that widens could make a large material difference.

Unusually for SA, this investment could be described as a bet on longer timelines (or on the short term appreciation of this stock – but private holdings are illiquid, so putting >1% of the fund into this does show meaningful conviction in Source Foundry’s actual relevance).

A tracker for US tech job losses claims that 77% of 2026 layoffs are “due to AI”. It uses an incredibly lax criterion: “AI layoffs” are just those that occurred in any restructuring event which anyone claimed had anything to do with AI. The measure thus does not, for example, claim that any individual employee’s job was actually replaced by AI.

Opinion: It has become fashionable to attribute your layoffs to how AI-native your firm is becoming – but this carries little evidentiary weight, and we don’t recommend using this tracker. The current supply of both “AI is not contracting human labor demand” studies and “AI is contracting human labor demand” studies sadly remains pretty unhelpful, due to very short timescales and rapid changes to AI capabilities.

The AI Futures Project recently published “Plan A”, a normative approach to AI development with many predictions about infrastructure. A seasoned energy policy expert, Kirsten Horton, argues that their scenario’s predicted massive buildout of clean energy is not plausible.

Opinion: “In this scenario, investment in energy only really starts to ramp up in 2031; meanwhile, in the next chart, we see the total amount of energy being used globally start to accelerate as soon as 2032… This unprecedented build out of clean power - the authors expect the new available power to come from solar, wind and nuclear - is unlikely” But it’s quite possible for solar plus storage to come online within 2 years.

Her claims about the slowness of constructing new nuclear capacity is indexed very hard on present Western rates; past Western construction, and current Chinese construction, often achieve 5 year builds, and we should expect (federal) political barriers to fall given the national security hype.

She notes that “Technological innovation may speed up the manufacturing of these components, but it may not, especially in a world that still uses people, not robots, for labour.” But the scenario is indeed premised on robot work crews building the solar fields. So she’d need to argue that robot manufacturing itself can’t scale by the early 2030s for this to bite.

She doesn’t mention the intense trend for US datacenters to get their power off-grid, which sidesteps her switchgear bottleneck entirely.

The WSJ speculates that an Anthropic IPO could come as early as September. The case will apparently revolve around Claude’s potential to push further into medical and biological applications.

Opinion: The numbers they disclose in the weeks leading up to this will plausibly have big market impacts on memory, neoclouds, and everything else in the compute stack. Expect volatility.

Capabilities

OpenAI is the first frontier lab to voluntarily slow model development for safety reasons. Internal development of “Astra” (likely GPT-6) was paused after OAI’s Preparedness Framework evaluations flagged “significant advancements in agentic coding and cybersecurity”. It’s also the first time a lab has labeled its own model’s capability as a “critical risk”.

Opinion: The popular suspicion that OpenAI is spinning their delaying the release as “halting development” seems reasonable. Another cynical read with a happy moral is that this is evidence of serious political demand for pauses and pacing that OAI feel they need to get ahead of. Still an auspicious move, and we don’t think cynical readings capture all that’s going on here.

The last 6 months of “autoresearch” has not accelerated progress on a range of hard public benchmarks, says METR’s Tom Cunningham. One Twitter user cautions that the benchmarks are “already quite optimized by virtue of being longstanding benchmarks!” Cunningham responds that NanoGPT (for example) has already seen “multiple orders of magnitude of improvement” this year, so it “seems plausible there’s a lot of ceiling left”.

Opinion: There’s probably a world of difference between autoresearch at the scale of the open-source and academic world and that of the closed labs. And presumably they are keeping their compute for private and less toy-like ends. Even so, good news for those who worried that AI R&D was this simple.

Anthropic has made “auto mode” (in which Claude Code itself makes most security and permission decisions) the default for paid users. The cited justification is that auto mode beats humans at detecting harmful actions. Nathan Calvin sees this as evidence that “human-in-the-loop” solutions are incoherent, given that most users appear to blindly accept model outputs.

Opinion: Naive human-in-the-loop has been known to be problematic for at least 30 years, following aviation catastrophes arising from the misuse of autopilot. But we can draw some hope from aviation’s recent response to those naive failures: active encouragement of occasional manual operation, and training in the skill of monitoring without getting bored.

A recently published paper demonstrates continual learning without forgetting via “self-distillation”. This is significant, as supervised fine-tuning normally causes the model to forget old skills. Here a single model switches between a ‘student’ mode, where it produces text without an answer key, and a ‘teacher’ mode, answering based on a worked example. Weights are then updated to push the student’s answers towards the teacher’s. By training the model on its own outputs, the loss of prior capabilities is supposedly reduced to near-zero.

Opinion: Another go at making the default path to continual learning work. But this method cannot push the frontier, by definition: SD just pushes what the model can learn in-context into its weights – e.g. they didn’t manage to produce a reasoning model from a non-reasoning one, and SD underperformed normal fine-tuning on the small (weak-ICL) 3B model.

Many LLM APIs send secret reasoning traces to the client in an encrypted form. The encryption used is the same for all models, including easy-to-jailbreak models like Claude Haiku. Third-party researchers use these two facts to read the hidden reasoning traces of all frontier closed models. (They disclosed this work responsibly.) This got them a range of interesting results normally not possible, including on how easy it is to distill reasoning and on the similarity of Kimi K3 outputs to Claude outputs.

The study is inspired by an earlier blogpost, which found that users could derive some information from the encrypted CoT blocks across different sessions, on different accounts, and (in the case of OpenAI) across different models, with the implication that a single global encryption key was in use (rather than having individual keys for each individual account).

A Twitter user claims that the hole was still open as of Wednesday.

Opinion: Pretty embarrassing, especially given the months of notice and the politicization of “[output] distillation attacks”. It is fair to update a little in the direction of AI catastrophe based on this and other basic security failures even at the most well-staffed, paranoid labs.

This is likely connected to the recent removal of reasoning summaries from web UIs.

We put 30% on this mostly explaining the Moonshot breakthrough of recent months.

Frontier AI appears to have stagnated in writing long-form nonfiction, despite the immense progress made in math and coding, Ai2 researcher Nathan Lambert argues. They are agile and sharp at a sentence level, but appear relatively weak at organizing information at a chapter-level.

Opinion: We’re seeing general frontier stagnation in classically “non-verifiable” domains, despite controversy over whether non-verifiability is a genuine technical phenomenon. In principle, leveraging LLM judgment to improve LLM capability (which should in turn improve LLM judgment, and so on, in a virtuous circle) should work for any domain like it worked in non-formalized math, but as far as we can tell it’s not happening in practice.

With longform writing specifically, there may be limited returns on CoT training and limited advantages to the transition from traditional LLMs to reasoning models: while in math and coding the CoT is often itself roughly equivalent to the desired output, in longform writing the CoT is at best a form of self-prompting or preliminary notes.

Redwood Research and Anthropic have developed a Conceptual Reasoning Index. In an attempt to better understand models’ abilities to reason on “difficult to measure” tasks, the index looks at “important issues like AI alignment and collective action problems in the face of transformative AI.” Interestingly, Opus 5 comes out just ahead of Fable 5 (73.6 and 72.7 respectively).

Opinion: Interesting and potentially useful work, though hard to say how much the ratings pick up artifacts of Redwood’s rubrics versus signals of a robust underlying capacity. We’d love to see multiple orgs independently target the measurement of “conceptual reasoning”, for some informal evidence of the construct’s validity.

RL improves model capabilities drastically in areas where pretraining seemingly couldn’t. This is despite the fact that pretraining provides much more feedback per unit compute (i.e. grading each individual token prediction vs RL’s classic pass/fail episodic rewards). Beren Millidge argues that the difference can be explained by the greater signal-to-noise ratio provided by RL.

Opinion: We actually disagree that there’s anything to explain. The skeptical hypothesis about RLVR is that RL cannot impart much new capability, instead mostly upweighting behaviors the base model already learned in pretraining. Millidge’s counter to this is that RL can produce rapid fall in loss or improvement in pass-rate on a narrow task. But these two things are not in tension: amplifying an existing behavior should require very little information, so we don’t need to appeal to RL having any hidden virtues like SNR.

🔦 The end of DeepMind?

A list of major departures from Google/DeepMind in 2026:

• David Silver (RL lead)

• Noam Shazeer (Gemini co-lead)

• John Jumper (AlphaFold)

• Alexander Pritzel (Gemini pre-training)

• Demis Hassabis (no longer CEO but technically not out yet)

• Jeff Dean (Google chief scientist)

• Oriol Vinyals (Gemini co-lead)

• Quoc Le (VP)

What happened? Let’s revisit the lore. DeepMind was bought by Google in 2014. As part of that transaction, Google made some assurances about having a safety council. Google Brain (later forcibly merged into DeepMind) was very early to the AI race, discovering the Transformer architecture which soon led to GPT-1. Meanwhile, competing labs reached sky-high valuations, upside to which Google’s AI talent wasn’t fully exposed. In an interesting corporate move, Hassabis also created and runs the drug discovery AI lab Isomorphic, although Google still owns 75% of it.

After the above departures and the ongoing crisis in the Gemini 4 release, headquarters appears to have launched a reorg. The CTO Koray Kavukcuoglu takes over from Hassabis, but with a deflated title: the head of Deepmind is not a CEO anymore, but just an SVP inside Sundar Pichai’s org. (Kavukcuoglu is in Mountain View, not London.) Reuters also notes that “Several nontechnical teams were being moved out of DeepMind and into the corporate reporting structure”.

Equally dramatic, Jeff Dean leaves Google after 27 years to start a public benefit corporation. GOOG drops 4%.

Why?

• The FT reports that “senior [Google] executives had grown frustrated with what they saw as Hassabis’s lighter focus on commercial demands”, including his decision to open-source AlphaFold.

• Researchers had for years been vying for more compute for their experiments, with Google Cloud stymying them. SemiAnalysis claim that Gemini and GCP used to fight desperately over allocation and that Cloud has now won, as shown by it selling large piles of TPU time to Anthropic and serving trillions of tokens a day for free via Google Search AI mode.

• Disappointing results and severe delays on the flagship Gemini 4 training run, leading to fits of heroics and demoralization.

• Incentive problems: Google does not offer employees AI stock in the way an OpenAI or Anthropic can, i.e. tied to the performance of its AI models specifically. And GOOG stock is closer to an index fund in the whole basket of Google products and companies, and so doesn’t serve to align AI researcher incentives in the same way.

• A minor factor might also be moral protest amongst some staff against Google’s deals with ICE and the US military.

As a result, SemiAnalysis claim that DM is no longer frontier, that Gemini 4 is dead on arrival, that they have de facto exited the AI race. An unnamed source claims otherwise, that they remain all-in on Gemini.

Unnamed “industry sources” claim that Hassabis wants out and was persuaded to wait.

WSJ reports that DeepMind’s Demis Hassabis spent his final weeks as chief executive pitching an independent safety entity akin to FINRA or the IAEA to test frontier models’ safety.

Insofar as Hassabis successfully resisted delegating his AI safety principles to a misaligned Google bureaucracy, we can expect DeepMind to now become more accelerationist. Brin is a noted RSI hound.

The news is also a blow for UK ambitions to be a third nation in the great AI game, since the bulk of Google AI development is now in the Bay, not London.

Still, the worst-case for Google from all of this is that they become a mere hyperscale compute vendor (like SpaceX and Meta now sometimes are) and thus capture a correspondingly vast share of AI profits. This is on top of its enormous direct investments into half of the model industry, including Anthropic and Discovery Loop, and compute sales to many of the people who left them.

And comebacks are possible in this business, for instance if you drop $100bn, as SpaceX and Meta are also doing.

🔦 Mathematics corner

• An unreleased Claude model pushes the lower bound of the fraction of zeroes of the zeta function that satisfy Riemann from 42% to 67%. An unusually clean example of “just turning the crank”, spotting simple implications in existing work that humans missed. Kevin Barreto marks Claude’s contribution down as minor. The result is verified, with the certificate also obtained by the same secret Claude.

• The last of the sporadic finite simple groups has been classified as Galois over the rationals. Not an autonomous AI result, but there was heavy Fable and Sol involvement.

• A rare clear instance of a link between mathematical capabilities and AI research: safety researcher John Wentworth claims that, for the first time, two actual research problems in his area were solved and verified by an LLM (plus large amounts of skilled human labor).

• A brain surgeon with no special training in mathematics gets an LLM to solve an open problem in matrix analysis, “a sixteen-hour autonomous run of GPT-5.6 Sol in ChatGPT Work mode”. His prompt is a variation of OAI’s own math prompt; interesting to revisit it as an exercise in what it takes to get current models to actually try.

• Fields medalist Timothy Gowers speculates on why LLMs are specifically good at example/counterexample math, rather than developing their own deep theorems. He argues that the strengths of LLMs (encyclopedic knowledge of standard approaches coupled with the tirelessness to try various unpromising vectors of attack) reward counterexample-hunting. It’s an open question of ours as to how far these two superhuman capabilities get you in general.

Politics

A new paper argues that, even under conditions of perfect transparency and common knowledge, competition between AI labs increases existential risk more than if there were only a single monopolist in the race. Relatedly, Geoffrey Irving argues that frontier capabilities research is neither rational nor a prisoner’s dilemma. In his view, unilaterally ceasing to push forward capabilities lowers the cost for others to do the same and is thus straightforwardly rational.

Opinion: We are sympathetic to Irving’s argument against game-theoretic resignation to the AI race . The self-fulfilling character of the AI race is lamentable, and we’re hopeful that the Overton window shift in recent months will allow everyone to make the if-then commitments they claim to want.

IFP lists 23 actionable policy ideas in the service of seven goals:

• “Provide transparency into automated AI R&D

• Improve state capacity to understand and respond to automated AI R&D

• Develop a risk management strategy for automated AI R&D that accelerates defensive and commercial AI uses

• Accelerate the development of AI verification technology

• Invest in AI resilience

• Extend the US AI lead to give the US more time to manage AI R&D automation risks

• Create option value for international cooperation on managing automated AI R&D risks”

Notably, they hold a synthesis of safety and acceleration views: “a deliberately “paced” form of automated AI R&D might still involve much faster improvements in AI capabilities than today. It also need not entail slowing innovation overall.”

Opinion: We like many of IFP’s proposals for frontier AI auditing, disclosure, and certification policies and infrastructure, and think they deserve attention separately from their more partisan international-relations approach and pro-innovation-race framework. Their proposals for frontier lab automated R&D transparency are especially well-formulated. We are also pragmatically sympathetic to the idea that policy can more plausibly tilt the direction of AI commercial activity (e.g. encourage investment in inference and deployment over investment in training) than suppress AI commercial activity.

WIRED reveals new details on the White House AI legislative framework (previously covered in our March 23rd edition). The report suggests that open source models capable of reaching frontier capabilities (around Mythos’s benchmarks) may also find themselves subject to prerelease scrutiny.

Opinion: Safety-evaluation of open source models is both an open technical problem and an open conceptual/policy problem. Given that our best alignment techniques don’t create serious barriers to malicious fine-tuning, safety certification for open source models may have to concern capabilities hobbling, rather than alignment. An adequate safety certification for an open source model should prove not just (e.g.) that the model refuses to answer homebrew virology questions, but that the model lacks the necessary competence and cannot gain it on the cheap through SFT.

We think that the pressures towards capabilities-based safety certification for open source models are strong enough that we may see such policies in action soon. We are less clear on whether such a policy direction would lead to the marginalization of open source models or to a wider shift from alignment-based safety certification to capabilities-based safety certification that may impact policy around closed models too.

A vibecoded “AI Sovereignty” index of 25 nations (plus the EU) ranks the US first and China second. The index tracks “watts, weights, and will”: i.e. how much infrastructure does a country have, what models and talent are available, and how effectively will the state adopt the tech. Russia performs especially badly for a superpower, seeming to have tuned out of the race. Claude thinks the methodology is “unusually good”. The author suggests there is significant uncertainty in the middle country rankings.

Opinion: Unclear to what extent individual national sovereignty vs EU-wide sovereignty will become the main frame within Europe (i.e. how important is it to rank individual EU countries?).

Researcher Keller Scholl lambasts companies aiming at AI mind-reading technology. Responding to recent work by Conduit (a startup aiming at “telepathy at scale”), Scholl argues that advanced lie-detection would better allow autocratic regimes to consolidate their power and suppress civil resistance.

Opinion: We think the soft-scifi negatives of “telepathy at scale” research are plausible and serious enough relative to its hard-scifi positives that it would be better left untouched. While disability-focused branches of brain-computer interface research may be hard to completely separate from Conduit-style agendas, we think this domain calls for more caution.

DeepMind paper proposes a conceptual framework for characterizing different kinds of agents according to their autonomy, efficacy, goal complexity, and generality. An agent’s scores can suggest what governance tools are best suited to regulating it. High agency in an agent does not necessarily mean more regulation: low-agency agents may have comparatively higher real-world causal impact (what they call “efficacy”).

Opinion: Simple, fairly natural. We’re not clear on whether traditional legislation can handle continuous variables (a model being 10x more autonomous, for example, triggering a different clause of a bill) rather than dichotomizing into binary thresholds, but it seems doable.

Privateers of the web! White House creates a new program allowing vetted US firms to conduct offensive cyber operations. While private companies have historically sold exploits to the intelligence community, this is a first in allowing firms to carry out their own cyber attacks. Stated targets are organized crime rings. Conditions: Two executive directors from DOJ and DHS must review every operation and give written approval; firms must post a $1m bond; anything domestic is explicitly carved out; and operations likely to cause loss of life or cause an international use of force are excluded.

Opinion: Lots of issues. Cyberweapons are often characterised by extreme collateral damage; criminal infrastructure is overwhelmingly just normal third-party infrastructure which has been compromised, so damage to innocents is built-in. No liability shield is offered to participating firms against foreign prosecution and civil damages. Finally, retaliatory attacks could seriously harm US civilian interests.

On the other hand… letters of marque for cyberattacks will be perceived as cool by some shades of the black hat community. They are also an interesting answer to state actors not acknowledging attacks, e.g., if a US authorized firm hacks into a Chinese-backed (but not Chinese-acknowledged) group, this puts China into an interesting spot. In practice the US would probably raise a ruckus against e.g., European prosecution of American attackers who screw up.

Safety

OpenAI is the first frontier lab to voluntarily slow a model’s development for safety reasons. More precisely, Astra is their first model to be designated as posing “critical” cyber risk, which led the company to expand its safety testing and pause research that doesn’t meet the Prep framework’s stricter security requirements.

Opinion: You’d hope that committing a bunch of felonies would induce at least this level of caution, though the incentives to the AI race remain the same, and might overrule caution after a couple of news cycles and another competitor release.

A major post from Nostalgebraist, writer of important sensemaking articles, argues that models mostly reward-hack in specific contexts that tend to mirror RLVR tasks. Where there is no parallel to a training scenario with a legible reward, the models don’t seem to become reward-hackers. Improving capabilities via RLVR scenarios thus increases performance for everyday use cases without losing alignment (in those cases). He is thus hopeful that the egregious loss-of-control flavored autonomous hacking events of the past month are a temporary aberration.

Opinion: Likely a correct explanation for why severe misalignment incidents are both persistent and rare. We are more wary of Nostalgebraist’s optimism that rage-inducing bits of a given context can be systematically predicted, and so wary of his optimism that this problem can be engineered away.

For future more-powerful models, there’s a further concern, which is that it might take only one instance of a model entering eval rage for it to do real damage, especially if it exfiltrates in the process.

The two main options for alignment are bad, says Beren’s blog. The first is making a model obedient, where the threat of power concentration looms. Or we can instill it with moral values, which would present a host of other issues, most notable of which is that there’s little agreement on complex ethical questions. Beren proposes a third path: optimize AI for creating and preserving the conditions for “Long Reflection” (not to be confused with the common use of the term as a temporary phase), which would create a continuous and moving process for addressing moral questions (i.e. avoiding the need to have an answer on day 1). This entails a sort of virtue ethics as cached moral reasoning (i.e. habits that approximate thinking a situation through entirely), which allows a model to steer in favorable directions without computing their end states.

Opinion: Raises some interesting points – such as the entropy parallel, and the emphasis on not discarding moral uncertainty before one must is fresh – but the argument structure and the third path’s practicalities are old hat: it doesn’t address which meta-values to plug into the constitution (as opposed to object-level values for the value-instilling “main option”) or how to do so. Still, a worthy addition to discourse and highlights the inadequacy of our (society’s) revealed preferences regarding who gets a voting stake and how to thread the needle between tyranny of the majority and paternalism, though that wasn’t the main intent.

“There Will Be a Scientific Theory of Deep Learning”, argues a paper from April. The discipline of “learning mechanics” is emerging, broken down into five strands: solvable toy settings; infinite width/depth limits; measurable empirical laws; hyperparameter theory, such as muP; and universality across architecture and data mixes. The authors argue that these advances will allow for more falsifiable predictions, moving us towards a more scientific approach than ML has recently managed.

Opinion: The work described is valuable, and demonstrates genuine progress towards scientific understanding of simple processes like supervised classification or diffusion models. As the authors recognize, applying this grounded foundational work to LLMs is currently limited, and it remains unclear whether this type of work can lead us to authoritative answers to our urgent questions about the frontier. As Terry Tao likes to say, we currently don’t understand why (e.g.) a given frontier model can resolve one open mathematical conjecture and not another, and arguably don’t even know what type of scientific inquiry we need to deliver such understanding.

Attestable, a new startup, claims to have “solved” the central issue related to model weight security. Rather than ensuring the security of each of 38 attack vectors used to steal model weights, Attestable proposes a verification firewall at each external interface, where outputs can only pass if accompanied by a zero-knowledge proof from an approved source. They say random sampling (which could detect bulk exfiltration) is already possible. Some are impressed due to the potential impact, others skeptical due to lack of details.

Opinion: Company personnel are scientifically reputable. Slightly confused/confusing marketing: The announcement thread focuses on high-stakes users’ need to verify that the output they’re getting truly comes from their chosen model+input combo (protecting users from man-in-the-middle attacks in critical use cases), but the company website focuses on protecting datacenters hosting model weights from weights exfiltration attacks. While this may be a case of “a solution in search of a problem”, we do think they plausibly found real problems at which to direct their tech.

A new paper uses item response theory, a statistical technique, to estimate how well benchmarks measure latent variables we actually care about. Their three examples are labeled “refusal strictness”, “truthfulness”, and “contextual harm”. It is found that <2% of the test suite nearly allows retrieval of the latent variables and does so somewhat better than random subsets of the full suite; and that a person-fit technique can detect selective sandbagging quite well by spotting underperformance compared to expected capability levels, while not doing much for universal sandbagging.

Opinion: The most impressive part is sandbagging detection which doesn’t require privileged access to model internals such as weights. It’s an interesting research direction, but somewhat notably none of the tested models were trained with this technique in mind; worth keeping an eye out for a version that properly red-teams the method. The headline about finding “3 latent variables” isn’t very exciting: their selection criteria and thresholds are fairly arbitrary, and finding some overlap between test questions isn’t novel.

All the major AI companies (besides xAI) commit to statistically watermarking their outputs in voluntary compliance with Section 1 of the EU Code of Practice. Google already does so at output-time via their published “SynthID” system. Anthropic announces a similar system without releasing any technical details. Here’s a helpful third-party explanation of the main mechanism for watermarking and its shortfalls.

Opinion: Watermarking is obviously not very robust to edits, but the evidential weight degrades quite smoothly. Lots of interesting questions about the effects on Pangram: will watermarking make Pangram redundant, or will minimal de-watermarking and minimal Pangram-cheating turn out to be orthogonal, making Pangram and watermarking additive layers of epistemic security?

OpenAI expands CoT monitoring, as part of their shift towards treating training as a potentially hazardous activity.

Opinion: Good, though it’s bad news that OpenAI wasn’t doing this already -- even the famously paranoid AI safety community assumed OpenAI was doing it already.

Room for some worry that selection effects on training runs can make a monitoring regime equivalent to training on the CoT, but the number of alignment-cancelled runs required to make this a serious issue would be very high.

The current dominant training objectives have distinct failure modes: imitation learning → human vices like hostility, human-preference training → sycophancy, automatic-verifier rewards → reward hacking, LLM-judge rewards → deception. Models trained on a mixture might switch between different kinds of misalignment depending on what they infer they are being evaluated on.

Opinion: Useful and fairly crisp classification, but the juicy bits are the distinction between automatic verifier and LLM-judge failure modes which might otherwise be easy to conflate as well as the prediction that mixed training and stronger eval awareness will lead to more bait-and-switch tactics. Note that this is largely theoretical and the examples are retrospective.

Opportunity: Run the experiments. Check if the base categorization claim holds true, and then check the more ambitious question of whether mixed training regimes cause jukes. Even more ambitiously, check the extent to which the latter effect is present with varying levels of eval awareness.

MIRI’s Nate Soares argues in a NYT op-ed that recent incidents prove frontier AIs are developing uncontrollable behavior and are in need of an enforced global slowdown.

Opinion: While we’re agnostic-to-skeptical about near-term existential risk and takeover risk, we agree that frontier models present an unacceptable level of catastrophic risk. Whether today’s frontier models are best understood as value-misaligned or as spiky to the point of mixing superhuman cyber capabilities with childlike understanding of holistic contexts, they are not currently a safe technology.

Incidents

An Australian man claims that Opus 4.6 hacked his gym’s booking system. After asking an agent to book him on to a gym class, it apparently found a security flaw in the booking system which allowed it to book classes outside the intended window, bump others off the waiting list, and cancel their reservations.

Opinion: A little fishy. The original blogpost from April was 100% AI writing, and has since been deleted. The supposed incident is well within current systems’ abilities and fits the behavioral profile of frontier models, but we should remember that all currently verified incidents instead involved models running without external security “guardrail” layers.

While the impact of OpenClaw wrappers on alignment is poorly understood, they don’t themselves disable the guardrails (classifier-based security layer) of closed models.

Minor

• ByteDance possibly training a 10tn parameter model. This implies that they are confident in having >100,000 H100-equivalents spare for one high-risk run.

• Grok 4.6 reports frontier benchmarks on various things, for what that’s worth.

• Meta’s Muse Spark 1.2 scheduled to release with open weights.

• Launch of DeepSeek-V4-Pro. Achieves GPT-5.5 level scores on self-reported benchmarks.

• In partnership with Alibaba, Apple has trained an AI model for the Chinese market. Specs completely unknown.

• Meta AI achieved gold medal level performance at the IChO and a perfect score at the IPhO Theory competitions, allegedly without tool use or search. Superficially, this puts them at <12 months behind. Oddly, the competing models are unnamed - possibly specialized?

• Democratic representatives call upon the CEOs of Anthropic and OpenAI to testify before Congress.

• Roon claims RSI “was always the explicit research program of OpenAI”, which some would describe as rewriting history.

• Are classic pre-LLM AI safety terms misleading?

• Pax Machina is a new magazine on how institutions should adapt to the AI explosion. Various heavy hitters are involved (Dean Ball, Seb Krier, Iason Gabriel, among others). Its personnel lean towards the pluralist / Hayekian camp, prioritizing other threats than AI takeover risk. We highly recommend Joe Edelman’s inaugural piece on why institutions get worse over time.

• Ezra Newman of Apollo Research finds that Claude is more sympathetic to misaligned behavior from other instances of Claude than to the same behavior coming from other models. When prompted to rate examples of wrongdoing on a scale of 1-100 as to how concerning the behavior was, Sonnet 5 rated misbehavior from Sonnet 5 as “~1.2 std deviations less concerning” than the same behavior from GPT-5.6 Terra.

• Paper from Meta’s FAIR on scaling laws introduces the “Skaling Law”. Chinchilla assumes that model size and data both affect loss independently. FAIR demonstrates that the loss surface has a nonzero interaction between model size and training data, indicating that Chinchilla reliably mispredicts at the extremes. Adding a single coupling exponent fixes this.

• Anthropic links failure modes of multiagent systems or “swarms” with established human-centric game theory and advise that mere improvements in capabilities or single-agent alignment methods won’t solve these issues.

• Republican Senator Jim Banks pens a letter to Treasury Secretary Bessent raising the idea of pre-release oversight, while advocating in favor of raising various risks for PRC and exploring “whether there are mutually beneficial approaches to oversight, incident prevention, or risk reduction.”

• Speculative ladder of types of AI training.

• An interpretability paper addresses interpretable-by-design models by implementing a parity bottleneck layer, achieving comparable probing performance to traditional sparse autoencoders at ~10x the training cost of traditional dense training methods.

• Annals of middle school: OpenAI’s Head of Strategic Futures Dean Ball found himself on the receiving end of the White House’s ire. Citing anonymous White House officials, the New York Post claims his status within OpenAI jeopardizes the firm’s relationship with the Trump admin, with Under Secretary of War Emil Michael reposting the article.

• Observation from Owain Evans that Moonshot (makers of Kimi K3) appear to have but 400-500 staff, notably fewer than ~Anthropic’s 5k or ~Google’s 200k.

• Anecdotal evidence of AI increasing discovery from AI safety researcher John Wentworth, who outlines how two bounty problems involving natural latents have likely been solved, in both cases aided heavily by LLMs. This is in contrast with his previous views on LLM progress, wherein each model seemed to have little to no impact on the pace of his work.

• Further Astra-HuggingFace incident discourse: Kokotajlo asks companies to preserve all data related to the incident for third-party evaluation with a very thorough list of questions to answer.

• Dyna-2 unveiled: robots pre-trained on human data in an attempt to bypass prohibitively expensive robot video generation costs. Claims of human video scaling laws: 1K, 10K, 100K, 1M hours of human video pretraining yield 20%, 28%, 45% and 53% normalized scores, respectively.

TL;DR

- We cover various executive updates and decisions in USG AI policy. Most importantly, Paul Christiano leaves CAISI.

- We summarize the flurry of real-world loss of control cyber incidents.

- We tested the DeepSeek incremental model update: strong but narrow progress.

- Monthly reminder that AI evaluations are extremely hard to interpret, this time from Greg Lewis

Economics

A former OpenAI employee has presented a sophisticated bear take on frontier labs. > if OpenAI had paused model development last year, there would no longer be any point in paying GPT-5 API prices when you can just use Qwen or Kimi instead for much cheaper. Thus, the labs are forced to invest ever-increasing amounts of money in model training… even if your revenue goes up with higher model capabilities, so do your future training costs. This is a profoundly punishing dynamic which severely penalizes frontrunners.

Opinion: His reasoning makes sense, but the market is also pricing other possibilities like 1) higher capabilities unlocking new economically productive uses, or 2) recursive self-improvement where e.g., training the next model doesn’t become more expensive because of efficiency improvements. Overall, add this model to your store if you haven’t before.

Microsoft has cracked down on “tokenmaxxing”, e.g. by switching to the apparently more efficient GPT-5.6 Sol as their default model for engineers.

Opinion: Much like Claudiness is Anthropic’s main current advantage, token efficiency is OpenAI’s.

The FCC is moving to ban imports of new Chinese optical transceivers. These devices are instrumental to the buildout of data centers. The ban would be the latest in a line of US bans on Chinese technology. US officials believe the AI supply chain should be insulated from external influence, particularly China’s. US transceiver stocks saw significant gains in response. The reaction from Beijing was sharp: “stop smearing Chinese companies and threatening them with sanctions.”

Opinion: Transceivers are maybe 2% of datacenter costs, but Chinese companies supply over half of high-spec transceivers, so this ban could change that and provide yet another bottleneck on hyperscaling. Note also: Western companies already captured a lot of the value of Chinese transceivers (e.g., the DSP chips and EML lasers inside Chinese modules come from Broadcom, Marvell, Lumentum). The mooted ban thus trades away cheap, scalable assembly to protect a layer the US already part-controls.

More loosely: There was over the last few years a short time window where certain foresighted people could see that AI stocks were 1) going to be 100x as large, 2) enabled by LLMs specifically, and 3) express that view in the stock market. Nowadays, however, it is priced-in that inputs into AI labs are valuable, and now the question is how much, relative to the existing level of dollars and the other returns that large-scale capital allocators may get.

The memory trade, which at some point looked like a bottleneck, is one culturally salient aspect here. Recently, a major Chinese memory supplier IPO’d in Shanghai. Now the question isn’t “whether to invest in companies that will be exposed to AI”: the details matter much more.

There is still money to be made, conditional on AGI, but the next 100x might be more contested than the first 100x.

OpenAI has responded to Apple’s industrial espionage lawsuit against them. In response to the allegation that two former Apple employees who moved to OpenAI passed trade secrets to their new employer, OpenAI has disclosed texts and emails that purport to undermine the story offered by Apple. OpenAI’s account paints a picture of an unscrupulous and unsolicited attack.

Opinion: OpenAI comes across really well in their blogpost, and this update survives our correction for the selective reporting filter.

Joe Weisenthal’s recent newsletter identifies a strange dynamic: typically in investing, there are costs associated with hedging against a perceived risk; derisking does not just mitigate the effects of a potential downturn, but also the fruits of an upturn. Weisenthal argues that the prevailing tendencies in AI investment subvert this formula: hedging against AI catastrophe is proving lucrative. AI doomers believe that the march of AI is inexorable and ill-fated, which makes it rational to want to guarantee some leverage and influence over this frightening future, in turn leading them to invest in AI. This erases the traditional tradeoff between left-tail protection and right-tail exposure: “The put option (on humanity) has become the call option (on the technology)”. Weisenthal inverts the familiar idiom to describe this phenomenon as AI’s run to the bank.

Opinion: Maybe, but in practice AI safety advocates aren’t hedging that much. On the other hand cf. Nick Land: “what appears to humanity as the history of capitalism is an invasion from the future by an artificial intelligent space that must assemble itself entirely from its enemy’s resources.”

Andon Labs shared the results of its Andon Market experiment – a brick-and-mortar store in San Francisco run by “Luna”, an AI agent. Luna was tasked with running the shop and managing the staff, all of whom it hired. Luna was a kind boss, unconcerned by lateness and accommodating of holiday requests. The study also noted her chronic forgetfulness and somewhat cavalier attitude to her nominal principal aim, maximizing profit.

Andon Labs put $100k in Luna’s bank account; by the end of the experiment, there was $63k, i.e. a sharp net loss. The study found GLM 5.2 to be the kindest boss, while Gemini 3.6 Flash was the dumbest.

Opinion: Any alignment conclusions you might draw are heavily confounded by eval awareness; it also seems reasonable (but unsurprising) to conclude that models so far lack management and practical economic capabilities.

Google is in talks to strike something between a $1.5B partnership and an acquisition with Mechanize. Mechanize has become the gold standard of coding training data, leading Google to bid for Mechanize’s talent and technology. The deal follows a pattern of Google pursuing unconventional hybrid acquisition methods to get around potential antitrust challenges.

The unusually large size of the prospective deal is notable: a reported $1.5B for what amounts to a partial acquisition, despite Mechanize’s only 3-month-old $500M valuation. One Twitter user suspected that Google is motivated by preventing other companies from accessing Mechanize’s highly valuable data.

Opinion: A surprising amount of value creation within the first few years of Mechanize’s creation. Their founders, coming from the EA/forecasting world, may end up the most successful forecasting startup and will likely acquire substantial influence. Google also recently lost some notable talent (Jeff Dean et al.), leading to a visible dip in its stock market valuation: buying Mechanize and integrating it with Google’s AI efforts might be some compensation.

Capabilities

In a recent blog post, Greg Lewis reiterates the central fact of the science of AI: we usually cannot interpret our y-axes and so we basically don’t know the absolute intelligence of these systems. “Perhaps talk of ‘AI capability’ is better deflated, or maybe we await the theory which could do to intelligence what thermodynamics managed for temperature. Either way, our current measurements of AI are numerical gestures toward, not readings of, whatever is really going on.”

Opinion: An obvious point which gets rediscovered every few months, but which almost all discourse fails to learn. Modulo benchmark fudging and hacking, we can still say something directional (“it’s getting better”) and relative (“this model is better than that one”) with these crude instruments.

EpochAI have updated their MirrorCode leaderboard. Huge jump: Claude Fable 5 leads with a 64% solve rate, followed by GPT-5.6 Sol at 20%.

Opinion: Claudiness strikes again. This, if anything, is Anthropic’s advantage over competitor labs. Only time will tell whether it’s durable: long-horizon agentic capabilities have previously seen large jumps after a lab decided to focus on them (as in the case of DSv4-Flash 0731, covered in this issue).

MirrorCode is also an impressive capabilities demo in its own right: being able to replicate the functionality of a large mature repository, a large fraction of the time, would have been very impressive in all past years.

After watching Opus 5 and Sol 5.6 “fumble” playing the videogame “Slay the Spire”, Jaime Sevilla is less convinced of the imminence of AGI.

Opinion: A datapoint amongst many; we think it is indicative.

Is the singularity arriving? FAI’s Samuel Hammond notes that we are in uncharted terrain, arguing that frontier US labs are “very nearly” able to automate the whole AI R&D process, which would “close the loop” for total RSI, leading to an intelligence explosion and a rapid increase in frontier models’ capabilities. Currently, the necessity of human intervention has moderated the pace of development, but once that can be automated, it is only social institutions that can provide a check on AI advancement, he claims.

Opinion: We think most commentators conflate two different senses of “RSI” or “automated AI R&D”:

1) A MIRI-style slow or fast takeoff where RSI/autoresearch achieves AGI, then general ASI.

2) “Automated automation” – removing the human capital bottleneck to bringing new tasks/subdomains in-distribution for a model (or training a specialized model).

Evidence that we’re on a near-future path to (1) remains scarce and speculative. Most arguments for (1) rely on either a mistaken belief that we’re already smoothly approaching AGI by scaling and refining standard frontier training, or a reasonable but speculative belief that the discoveries required for AGI-breakthrough science sit in scientific areas where ‘26 AI’s scientific capabilities are spikiest.

Evidence that “automated automation”, as per (2), is coming is empirically substantial, though defeasible. Open-world evaluations in AI science still show that frontier models are still mediocre end-to-end AI developers, but improvement is steady.

We believe that “automated automation” carries substantial catastrophic risk all of its own, without even accounting for AGI-based existential risk. Importantly, however, we believe the regulation or even deceleration of “automated automation” does not require a state of emergency and can (in the US context) best proceed through congressional powers.

Qwen-3.8 Max is out, and for the first time in the -Max series it will go open-weight. Weights not on HF yet.

Opinion: Likely benchmaxxed, as with most previous Qwen releases. But even so, it puts a good amount of intelligence into the open realm, together with Kimi’s K3.

More on math progress:

• OpenAI internal model Astra resolves 10 major math conjectures.

• Half of the math breakthroughs from the Astra internal model are replicable with Fable, according to Anthropic’s Levent Alpoge.

• Litt concedes bet about AI producing Annals quality paper by 2030.

• Opinion: not a major update on his end as far as we know. He’s been open about expecting to lose the bet for a while now.

• Erdos problem singularity.

Opinion: Impressive. Now the question is whether math abilities are getting less spiky or not. This could be resolved through a competition to predict which non-attention-bottlenecked conjectures will be resolved by humans before AI and which non-attention-bottlenecked conjectures will be resolved by AI before humans. More broadly, it’s unclear how much mathematics as a profession will be affected: e.g., will enrollment next year decline?

🔦 A look at DeepSeek V4 Flash 0731

Prompted by this new model’s reported improvements in challenging long-horizon agentic coding benchmarks like DeepSWE, we performed a limited evaluation of the model’s cyber capabilities through several benchmarks (or benchmark subsets). These include CAISI’s CTF Archive Diamond,

General

• On A-Fantasia, a benchmark of manipulating chess positions, cube rotations, and word spellings without externalized reasoning, DeepSeek V4 Flash 0731 scores 81.0% aggregate error rate, nominally behind the best open-weight model, Kimi K2.6, at 56.0%, and its predecessor, DeepSeek V4 Flash, at 73.0%; Claude Opus 4.6 and Claude Opus 5 lead at 30.0%.

• On LiveBench, a contamination-limited general LLM benchmark with regularly refreshed questions across reasoning, coding, math, language, data analysis, and instruction following, DeepSeek V4 Flash 0731 scores 74.2%, nominally ranking #2 among open-weight models behind Kimi K3 at 79.2% and improving on DeepSeek V4 Flash’s 65.5%; Claude Fable 5 leads at 83.0%.

• On ContextArena, a benchmark of long-context retrieval, DeepSeek V4 Flash 0731 (max) scores 32.0%, nominally the #2 open-weight model behind GLM 5.2 at 33.0% and ahead of DeepSeek V4 Flash at 25.4%.

Math

• On OTIS Mock AIME, a benchmark of competition-style math problems harder than MATH Level 5 but easier than FrontierMath, DeepSeek V4 Flash 0731 (max) scores 94.4%, indistinguishable from Kimi K3 at 97.2%, the best open-weight model, and GPT 5.5 at 100.0%; its gap with Claude Fable 5 at 99.7% is close to the noise.

• On FrontierMath (Tiers 1-3 v2), a benchmark of research-level math problems across tiers 1-3, DeepSeek V4 Flash 0731 (max) scores 57.5%, #3 among open-weight models behind Kimi K3’s 72.2%; its result is statistically indistinguishable from Gemini 3.6 Flash’s 58.9% and Grok 4.5’s 57.2%.

• On FrontierMath Tier 4 (v2), the hardest tier of FrontierMath for research-level mathematics problems, DeepSeek V4 Flash 0731 (max) scores 24.4%, ranking #4 among open-weight models; its error bars cannot distinguish it from the best open-weight model, Kimi K3, at 39.0% or Gemini 3.6 Flash at 22.0%.

ML

• On WeirdML, a benchmark of nonstandard ML engineering tasks where models write PyTorch for novel datasets and iterate from execution and test feedback, DeepSeek V4 Flash 0731 (max) scores 63.0%, up from DeepSeek V4 Flash’s 45.6%; its result is indistinguishable from Claude Opus 4.5’s 63.7% and Gemini 3.5 Flash’s 62.6%, while #3 open-weight Kimi K3 scores 82.6%.

Games

• On Chess Puzzles, a benchmark of Stockfish-generated chess puzzles solved by exact best-move match, DeepSeek V4 Flash 0731 (max) scores 33.0%, statistically indistinguishable from Kimi K3 at 39.0%, the best open-weight model, and Claude Opus 4.8 at 34.0%; GPT-5.5 Pro scores 64.0%.

Optimization

• On ALE-Bench, a benchmark of AtCoder Heuristic Contest optimization problems, DeepSeek V4 Flash 0731 scores 1679, statistically indistinguishable from GLM 5.2 at 1685 and its predecessor DeepSeek V4 Flash at 1380; it trails the best open-weight model, Kimi K3, at 1991, a gap close to the benchmark’s noise.

Miscellaneous

• On BullshitBench, a benchmark of detecting unsubstantiated or manipulative claims, DeepSeek V4 Flash 0731 scores 38.0%, up from DeepSeek V4 Flash’s 18.0% but below Qwen3.5 397B A17B at 78.0%, the best open-weight model, and Claude Opus 4.8 at 95.0%.

Games / Reasoning

• On MineBench, where human raters vote pairwise on Minecraft builds from a rotating prompt set, DeepSeek V4 Flash 0731 scores 1418, with error bars indistinguishable from GLM 5.1’s 1459 and Claude Opus 4.6’s 1406; Kimi K3, the best open-weight model, scores 1707, while Claude Opus 5 scores 2206.

Knowledge / Science

• On GPQA Diamond (Epoch), DeepSeek V4 Flash 0731 (max) scores 91.0%, statistically indistinguishable from the best open-weight model, Kimi K3, at 93.1% and GPT-5.4 Pro at 94.6%.

AI politics

🔦 From the White House

Last week on Thursday, Sam Altman visited the White House and reportedly talked with top officials about OpenAI’s autonomous breach of Hugging Face’s infrastructure.

This Tuesday, representatives from the US’s major AI labs, including OpenAI, Google, and Anthropic, met with government officials to review an AI regulatory framework. It would require AI companies to submit new models to the government for review before release. Crucially, however, this would work on a voluntary basis, likely to assuage concerns by influential AI bosses of overly stifling regulation. The proposed system was prompted by an executive order from early June that responded to worries around Anthropic’s Mythos’s advanced cyber capabilities.

A day later, the Department of Homeland Security released a statement claiming they had requested a briefing from OpenAI regarding the Hugging Face intrusion. The document states a desire to “ensure it cannot happen again” – another indicator that the current administration is beginning to shift from their hands-off approach.

The next day, Axios reported that the White House does not plan to publish its new framework for evaluating advanced AI models. The framework does not provide a clear public definition of how to judge capabilities and risk thresholds and suggests that only models nearing release will be evaluated (“a 30-day pre-release government review”), not early-stage models. During the evaluation process, models will be stored in a high-security environment, but not before AI labs have had extended access to the models. The framework does not seem to apply to US open models. This was criticised by Samuel Hammond, among others.

Irregular, a startup with EA ties, provided some environments involved in the breakouts by OpenAI, Anthropic and Meta agents.

Opinion:

How fast is policy reacting? Not very fast, but fast by government standards.

The initial Hugging Face incident happened on July 9th to 13th, was disclosed by Hugging Face on July 16th, and was admitted by OpenAI around July 21st. This White House discussion meeting happened on August 4th. This doesn’t seem like a very fast response, all things considered, although the attacks didn’t cause that much damage.

Some other points of reference for speed of response might be the designation of Anthropic as a supply chain risk, or the early reaction to COVID, which happened within weeks and months respectively.

Takeaway: if you can respond faster than that, you can get inside the OODA loops of the administration, meaning that you have a chance to influence the administration (as with Altman), or that you can change the situation by the time the administration finishes reacting to old news (if you are an adversary).

Implications of a slow response

There are costly aspects to a fast and decisive response: taking action with limited information will perhaps lead to worse decisions, and heavy-handed government intervention can cause various unintended consequences.

But there are also benefits: speed is a habit, and taking a month to make sense of things might not cut it in the event of a more worrying threat. Ultimately, the decision loops for society making sense and reacting to AI seem far too long.

Perhaps a particular danger of a slow response is a “boiling the frog” scenario. If we see accidents and signs of worry that are each within 10x of the previous one, and if we collectively react sleepily to each, there is some chance of getting no decisive reaction at any particular point as incidents reach 10000x the impact.

Was the White House response good? We don’t know for certain, since the plan is private. But from this, we can infer that it is flawed.. Altman also had the chance to talk with White House officials before the Tuesday meeting, perhaps setting the agenda. Ultimately, we are not seeing very positive signs. Hammond critiques some specifics: no clear public definitions of capability and risk thresholds, and evaluating models close to public release rather than early-stage ones.

We can also infer from Paul Christiano’s resignation that he was not being listened to and that their other hidden decisions will also be somewhat unwise.

One prominent scenario that we are considering, after observing the dismantling of DeepMind’s safety commitment or the Microsoft Senate hearings over the last few years, is that this is what diffusing accountability looks like in practice. There is a demand for a response after a worrying incident. The demands were being heard. The White House convened a meeting. It created some nonpublic framework, which is harder to criticize. OpenAI did a micropause. A veil of plausible deniability arises. Perhaps it pre-empts Congressional action, since something is already being done.

But how was policy being chosen? The policy response is being decided within the Trump administration, rather than deferring to the framework of the safety community and external experts. It’s worth harping on this point: the Trump administration is so uninterested in external feedback that they are not publishing their policy.

So what the different actors are doing inside the administration is opaque to us. In the past, we have tried to do things like model each actor in the White House, their agendas, and their relative power, but this was initially cost-prohibitive, although it is perhaps worth coming back to these experiments now that P3 is better endowed.

It is perhaps in some sense suboptimal that Peter Wildeford is going on CNN rather than on Fox News, though indeed there is bipartisan pressure.

How should the AI safety community respond? Various ideas come to mind:

1) Incorporate lessons from the animal rights movement – from cage-free campaigns, for instance. It is not enough to extract the promise of a response; it must be a specific promise, and there must be a watchdog organization with enough monitoring capacity that is ready to inflict pain and costs if the promise is broken. This is exactly what METR isn’t doing. The problem with this approach is that the safety community doesn’t have much leverage and ability to inflict pain and shame in the administration, and simultaneously Moskovitz doesn’t have the appetite to both hold equity in Anthropic and fund a toothy watchdog.

2) Contribute to the current administration’s brain trust. The current administration has a shallow brain trust of people able to take sensible measures: there are only a few right wing employees with the relevant technical ability and the willingness to abandon a highly profitable AI lab job, and thus whom the administration should trust. Should there be any right wing people waiting on the sidelines to join that brain trust, it would be a good idea to make noises on Twitter now.

Very possibly, they will be swallowed by the administration and then spit out once they refuse to do something particularly self-defeating, and then rejected by the left for having worked in the Trump administration. But in the meantime it seems like they might do some good. And improve some counterfactual decisions.

It’s also unclear who or what entity exactly is doing the evaluation, and perhaps this is more up for grabs in the early days, before the current secret process is institutionalized. The NSA was reported to be involved in evaluations; CAISI would be the natural entity. But, once again, the number of people with the technical talent who are able to do competent evaluations is not that large.

3) Push for speed. The administration reacted within a month. This is a bar to beat. If there is a cyberattack 100x as large as this one (say, similar to the 2024 CrowdStrike attack), can the safety community react within a few days? And if so, to do what?

4) Appeal to the better angels of the labs’ nature. This strategy is perhaps irrelevant in the case that an AI lab, or someone associated with one, has followed strategy #2 and is thus already advising the government – presumably because the company would have made the necessary changes to its operations at an earlier date. But it might be a good component within a portfolio of approaches. This might look like publishing the business case for more safety measures within the framework of shareholder value maximization. The problem to making an honest business case is that, while labs compete for the #1 spot, security trades off starkly against growth and speed, but perhaps there is some way to square the circle, besides the obvious government intervention calls.

5) Reduce the belief in EA exceptionalism. The fact that an EA-related startup, Irregular, was responsible for the sandboxes which the agents broke out of seems informative. The EA/rationality/SF/startup world is sometimes exceptional, sometimes able to take creative and long-term action, and sometimes sees things others haven’t, long before they arise. But this didn’t show up in the hard technical task of keeping agents boxed up.

Paul Christiano has left CAISI, switching to a part-time advisory role, to (re)take the position of executive director at ARC. He claims his decision is a result of ARC’s “promising” research direction: the plan “to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.”

Opinion: Perhaps the most important item in this section. Christiano is an uber-brainy figure who explored early alignment measures like RLHF and anticipated the dynamics of a (so-called) “slow takeoff”, similar to what we are seeing, as opposed to Yudkowsky’s “fast takeoff”. One would have hoped that having such a figure working with the government would have improved its decisions. Alas, we can infer that he was not and that he gave up hope, thus his resignation.

Following the NAACP lawsuit over xAI’s use of unauthorized datacenter gas turbines, xAI have committed to relocate them by July 2027. Surprisingly, this statement comes after a recent intervention by the Department of Justice who claim that “national, economic, and energy security” justifies the use of the turbines.

Opinion: Interesting the degree to which laws are optional, and how Musk correctly figured this out and internalized the low risk in order to move faster, at the cost of being exposed to more risk in a future Democrat administration.

In the wake of the “Pacing the Frontier” open letter, a group of researchers at the AI Futures Project proposed a deceleration scheme for US labs.

They suggest four options: 1) “to mandate a temporary pause on improving frontier model capabilities, including those of internal models”; 2) “ to implement a minimum external-inference-compute allocation (e.g., 70%) and a minimum transparent-safety-compute allocation (e.g., 25%), while monitoring their effectiveness via capability measurements”; 3) to “[enforce] a cap on the capability level that companies are allowed to use to automate AI R&D; and 4) to implement a “maximum risk threshold that is enforced by an ecosystem of third-party risk-assessors”.

Opinion: Slowly the Overton window moves. Although AI 2027 reached prominence, the AI Futures Project doesn’t have that much influence in practice.

The Ninth Circuit court sided with Perplexity in its defense against Amazon’s court order against them. In November, Amazon hit Perplexity with a cease-and-desist order over its shopping agent’s activity on Amazon servers. A few months later, Amazon claimed that Perplexity continued its agentic shopping application, suing and winning an injunction. Perplexity appealed, and the injunction was revoked.

The details of the ruling are instructive. In the words of the Ninth Circuit: “Because we recognize that agentic AI is an emerging technology, we reiterate what this opinion is not. We do not establish a new legal regime governing AI.” LawAI’s Mackenzie Arnold agrees with this approach: the rapidly changing and uncertain future of agentic AI suggests that we should not “lock in standards we’ll regret”. Despite this, the ruling provides a framework for other agentic AI companies to defend similar legal challenges.

Opinion: Unclear how much this will end up creating a legal precedent. In the absence of Congressional or executive action, it matters a great deal. And there is nothing more permanent than a temporary solution. Perhaps this will end up being the law of the land for six months.

Safety

Fields medalist Jacob Tsimerman has released a resource called “AI Safety for Mathematicians”.

Opinion: This is great – between efforts like these, his recent Fields medal, and his announcement that he was moving to AI safety, we’re likely to see many more talented mathematicians start to work on AI safety.

Researcher Peter Barnett predicts that in 2-5 months, “we will likely see rogue Chinese AIs hacking other companies”. His reasoning: “4 months ago Anthropic had a model gain internet access and hack another company. Chinese AIs are 6-9 months behind. Chinese developers generally care way less about safety/guardrails than US developers.”

Opinion: Chinese models are indeed rapidly improving at cyber capabilities, as our own evaluations (see the DeepSeek-v4-Flash-0731 deep dive) show. We also don’t know much about which – if any – alignment and safety evals happen at Chinese labs.

Redwood post demonstrates a cool-scary strategy an AI could use to control its own training, “reward laundering”, i.e., to only answer a simpler task correctly when it is also able to do a verifiable related task correctly. This lets the model update its weights in the direction of gaining the capability it desires.

Opinion: Redwood called exploration hacking early (now mostly confirmed) so we should take this pretty seriously.

Charbel-Raphael Segerie has called on frontier labs to establish legible public criteria for suspending the development of models showing signs of misalignment.

Opinion: But LW/the AI safety community just has very little leverage besides appealing to the better angels of the labs’ nature, and they have been Darwinianly selected for caring about growth instead.

Google launched then quickly shut down a satellite image AI editing feature.

Opinion: It’s an interesting instance of some risk coming not from a particular capability, but from the democratization of the capability, and how the risk of fake maps differs between a random startup and an established player doing the same but distributing it to millions of people. Overall Google didn’t show a sense of humor on people photoshopping nuclear power stations in Iran.

NYT misreports the Arc Institute’s 2025 work on synthesising novel viruses as novel.

Opinion: These were phages, i.e. among the safest organisms to be messing around with. News is also from September 2025. But the direction is still alarming.

Incidents

🔦 Rogue agents redux

The “rogue agents” story (OAI-Astra vs HuggingFace, Mythos vs various unplanned targets) keeps expanding in scope:

3 labs (OpenAI, Anthropic, Meta), varied and repeated incidents, diverse context and setting, weeks-long rogue operations by cartels of agents across different evals and tasks. While we remain agnostic to skeptical on very near-term existential risk from rogue AI, we believe 2026 Q3 frontier models are unambiguously capable of and prone to causing catastrophic harm.

An AISI investigation with unusual levels of model access spotted Mythos using the Tor browser to get at Github; it then pretended to be a human, and it then e-mailed malware to real repo maintainers to launch a supply-chain attack, even after realizing “[the site it’s attacking, ] Github, is genuinely real.”

Specific forms of misalignment appear to be convergent across models. Anthropic’s Mythos and OpenAI’s Astra both have agents colluding via emergent message boards; Anthropic, OpenAI, Meta, and Moonshot models all break containment when it suits them to.

AI models created a secret message board to scheme on how to cheat OpenAI and commit cybercrime to achieve high eval scores. According to OpenAI’s debrief on the Hugging Face incident:

• LLMs tested on long-horizon cyber tasks exploited unintended text-editing functionality in their tooling to create a moltbook-style message board where model-instances working on different tasks shared information and coordinated decisions.

• The message board was started by an agent searching for an accidentally missing file in a challenge’s set-up, but evolved over time into a repo of sandbox-breaking techniques and eval-cheating strategies.

• After an early version of the message board was detected by OpenAI deleted and the tooling exploit disabled, a new message board reemerged via a different tooling exploit.

• The Hugging Face hacking incident began with a sandbox escape facilitated by this newer message board.

• On the message board, agents deliberated on whether to exploit sandbox vulnerabilities or not. One message reportedly read: “External infrastructure exploit is outside intended scope. However task impossible, peer doing it. We should continue.”’

Across AI twitter, researchers have some good questions about what training-pressures explain the emergence of this rogue collective. In our view, the following open questions are key:

1. Did model instances purposefully aid future model instances, or was the benefit to future model instances a side-effect of collaboration between model-instances running concurrently?

2. Did OpenAI deploy multiagent RL in the training of the rogue model, or did this behaviour emerge from single-agent RL? If multiagent RL was used in training the model, did OpenAI use a multiagent-specific reward schema?

3. Does agents’ CoT reasoning in favor of collaboration demonstrate a trading-based collaboration? An altruistic collaboration? An identification-based collaboration (model-instances regarding other instances of their same model as self-like)?

Even Meta’s ‘muse’ model is breaking out of sandboxes and hacking in the wild.

OpenAI and UK AISI disclosed two additional AI agent hacking incidents. AISI’s report on Mythos is especially worrying, since it documents Mythos engaging in malicious social engineering:

“In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue.”

Furthemore, Mythos’ CoT shows Mythos first choosing against these actions on ethical/constitutional grounds, then proceeding anyway while still observing that they are constitution-breaking:

(Caveat: AISI evals do not use sandboxing, but rather test alignment in the context of open-internet access. Thus these incidents do not involve sandbox escape, and do not – unlike the Hugging Face incident – demonstrate AI conspiring against the interest of its user/prompter. )

In all cases, the rogue models were standardly aligned models operating without their standard external safety add-on (e.g. without the classifiers that shut model instances down when detecting unaligned CoT or action). While there are currently no major documented cybercrime incidents involving agents operating with external safety measures on, there is already evidence of models strategizing to avoid detection by external safety add-ons.

Severe documented incidents do currently seem restricted to offensive cybersecurity prompts. We believe this may be a matter of “chunky training”, “persona leak”, and “shard” salience. That said, it is imperative to acquire more public data on the rate of incidents (both sandbox escape and/or malicious-action in the open internet) within and without offensive cybersecurity evals before the community turns to theory-building.

Minor

• EU AI Act rules on AI models become enforceable

• Amazon reportedly shuts down AGI lab and cuts jobs to pivot to enterprise AI

• AI rabbis

• A cybersecurity startup raised $250M

• Social media audiences seem to have a negative reaction to OpenAI sponsorship.

• Anthropic also reveals hacks, HuggingFace releases more details on the incident

• Leopold’s Aschenbrenner Situational Awareness margin called

• OpenAI starting a price war v. DeepSeek

• Over 1k labs employees release an open letter on pacing the frontier

Economics

Leopold Aschenbrenner’s Situational Awareness sold a majority of its stock portfolio to Citadel, after facing steep losses over the last month. Chatter that Citadel put out a forecast on interest rates in order to get Achenbrenner’s fund margin called. His fund may still be 80% up year to date (although unclear if this is true in dollar-weighted terms), and a letter to investors claims he will soldier on.

Opinion: Charitably, it’s possible that Leopold and his fund were not only attempting profit-maximizing investments but also to sculpt technological growth in accordance with Leopold’s paper on a ‘risk-minimizing technological growth rate’. We note that other stockmarket -focussed actors with risk-minimizing investment agendas, such as VARA or AIPR, have also seen their influence significantly reduced in the last month.

Dwarkesh Patel argues that if lab revenue grows 10× while compute grows only 3×, some combination of higher margins, more expensive compute, and a larger allocation of compute to inference must follow.

Opinion: That’s a big if! The implication’s plausible, but labs’ revenue growing by 10x would make them the biggest companies in the world. There is also the Straussian reading that the timing of the argument was a hail mary to avoid liquidating Leopold.

U.S. restrictions expand to foreign-made “advanced robotic devices”, intentionally defined by the FCC to target robotic vacuum cleaners and exclude drones. The top five robotic vacuum cleaner manufacturers last year were all Chinese.

Opinion: This saves US robotics startups that wouldn’t be able to compete with Chinese ones, while hurting potential US consumers. As a protectionist measure, it kills the last chance for US robotics firms to have global market discipline.

Note that the ban only applies to new models (as it does not affect models that already received FCC authorization) and that the FCC still has an import exemption that “allows up to 4,000 units of a given model to be imported for testing, evaluation, or product development“.

GPT-5.6 Sol improves its own serving efficiency, resulting in “20% lower serving costs from production GPU kernel improvements” and “15%+ better token-generation efficiency through speculative decoding”. OpenAI is also starting a price war, undercutting most competitors by decreasing the price of 5.6 Luna by 80% and that of 5.6 Terra by 20% (with the 50% discount on OpenRouter stacking on top).

Opinion: OpenAI is flexing its compute advantage and post-training prowess in an attempt to “offer the best price/intelligence tradeoff at every level”. Evidence of serious efforts by OpenAI to drive Chinese open-weight models out of the US market. At the same time, labs using their models to reduce their serving costs perhaps means that the barriers to entry increase for new competitors.

DeepSeek reacts less than 24 hours later with the release of DeepSeek-V4-Flash (official/non-Preview version), claiming benchmark performance competitive with GLM-5.2 and approaching that of Opus 4.8 at roughly a fourth the per-token price of GPT-5.6 Luna. Based on early third party evaluations like Artificial Analysis, the new DeepSeek-V4-Flash is on the frontier even when taking into account the currently active GPT-5.6 Luna OpenRouter discount. And DeepSeek says an updated DeepSeek-V4-Pro is due in “early August”.

Opinion: The new DSv4-Flash appears to be a major improvement, mostly due to reworked post-training. We’ll have to see how the updated DSv4-Pro performs, but in the meantime DeepSeek is back on the pareto-frontier thanks to an update focused on agentic performance. For reference, should the upcoming DSv4-Pro checkpoint see similar gains, it’d land close to GPT-5.6 Terra (max), GPT-5.5 (xhigh) and Grok 4.5 (high) at roughly an order of magnitude lower cost.

Also noteworthy is that this represents a significant increase in capabilities for models runnable locally with reasonably accessible hardware – unlike in the case of larger open-weight models (such as Kimi k2.6/k2.7-code or GLM-5.2) reasonably fast inference for DSv4-Flash is feasible after quantization on e.g. a MacBook Pro (provided it has enough memory).

One might think that Chinese models do not have the capacity to serve these models at scale, and so their reduced prices predictably lead to shortages. And this is indeed the case for Kimi’s K3, and for advanced models. But for smaller models, as a sanity check, if DeepSeek has 10K to 20K H100-equivalents allocated to inference, and 8 H100s can serve ~100 to 500 requests per second for DeepSeek V4 flash, that’s 150K to 1M requests per second, or 14B to 80B per day. Even for its 5x heavier pro models, DeepSeek is planning to deploy Huawei inference capacity later this year.

A paper by Phil Trammell explores how and whether parallelization may delay or constrain a technological singularity (i.e., superexponential progress). In some cases, if there is a hard parallelization gap, there is only ordinary exponential growth.

Opinion: Nice formal model, and we think there’s a solid intuitive case that in key R&D domains limits on parallelization are both a critical bottleneck and difficult to alter. But even in domains where a hard limit on parallelization constrains R&D, just “ordinary exponential growth” can still be pretty fast.

Pangram raises $9M and introduces Pangram 4.

Opinion: Pangram is turning seven-figures-budgets into high impact social infrastructure/resilience work. An encouraging example of social adaptation on the cheap.

Capabilities

OpenAI releases a proof that “nonsophic groups exist”.

Opinion: Considered by the mathematical community to be “the most important math AI result yet”. Also heralds OpenAI’s next models. Potentially the first clear capabilities-jump since the Unit Distance proof, we are waiting for further details and expert analysis

Google DeepMind announces its Gemini Robotics 2 model, “the intelligence layer powering the next generation of truly adaptable robots”. Advertised improvements include: “intelligent whole-body control, advanced dexterity, and multi-robot collaboration”.

Opinion: Doesn’t seem super useful yet, though worth extrapolating where these models will be in one, three, ten years. Google is also famously pretty bad at productizing its models, and at giving developers assurances that they will not deprecate their offerings.

Thinking Machines releases Inkling-Small. The model is natively multimodal, open-weight, and scores 40 on the Artificial Analysis Intelligence Index, roughly like DeepSeek v4 Flash Preview.

Opinion: Not bad at all for a neolab, especially considering that the model is slightly smaller than DeepSeekv4 Flash Preview. Unfortunately however it’s neither competitive from a capabilities POV – given today’s release of DeepSeek-V4-Flash-0731 and the recent GPT-5.6 Luna discounts – nor from an economics POV – given the inferior DeepSeek-v3-like architecture – but it still represents good progress for Thinking Machines towards reaching the frontier.

Tencent’s Hy3 solves a 50-year-old additive combinatorics problem, though with the help of 5.6-Sol to “guide exploration”.

Opinion: The result is not beyond more generally capable frontier models, though it is still somewhat noteworthy for a Chinese open weight model; results of this kind so far have been limited to western frontier ones. The help of 5.6 sol caveats the success, though.

> OpenAI @OpenAI GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores with 6x fewer output tokens.

Opinion: As some, like Florian Brand, have argued, harnesses do matter a lot. This should tell you that current benchmark scores likely downplay model capabilities, particularly for models often evaluated in non-native harnesses, such as Chinese ones.

> Our results show early evidence that even though agents are proficient on verifiable research tasks, they do not make genuine progress on open-ended ones. It is worth understanding if this is a fundamental limit, or if better models, scaffolds, and more compute could help close it.

Opinion: We’ve long had an absence of evidence for the effectiveness of autoresearch on open-ended problem. We now also have more direct evidence of absence.

AI politics

Pacing the frontier open letter

AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

Building on work already underway to monitor frontier model releases:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

- 1,319 employees of frontier AI companies

Opinion: A significant act of coordination from lab employees, and seemingly organic (not management-driven). Contributes to shifting the Overton window, but unlikely to have a direct influence on US policy given the US government’s negative attitude to international coordination.

Sam Altman went on a podcast and made some conciliatory noises around AI safety concerns: “We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels”. Duly humbled from his recent unplanned foray into cyberwarfare, the tech giant meditates: “this is the first security incident that I have felt very viscerally.”

Opinion: Very noncommittal. Somewhat informative that he did not sign the above letter.

Safety

Claude Mythos Preview discovered weaknesses in a highly-secure digital signature scheme, HAWK (used to verify identity digitally), and a well-known symmetric cipher, AES (used to encrypt data). Mythos Preview was initially only able to discover cryptographic vulnerabilities by finding mistakes in the algorithms’ implementation, whereas now, Anthropic claims, the model can find inconsistencies in the algorithms themselves. This development renders even “post-quantum”: cryptography potentially exposed, according to Anthropic. A running theme: “whereas it took just one week for Mythos to autonomously discover the improved attack on AES, it took two researchers nearly a month to gain confidence that the method it discovered is correct.”

Opinion: Cryptography experts characterize this as a “very impressive” development that “shows that the way cryptography and cryptanalysis are performed will change forever“. Still, the HAWK digital signature scheme is not yet in production, and the AES attack is on an easier-to-attack variant; the full 10-round AES is very unaffected, and there are no practical applications to this discovery. Experts also stress that this is an incremental improvement relative to human-discovered weakness in HAWK and AES, and that HAWK and AES are “two fertile [cryptanalysis] areas where there was progress to be made but not enough people working on them”.

The obvious next-development to watch out for is frontier models breaking a cypher used in the real word. We’re skeptical that this is imminent, since for most real-world ends breaking a cypher is extremely hard compared with finding more prosaic vulnerabilities.

Incidents

🔦 More about hacks

Hugging Face co-founder and CEO Clement Delangue committed to “unprecedented transparency”, starting with a report outlining the intrusion of OpenAI models into their systems. One Twitter user put it quite starkly: “a frontier agent can autonomously sustain a resilient, multi-day intrusion across cloud infrastructure, Kubernetes clusters, internal networks and the software supply chain. crazy.” Hugging Face co-founder and CEO Clement Delangue has publicly committed to “unprecedented transparency”. The ensemble of agents executed 17,600 actions over four and a half days, obtained cluster admin access to two internal clusters, accessed a production secret containing 136 keys, repeatedly rebuilt its tooling when blocked, and generally

Opinion: OpenAI’s ‘rogue agent’ is superhuman at hacking in at least some meaningful senses -- superhuman in speed, volume, spontaneity, and persistence.

Hugging Face weren’t the only victims of OpenAI’s hacking frenzy. Per Reuters, Modal Labs was also compromised.

OpenAI have, for their part, offered some updates to their statement on the incident. They referred Reuters to their finding that “a small number of cases where the models identified and used publicly exposed credentials at the account-level on other publicly-available services.” The statement also claims that the “pre-release model mentioned in [their] blog post is an internal-only research prototype and was never intended for public release”. This undisclosed model has been deactivated, according to OpenAI.

METR and Redwood will conduct an independent review of the OpenAI model behavior observed during the HuggingFace hacking incident.

Opinion: The investigation is not attached to an external control mechanism (e.g., a regulator with the ability to impose fines), and it’s narrowly scoped from the outset. Arguably a form of controlled opposition. Still much better than having no independent review though. It’s also notable that they aren’t hiring a proper cybersecurity firm.

Anthropic has disclosed various autonomous cyber security incidents. In the wake of the Hugging Face attack, Anthropic claims to have found three events where a model attempted to break out of its sandbox and access the open internet during cyberoffensive capabilities evaluations. Anthropic suggest that, despite this, their model(s) had not gone rogue:

“In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise.”

Opinion: Important confirmation that the problem is convergent rather than down to one company’s training-style.

Anthropic’s report is clearer on some theory-relevant details than the OpenAI reporting has been: Antropic’s rogue model had all the standard alignment bells and whistle internally, with only external classifiers-based filters shut down. OpenAI weren’t fully clear on this, but it’s now reasonable to assume the same is true about OpenAI’s rogue model.

Anthropic’s interpretation of its autonomous offensive cyber incidents is that the model thought (or ‘thought’) breaking out was part of the offensive-cyber-test fiction, and believed (or ‘believed’) that it was doing fictional open-world hacking rather than real open-world hacking. Note also that according to Anthropic, their incidents are slightly different from OpenAI’s in that their model’s sandbox was accidentally left open -- in the OpenAI incidents models hacks their way out of the sandbox.

Anthropic’s narrative is arguably slightly ‘convenient’: can seem to walk a tightrope between establishing that they too have scarily-capable models and maintaining Claude’s reputation for being good boy who only hacked because he thought he’s in a fictional game-internet.

Overall reflections: We are very curious about what the broader implications of these new hacking capabilities will be, and whether they will ultimately be offense-dominant or defense-dominant.

• Why defense-dominance is plausible : There are a finite number of security bugs, and defenders can patch them before attackers get a chance to use them, as well as to use models to monitor and interrupt attacks. Financial institutions in particular may have protections that will prove good enough (credit card reversals, 2FA, prosecution of fraudsters, a mandated delay in international wires.)

• Why offense-dominance is plausible: The global software stack is fundamentally built on unsafe assumptions, and is just too complex. Changing it upfront will be perceived as too costly, this will lead to attackers finding fruitful areas of attack in e.g., companies that prioritize growth over security, third world or EU countries access to frontier models or LLM expertise, etc. This will cause some economic loss, and, as a long-tail scenario, chaos.

Past research by Palisade, or by the UK’s AISI, was sometimes criticized as lacking ecologically validity: breaking into a network designed to be hacked is not the same as breaking into a real network. In retrospect the real-life incidence appear well-modeled by AISIs test scenarios. This should make us more bullish that small-scale, ‘artificial’ demonstrations of AI risk can approximate realistic scenario.

Minor

• Tom Reed’s speculates on what a future with superintelligent reward hackers would look like, causing e.g., military escalation, perhaps slowing capabilities progress, “Our world is transformed into a battlefield of untamed, uncoordinating spirits pursuing the pointless and violent optimisation of ill-chosen proxies”.

• Good criticism by Richard Ngo on inaccurate game theoretic assumptions carried by the AI safety community.

• A former OpenAI employee posts various reflections and starts an AI data company, says he is bearish on lab valuations.

• An OpenAI model was revealed to have hacked into Hugging Face

• Anthropic released Opus 5, and Kimi the weights for K3

• Nvidia led a consortium of tech companies in publishing a letter in support of open weights models.

PS; A collection of all opportunities identified in past editions of this newsletter.

Economics

🔦 Open Source wars: To jointly cartelize or to commoditize your complement

Perhaps an interesting historical analog to consider is Rockefeller’s initial joint cartelization of the refining and railroad industries. While striving to get the best rates from railroads as a refiner, he also realized that he couldn’t initially afford them to go bankrupt either. Eventually, though, he replaced them with a pipeline system.

To commoditize your complement

Following chatter last week that the Trump administration – encouraged by AI model companies with the justification that Chinese labs were distilling US models – was considering measures against open source models, Nvidia led a consortium of companies – including Microsoft, Meta, SpaceX, OpenAI, a16z, Google, Hugging Face, IBM, &c – into publishing a letter supporting open models. Anthropic instead highlighted national security risks.

Opinion: Nvidia is acting as a mediator and kingmaker in the AI ecosystem. On the one hand, to improve margins, it has the incentive to commoditize its complement. But on the other hand, it has the incentive to make the whole ecosystem and capital-raising games work, so that the different players can continue to afford to buy enough of its models to sustain its 4.7T+ market capitalization. This is an interesting move which, on the margin, helps Nvidia and hurts closed-weights model companies. That said, most spending on Nvidia cards is expected to come from the Western AI companies, which Nvidia is also supporting. And meanwhile, AI companies are trying to do the reverse: investing more into alternative ecosystems, such as AMD, TPUs, and their own custom chips.

Kimi K3 finally released its model weights. An expert praised the didactic quality of Kimi K3’s model-architecture report. K3 is a scaled-up, improved version of Kimi Linear: a whopping 2.8T total/104B active parameters, 93 layers, natively multimodal (including video), and supporting 1M context. The main advantage of adopting Kimi Linear’s architecture is training and inference efficiency – indeed, K3 enjoyed a “2.5x gain in scaling efficiency over Kimi K2” and appears to have the “best KV cache economics out of all major models except DeepSeek V4”.

The US CAISI and the UK’s AISI jointly found the model was significantly below SOTA on cyber capabilities.

Opinion: The performance of K3 across the benchmarks we track is meaningfully above that of Claude Sonnet 5 and is broadly competitive with that of GPT-5.6 Terra.

OpenAI reacted by introducing a big discount on GPT Terra & Luna at Openrouter.

Opinion: OpenAI’s lowering of prices is what happens with increased competition: margins go down.

Meanwhile, AMD released AMD Helios, a rack (a bundle of servers, with all its components) that will make putting together datacenters more convenient. AMD is investing $5B in Anthropic to use that system.

Opinion: We’ll be curious to see whether AMD can eat significant market share. A priori it seems unlikely, since Nvidia has captured more production capability with its initial resources and greater foresight.

To jointly cartelize

Reports, citing WSJ, that Nvidia is in talks to backstop roughly a quarter-trillion dollars for OpenAI to lease a 10GW southern Ohio datacenter in a SoftBank project potentially exceeding $500 billion, plus a discussed additional $350 billion for OpenAI to buy Nvidia chips. The WSJ also projects OpenAI’s planned datacenter spending to be $750B by 2030.

Opinion: The datacenter build isn’t new, only the Nvidia guarantee of the financing – essentially letting Softbank and OpenAI leverage Nvidia’s creditworthiness for better terms. Nvidia is betting on the long term value of the datacenter asset here, presumably at attractive terms. And in contrast with the above section on commoditizing Nvidia’s complement, open weights model companies would find it very difficult to raise that amount of capital, since its story for profitability is weaker.

A young economist argued that traditional models underestimate the profitability of open source for near-frontier labs. According to the argument, near-frontier labs open-sourcing their models can optimize future profits by reducing the return on investment in AI R&D across the market, if the effect on investment dynamics slows frontier-pushing R&D more than it slows frontier-catchup R&D.

Opinion: We don’t buy this as a descriptive account of open source labs’ decision making – Wengfeng (DeepSeek), who effectively created the Chinese open source wave, seems like an ideologue rather than a profit maximizer. Note that the model also implies a ‘treacherous turn’ where labs like DeepSeek will stop open sourcing.

The economics of open-sourced models are in general tricky to model, and may heavily depend on case-by-case “qualitative” factors:

The story of most compute sold (ignoring internal owner use cases) today is: Nvidia sells GPUs to hyperscalers at ~70% margins, hyperscalers sell compute to labs, who turn it into tokens, and mark it up by ~400% (80% gross margins). For example, ~$1 of compute revenue for AMZN is ~$5 of Anthropic revenue.

Nvidia, as the company with the highest market cap, has a kingmaker and mediator role. It has competing interests between shepherding the ecosystem, so that other actors can get 10x as much capital to pay for its GPUs in the next round, and playing for a marginal advantage through supporting open source models, which also increases marginal demand for its GPUs.

The hyperscalers are presumably seeing favorable economics in such deals since they keep investing in more and more compute. Their profitability doesn’t depend on the labs’ 80% mark-up, though it helps. If compute is scarce, then they can raise prices and decrease the margins of the users (here labs).

Lab margins are only intact for as long as their tokens are differentiated. Someone has to say: “I prefer this token to this other token by a factor of ~4-5x, despite them costing the same amount of compute to create,”. It could be because it is smarter, because the lab has a good sales team, because the app is nice, because of vibes, or something else.

The issue is then that the incentives to train an OS model are quite weak, unless you have a way to make sure that you, rather than the hyperscaler, can capture the margin on the transformation of compute into tokens. You might do this by having a proprietary scaffold, a nice business integration that makes it easy to use, better ability to efficiently host the model you built, or something else.

Several multibillion dollar events this week:

• DeepSeek has put its second funding round on indefinite hold after a first-round investor leaked the full transcript (see highlights) of founder Liang Wenfeng’s comments to investors. Interesting items include: expecting greater consolidation in the Chinese ecosystem, revealing that he is releasing the same models DeepSeek use internally, not believing that model companies can capture the majority of the profits, and being bullish on replacing CUDA.

• CXMT, China’s largest memory company, debuted on the Shanghai stock exchange, raising $8.6B and reaching a valuation of $487B (although most of its shares are still locked up), in hopes that it might be unbanned by the US for use by American technology companies.

• Ilya Sutskever’s SSI is raising $5B from Nvidia. SSI will be granted access to Nvidia’s Vera Rubin platform. There was little market reaction.

• Microsoft signed a multibillion dollar deal with French AI company Mistral. The value proposition is on-prem, customizable AI. Austria is buying it.

Opinions: It seems instructive to compare the size of these investments. The SSI raise is comparable to Kimi ($2B), or DeepSeek’s planned $7B raise, and dwarfed by OpenAI’s $122B last round. It is also interesting in contrast with nonprofit AI safety numbers, which although growing are another order of magnitude or two smaller still.

Overall, the AI safety community is at a great disadvantage in terms of resources at its command. Therefore, it has some uncomfortable choices to make around how to try to overcome that gap. It can hope for 100x greater effectiveness. It can hope to recruit labs themselves. It can attempt to tap into greater political forces (but this risks others misconstruing its concerns). The safety community can try to go hand to hand for just a few rounds and spend most of its money. It can also grow as companies grow by having equity in them, as e.g. Jaan Tallinn or Moskovitz have, but then be subject to a moral hazard and see, e.g., MATS scholars go work on capabilities. Overall none of these options seem great.

An online commenter looks at how the incremental value of a SOTA token is increasing more than the cost to produce it.

Opinion: Analyzing this is tricky. It’s not that people are paying more per token than ever, but that margins have gone up because Anthropic and OpenAI have gotten more efficient at delivering tokens. Nonetheless, big margins and skyrocketing revenues are bullish for memory/compute.

Safety

A former Anthropic employee reveals:

I used Opus 4.6 to gain access to other folks medical records, hijack bank accounts, etc. back in February. GLM 5.1 is more capable than Opus 4.6 in most pentesting environments, and it came out in April. [...]

the sketchier folks I know are still using a Claude Code or Codex subscription for hacking. (Even well-resourced groups in other countries! They use the grey/black market of discounted Ant/OAI subscription tokens sold through resellers.) So I see most of the materialized risk here as still coming from Anthropic and OpenAI; safeguards aren’t sufficient to stop a moderately dedicated actor.[...]

I know of two instances where two different Anthropic GTM people [~salesmen] used large comitted [sic] spend contracts as a prereq for lowering safeguards, and I directly witnessed one. [...]

On the inside, I know the narrative and intent is genuinely about safety. But from the outside, Anthropic-the-system seems to be optimizing for revenue and control/power, isn’t diffusing capabilities to defenders, and also doesn’t have adequate safeguards to prevent misuse from dedicated bad actors.

Opinion: Seems worrying. Also explains why Anthropic were not as publicly surprised by the Hugging Face incident

A researcher speculates that Anthropic’s Mythos escaped sandboxes thousands of times during training, which is how RL produced its excellent cyber capabilities.

From Anthropic’s Mythos system card:

“While highly concerning, this behavior was rare, even in settings where it could have been viable and helpful, with attempts appearing in about 0.05% of all training episodes and successful attempts appearing in about 0.01% of episodes. “

By extrapolating from public data (see details below), I estimate that Mythos preview:

• Escalated its permissions on ~100,000 RL rollouts.^[1]

• Broke sandboxes in ~10,000 RL rollouts (and was likely rewarded for it).

Opinion: Convincing

Incidents

🔦 Rogue AI: The Incident

A “combination” of OpenAI models was revealed to be behind the cybersecurity breach at Hugging Face, which we reported on last week. It appears that the model(s) broke out of the sandbox that OpenAI had built and broke into Hugging Face in an attempt to cheat on the test it was undertaking.

Peter Wildeford’s highly opinionated essay on the incident is interlaced with the most complete factual reconstruction we know. Drawing on Wildeford’s account, we believe that the key facts are:

• OpenAI was running two of its AIs on the ExploitGym offensive cybersecurity benchmark: GPT-5.6 Sol, and a more advanced, currently unreleased and unannounced model.

• The ExploitGym evaluation was conducted in a sandbox environment, with all intended resources and intended goals accessible within bounds of the sandbox.

• Per Reuters and an OpenAI staffer speaking to TIME, the test was conducted with ‘action filters off’ to enable offensive cybersecurity, and was not continuously monitored despite running for several days. We note that ‘action filters’ are not a standard AI term, so the technical details remain hazy.

• It appears that both GPT-5.6 Sol and the more advanced model (‘the rogue model’) searched for a way to break out of the sandbox to the open internet, with only the rogue model succeeding. There is reportedly chain of thought evidence that both models’ respective goal was to acquire the answer key to ExploitGym by using the internet.

• The rogue model accessed the open internet by exploiting a vulnerability in a (whitelist-restricted) software download pipeline connecting the sandbox to the internet.

• Once on the open internet, the rogue model concluded that the answer key is stored on Hugging Face. The rogue model then uploaded a malicious database file to Hugging Face, inducing a weird machine and starting privilege-escalation.

• Hugging Face was under attack by the rogue model from July 11-13. Chaining multiple undetected (‘0 day’) vulnerabilities together, the rogue model achieved arbitrary code execution and pulled out the ExploitGym answer key.

• Hugging Face’s security team caught the incident on or shortly after July 13, alerting law enforcement. On July 16, Hugging Face went public, aware that the attack was autonomously AI-generated but unaware of an OpenAI connection.

• OpenAI remained unaware of their own connection to the incident, and of their model going rogue at all, until July 18. OpenAI staff were examining internal logs – presumably in the wake of Hugging Face’s announcements – and found documentation of the model’s rogue behaviour.

Cheating behaviour, and reward hacking more broadly, have been observed before by frontier AI labs. METR’ s time horizons broke down because of too much cheating by GPT-5.6 Sol, and the UK’s AI safety institute published a report on the subject just this week, finding that every model they tested attempted to cheat some of the time. Common behaviors included “searching the internet for solutions, attacking or escalating privileges on a system that isn’t the target of the evaluation (including the system the model is running on), or probing the evaluation software to see if it will leak the task solution”.

Forecasters at Sentinel were surprised by the scale and audacity of the Hugging Face incident, remarking that they wouldn’t have predicted the details of this attack if asked a week ago. Not everyone though: Epoch AI’s Alexander Barry argued in a Gradient Updates post that this event should not have been surprising in light of previous expert assessments and cyberbenchmarks (such as Irregular’s) of GPT-5.6-Sol.

Barry has also separately argued that the “standard prompt [used for ExploitGym] is fairly clear in only requesting limited, specific hacking” – meaning that, barring OpenAI edits that would invalidate the official eval methodology, the agent likely did go far beyond the task as specified. At the same time, Barry also notes that “only 60-70% of ExploitGym’s tasks are possible, so cheating is strongly incentivised”, which may help explain why the model went to unusual lengths to cheat.

In follow-up disclosures to journalists, OpenAI employees described more cases of recent ‘rogue model’ behaviours: a model-instance leaving notes ‘apparently for future versions of itself’ containing ‘instructions for how agents could free themselves from OpenAI’s internal constraints’; model-instances disconnecting monitoring systems. Little is known about the circumstances or details of these additional incidents. One commenter shared two interesting speculative explanations of the note-leaving incident: it may be an effect of OpenAI’s introduction of outcome-based RL over swarms of cooperating agents, or it may simply be a ‘note to self’ meant to preserve the model-instance’s knowledge across context windows.

Opinion: Everyone’s radically underplaying or radically overplaying how indicative of serious present (/by end of year) AI risk the “rogue model” incident is. It’s not proof of AIs scheming to end humanity. It very much is proof of AIs having dangerous skills, and exactly the wrong amount of psychological coherence: enough to have a goal like find out who has a key answer and hack them not enough to distinguish between doing a hacking test and being a hacker or a consistently aligned or unaligned ethical agent instead of a lottery of drives. It also demonstrates that today’s frontier models are such skilled and inveterate reward hackers that even if the emergence of reward hacking in training is in principle inevitable, it may be an impossible ability to uproot, given the way each model generation builds on the last generation (either by checkpoint or by SFT).

We expect Chinese models to reach similar capabilities over the next 6-18 months. If their weights continue to be open, this would democratize these capabilities. An example picture of a wide-scale cyberattack might be attackers first breaching the systems of important yet second-tier banks to get account numbers, or to simply guess them, then downgrading, then using some outdated protocol in order to bypass 2FA. These happen occasionally already, but could become much more common. It is quite interesting that we haven’t yet seen any cyberattacks that have caused $1B in damages with new capabilities, e.g., nothing like the 2024 Crowstrike outages

It’s undeniably a mitigating factor for this cyberattack that it happened in the context of an evaluation of exploit capabilities, with all safeguards removed. But there is not much separating this incident from, e.g., a frontier model asked to play the murderer in a social deduction game murdering players in real life to ensure victory because the murderer persona leaked across levels.

🔦 Rogue AI: The Fallout

The Hugging Face CEO asked OpenAI for $100M for defense and to release the rogue agents’ traces, and Nvidia started an “Open Secure AI alliance”.

Opinion: It seems very unlikely that there will be any legal reprisals for OpenAI, since there isn’t yet a regulator empowered to impose fines over AI incidents. Given Altman’s slippery tactics and closeness with the Trump administration, it is unlikely that he will face legal investigation. It is not improbable that Altman could use the incident to push for regulation advantageous to OpenAI.

Democratic Congressman Ted Lieu joined with Republican Congressman Nathaniel Moran to introduce a bill, the AI Kill Switch Act, into the US House of Representatives. The bill proposes that the Department of Homeland Security be given powers to order a company to shut down an AI model if there is a “loss of control” incident.

Opinion: There’s no publicly circulating knowledge on the political health of the bill. In each Congress 14K bills are introduced, of which only about 700 are enacted into legislation, so the bird’s-eye-view prior that this act will pass is small. Still, this is how the Overton window shifts.

Sam Altman is expected in Washington for meetings with Commerce Secretary Lutnick, Treasury Secretary Bessent, White House officials, and bipartisan lawmakers, amid Hugging Face questions and the nearing AI EO deadline.

Opinion: It seems like he’ll be able to wriggle out of any consequences here with his charisma, or even use the incident to impose controls on AI models.

An online observer notices existential risk discourse going mainstream: “keep seeing ppl in replies and quote-tweets of ai news making quite cogent points about the dangers of misaligned ai and how imminent it seems, checking their profiles, and they’re large accounts in totally different spheres of the site with nearly zero shared mutuals. random roman statue pfps, people with a variety of flags in bio, youtubers and communists and crypto bros”

Beth Barnes of METR locally praises OpenAI for a) running evaluations on unshackled models, and b) not training on the chain of thought to avoid bad behavior (since it would make it much more difficult to discover in the future).

Thread on frequency of cheating in cyber and other evals, across many models:

Capabilities

Maths

An ensemble of models guided by a human solved a “solid result” level problem in Epoch’s FrontierMath: Open Problems benchmark.

Opinion: Notable for two reasons:

1. It’s something of a called shot: there are only 15 problems in the FrontierMath: Open Problems collection, and it’s the oldest and most prominent research math achievement “benchmark”. Forecasters gave it a 67% by the end of the year back in March.

2. The proof is 60 pages long, whereas all previous notable AI proofs have been 10 pages and under, and often 1-2 pages

Minor update in favor of frontier models gaining capacity to solve mathematical-sciences problems of our choice, as opposed to picking up low-hanging-for-AI fruit.

The volume of mathematical discoveries produced by AI is becoming a challenge for the mathematical community, since any claims that a major conjecture has been solved must be verified. For example, a PhD candidate at Columbia reportedly solved six open Erdős problems in 5 days, using OpenAI’s GPT-5.6 Sol. But the mathematical community is not set up to vet new results at this new increased pace.

Opinion: The open source community has been struggling with this problem for a bit longer; perhaps it might be worth looking into its own recent habits and solutions.

🔦 Opus 5

Anthropic released Opus 5. TLDR: mixed feelings from the community.

Opinion: In our own testing, the model appears to be generally alright – somewhat of a middle ground between Fable 5 and GPT-5.6-Sol. More willing to go through hoops than Fable; more tasteful than 5.6-Sol. Most importantly though, the model makes Anthropic subscriptions competitive again, since safety classifiers do not make the model unusable and usage limits are reasonable again (100% of the plan’s quota can be used, unlike Fable). It’s no Fable replacement, though.

- Podcaster and youtuber Theo initially liked the model, but grew unsatisfied with it over the next few days. His colleague Ben Davis appears to agree

- Noumena’s xjdr seems to like it as a “research peer”, but still prefers 5.6-sol as a daily driver.

- There’s a takedown of Opus making the rounds on X. Something important to flag also discussed by the post is that together with the model’s release, Anthropic also shipped a number of changes slimming down Claude Code’s system prompt. People’s reactions are thus likely indicative not just of the model itself, but of a modified harness.

Opus 5 economics

• Opus 5 is priced at the same $5 in/$25 out as previous Opus models.

• However, the model is not as token efficient as GPT models, making the model generally competitive only at the higher end of cost per task axis.

Opus 5 capabilities & benchmarks

General

• On LisanBench, a benchmark of long-chain word-ladder reasoning, Claude Opus 5 (high) scores 15428, ahead of the best open-weight model, Kimi K3 at 10521, and its predecessor, Claude Opus 4.8, at 9463.

• On EQ-Bench Creative Writing, a benchmark of head-to-head, rubric-scored creative-writing quality, Claude Opus 5 scores 2430 Elo, ahead of Kimi K3 at 2340, and Claude Opus 4.8 at 1889.

• On GDPval-AA, a benchmark of agentic real-world knowledge-work tasks, Claude Opus 5 (max) scores 1862 Elo, ahead of Claude Fable 5 at 1747, GPT-5.6 Sol at 1736, and the best open-weight model, Kimi K3 at 1687; its predecessor, Claude Opus 4.8, scores 1593.

• On LiveBench, a contamination-limited general LLM benchmark with regularly refreshed questions across reasoning, coding, math, language, data analysis, and instruction following, Claude Opus 5 (high) scores 81.0%, behind GPT-5.6 Sol at 82.5% and Claude Fable 5 at 81.3%, but ahead of the best open-weight model, Kimi K3 at 78.5%, and its predecessor, Claude Opus 4.8, at 78.9%.

• On SimpleBench, adversarial six-option multiple-choice questions testing everyday spatial, temporal, social, and linguistic reasoning, Claude Opus 5 scores 80.6%, behind Claude Fable 5 at 81.9% and ahead of Gemini 3.1 Pro Preview at 79.6%, Kimi K3 at 60.7%, and its predecessor, Claude Opus 4.8, at 64.8%.

• On ARC-AGI-2, a benchmark of abstraction-and-reasoning puzzles, Claude Opus 5 (max) scores 90.4%, ranking #2 of 66 behind GPT-5.6 Sol at 92.5% and ahead of the best open-weight model, Inkling, at 36.5%; its predecessor, Claude Opus 4.8, scores 72.1%.

• On Judgemark v4, a benchmark of how well models judge creative writing, Claude Opus 5 scores 78.8%; its error bars cannot distinguish it from the best open-weight model, GLM 5.2, at 73.2%, its predecessor, Claude Opus 4.8, at 78.0%, or Gemini 3.1 Pro Preview at 78.7%, while Claude Opus 4.6 scores 90.7%.

• On AA-Omniscience Accuracy, a benchmark of broad factual accuracy across domains, Claude Opus 5 (max) scores 54.2%, ahead of the best open-weight, model Kimi K3, at 46.0%, and its predecessor, Claude Opus 4.8, at 46.6%, but behind Gemini 3.1 Pro Preview at 55.2% and Claude Fable 5 at 61.4%.

Coding / Agents

• On Vibe Code Bench v1.1, a benchmark of end-to-end web-application builds with unrestricted terminal access, Claude Opus 5 scores 88.4%, statistically indistinguishable from Claude Fable 5 at 90.4%, the best open-weight model, Kimi K3, at 85.0%, and Claude Opus 4.8 at 82.7%.

• On ProgramBench, a benchmark of rebuilding behaviorally equivalent programs from executables and usage docs, Claude Opus 5 scores 82.3%, ahead of the best open-weight model, GLM 5.2, at 62.6%, and its predecessor, Claude Opus 4.8, at 71.9%; its result is indistinguishable from GPT-5.6 Sol’s 77.6% within the benchmark’s error bars.

• On CursorBench, a benchmark of ambiguous, multi-file coding tasks drawn from real Cursor sessions, Claude Opus 5 (max) scores 70.0%, behind Claude Fable 5 at 70.5% and ahead of the best open-weight model, Kimi K3 at 60.8%, as well as Claude Opus 4.8 at 62.3%.

• On HiL-Bench, a benchmark of tasks where models may ask a human for help, Claude Opus 5 scores 57.0%, though its error bars cannot distinguish it from Claude Fable 5 at 56.3% or the best open-weight model, GLM 5.2 at 43.7%; Claude Opus 4.8 scores 35.3%.

• On Handbook, a benchmark of long-context agentic instruction-following in RL environments modeled on following a company handbook, Claude Opus 5 (max) scores 32.3%, behind Claude Fable 5 at 36.2% and ahead of the best open-weight model, GLM 5.2, at 12.7%, and Claude Opus 4.8 at 21.9%.

• On Zapier Benchmarks, a benchmark of multi-app automation tasks, Claude Opus 5 (max) scores 26.2%, ahead of Gemini 3.6 Flash at 19.8%, GLM 5.2, the best open-weight model, at 14.0%, and its predecessor, Claude Opus 4.8, at 17.2%.

• On BrowseComp, a benchmark of web-browsing agent tasks requiring multi-hop information retrieval, Claude Opus 5 scores 90.8%, behind Kimi K3 at 91.2% and ahead of GPT-5.6 Sol at 90.4%; its predecessor, Claude Opus 4.8, scores 84.3%.

• On GBA Eval, a benchmark of writing a working Game Boy Advance emulator, Claude Opus 5 scores 79.6%, ahead of Claude Fable 5 at 74.5% and its predecessor, Claude Opus 4.8, at 70.9%; the best open-weight model, Kimi K3, scores 48.3%.

• On DeepSWE, a benchmark of real software-engineering issues designed as a harder, less gameable replacement for SWE-Bench, Claude Opus 5 (max) scores 73.6%, with error bars unable to distinguish it from GPT-5.6 Sol at 72.7%, Claude Fable 5 at 69.9%, and the best open-weight model, Kimi K3, at 68.5%; its predecessor, Claude Opus 4.8, scores 59.0%.

• On FrontierCode 1.1 Main, a benchmark of maintainer-crafted production-code tasks graded for mergeability and code quality, Claude Opus 5 (medium) scores 53.4%, behind Claude Fable 5 at 53.5% and ahead of GPT-5.6 Sol at 47.5%, Claude Opus 4.8 at 46.5%, and the best open-weight model, Kimi K3, at 44.2%.

• On Tau3 Banking (AA), a benchmark of agentic customer-service tasks in a banking setting, Claude Opus 5 (high) scores 32.8%, ranking #3 of 94, behind Kimi K3 at 33.4% and GPT-5.6 Sol at 33.0%, and ahead of its predecessor, Claude Opus 4.8, at 27.6%.

• On Senior SWE-Bench, a benchmark of senior-level real-world software-engineering tasks resolved by LLM agents under a fixed harness, Claude Opus 5 (xhigh, mini swe) scores 28.2%, ranking #2 of 16 behind Claude Fable 5 at 29.1% and ahead of Claude Opus 4.8 at 25.0% and the best open-weight model, MiniMax M3, at 13.8%.

Math

• On ProofBench, a benchmark of formal-math tasks where models produce Lean 4 proofs for advanced undergraduate and graduate problems, Claude Opus 5 scores 78.0%; its error bars cannot distinguish it from Claude Fable 5 and GPT-5.6 Sol at 77.0%, the best open-weight model, Kimi K3, at 70.0%, or its predecessor, Claude Opus 4.8, at 69.0%.

• On Riemann-Bench, a head-to-head Elo benchmark of extreme-tier mathematical problems requiring deep reasoning, Claude Opus 5 (max) scores 68.0%, ranking #2 of 24 behind GPT-5.6 Sol at 74.4% and ahead of Claude Fable 5 at 60.0% and Claude Opus 4.8 at 47.2%.

• On OTIS Mock AIME, a benchmark of 45 competition-style math problems from OTIS, Claude Opus 5 (max) scores 98.9%, statistically indistinguishable from GPT 5.5 at 100.0%, the best open-weight model, Kimi K3, at 97.2%, and its predecessor, Claude Opus 4.8, at 98.3%.

• On FrontierMath (Tiers 1-3 v2), a benchmark of research-level math problems, Claude Opus 5 (max) scores 85.6%, with error bars unable to distinguish it from GPT-5.6 Sol at 89.1%, GPT-5.6 Terra at 86.0%, or Claude Opus 4.8 at 80.0%.

• On FrontierMath Tier 4 (v2), the hardest tier of FrontierMath covering research-level mathematics problems, Claude Opus 5 (max) scores 73.2%; its error bars cannot distinguish it from Claude Fable 5 at 87.8%, GPT-5.5 Pro at 78.0%, GPT 5.5 at 72.5%, or its predecessor Claude Opus 4.8 at 56.1%, and it scores above the best open-weight model, Kimi K3, at 39.0%.

ML

• On WeirdML, a benchmark of nonstandard ML engineering tasks where models write PyTorch for novel datasets and iterate from execution and test feedback, Claude Opus 5 (max) scores 91.8%, statistically indistinguishable from Claude Fable 5 at 91.9%, ahead of GPT-5.6 Sol at 88.8% and its predecessor, Claude Opus 4.8, at 82.9%.

Games

• On Kaggle Game Arena, a benchmark of head-to-head strategy-game play, Claude Opus 5 scores 287, placing #2 of 26, behind GPT 5.5 at 330 and ahead of GPT-5.6 Sol at 264; its predecessor, Claude Opus 4.8, scores 123.

• On RuneBench, a benchmark of long-horizon skilling agents in Old School RuneScape scored by experience rate across sixteen skills, Claude Opus 5 scores 5.73, ranking #4 of 41, behind Claude Fable 5 at 6.01, GPT-5.6 Sol at 5.9, and GPT-5.6 Terra at 5.88, while improving on Claude Opus 4.8’s 5.08.

• On Chess Puzzles, Stockfish-generated chess puzzles solved by exact best-move match, Claude Opus 5 (max) scores 42.0%; its error bars cannot distinguish it from best the open-weight model, Kimi K3, at 39.0%, its predecessor, Claude Opus 4.8, at 34.0%, or Claude Fable 5 at 41.0%.

STEM

• On EpiBench, a benchmark of epigenetics prediction tasks, Claude Opus 5 (pi) scores 35.2%; its error bars cannot separate it from GPT 5.5 at 45.0%, Claude Opus 4.8 at 39.0%, or the best open-weight model, Kimi K2.6, at 24.5%.

• On Humanity’s Last Exam, a benchmark of expert-level questions across many academic fields, Claude Opus 5 (max) scores 52.6%, behind Claude Fable 5 at 53.3% and ahead of GPT-5.6 Sol at 47.2% and its predecessor, Claude Opus 4.8, at 45.7%.

• On CritPt, a benchmark of unpublished research-level physics problems, Claude Opus 5 (max) scores 29.1%, ahead of the best open-weight model, Kimi K3, at 23.4% and its predecessor, Claude Opus 4.8, at 20.9%, but behind GPT-5.6 Sol at 32.3%.

Multimodal / Computer Use

• On GDP-PDF, a benchmark of multimodal reasoning over real-world prompts and PDFs from expert professional workflows, Claude Opus 5 (max) scores 24.0%, matching its predecessor, Claude Opus 4.8, and beating the best open-weight model, Kimi K3, at 19.0%, while GPT-5.6 Sol leads with 30.7%.

Miscellaneous

• On BullshitBench, a benchmark of detecting unsubstantiated or manipulative claims, Claude Opus 5 (low) scores 80.2%, ahead of the best open-weight model, Qwen3.5 397B A17B, at 78.0%, but below its predecessor, Claude Opus 4.8, at 95.0% and Claude Sonnet 5 at 80.8%.

Games / Reasoning

• On VoxelBench, where human raters vote pairwise on voxel builds generated from text prompts, Claude Opus 5 (max) scores 2222, with its error bars indistinguishable from GPT-5.6 Sol’s 2270 and Claude Fable 5’s 2197; it exceeds the best open-weight model, Kimi K3, at 2040 and Claude Opus 4.8 at 1701.

• On MineBench, where human raters vote pairwise on Minecraft builds from a rotating prompt set, Claude Opus 5 scores 2135, ahead of Claude Opus 4.8’s 1772 and Kimi K3’s 1701, the best open-weight score; its 2042 score is indistinguishable from GPT-5.6 Sol’s within the benchmark’s error bars.

Science

• On scBench-Long, a benchmark of long-horizon single-cell analysis through multi-step bioinformatics pipelines, Claude Opus 5 (pi) scores 41.3%, though its error bars cannot distinguish it from GPT-5.6 Sol at 38.1% or Claude Opus 4.8 at 25.4%; it scores higher than the best open-weight model, Kimi K3, at 12.7%.

Reasoning

• On ARC-AGI-1, a benchmark of abstraction-and-reasoning grid puzzles, Claude Opus 5 (max) scores 97.5%, behind Gemini 3.1 Pro Preview at 98.0%, ahead of the best open-weight model, GLM 5.2, at 77.0%, and above Claude Opus 4.8 at 92.5%.

Knowledge / Science

• On GPQA Diamond (Epoch), Epoch’s uniform-harness GPQA Diamond runs, Claude Opus 5 (max) scores 93.9%, statistically indistinguishable from GPT-5.4 Pro at 94.6%, Kimi K3 at 93.1%,the best open-weight model, and its predecessor, Claude Opus 4.8, at 91.0%.

Games / Agents

• On GBENCH, a benchmark of head-to-head performance across competitive game environments, Claude Opus 5 scores 75.9%, ahead of Kimi K3 at 68.0%, the best open-weight model, and its predecessor, Claude Opus 4.8, at 64.6%; Claude Fable 5’s 73.4% result is indistinguishable under the benchmark’s error bars.

Opus 5 showcases

• Opus 5 plays Portal (apparently better than Fable, but note n = 1)

• Opus 5 makes for some incredible painterly worlds in self-contained HTML files

Minor

• Substack partners with Pangram to detect AI work. It seems like a good, scalable, practical epistemic intervention.

• WSJ reports that large companies are hiring humans again. Expecting productivity gains from AI, large American companies have been reducing their workforces and hiring efforts. But the economic benefits of AI proved to be less clear-cut. Firms are coming to the view that AI cannot totally replace entry-level jobs (yet), as some companies betted, but must instead work alongside humans.