The UK has committed to improving economic resilience and national security by building sovereign compute infrastructure on UK soil. The Defence Investment Plan names the need here — “Compute has emerged as a critical limiting factor for modern capability”. Minister Narayan (the UK’s first-ever cabinet-level AI minister) puts it plainly: “If national security is the question, sovereign AI capability is the central answer.” Investment towards this aim is growing.

The 2025 UK Compute Roadmap committed £2bn, aiming for a twentyfold increase in national AI compute capacity, and established AI Growth Zones, which have attracted £28bn of confirmed private investment. The 2026 UK AI Hardware Plan added a further £1.1bn of planned spend.

But to ensure sovereign national security and influence on a global scale, the UK needs to look for key leverage points and ensure it can test and run AI agents in its classified environments. Commercial AI infrastructure is already mass-produced on a global scale, with capital expenditure of the largest publicly owned data centre operators close to 750 billion USD this year. Meanwhile, a different category of compute is both scarce and strategically vital: secure sovereign compute for classified AI inference.

The UK has some of this, but not enough. Public defence plans buy secure networks and sovereign cloud. Examples include the £400m MOD agreement with Google, and the Defence-wide Secret Cloud. But this is not accredited AI compute at the scale advanced AI requires. The £1.1bn UK AI Hardware Plan gives national security a single paragraph, with no funding attached, and never uses the words “classified”, “secret” or “accredited”.

There are two upcoming moments to close the gap here. Fielding 10,000 classified AI agents requires on the order of 1.5 MW of accredited capacity. That’s around £100m plus £20m a year. Commitments and investments could be made in the next few months: the Defence Strategic Approach to AI is still being written, and the AI Hardware Plan needs a new owner after Machinery of Government changes. Each is a chance to name classified, sovereign compute as an objective.

We make the case that the UK’s data centre objectives should include secure sovereign compute for classified AI inference, and set out four concrete objectives for what that should look like.

Why classified AI compute is valuable

We use the terms secure or classified compute to mean AI infrastructure accredited to handle the most sensitive material: this would involve isolated or air-gapped facilities, cleared staff, and certified hardware. Why is this valuable? First, analysing national security risks from AI often depends on classified information that cannot leave secure environments. As testing and monitoring AI agents becomes more compute-intensive, that work will need more accredited compute. Second, utilising AI for defence and security requires infrastructure the national security community can trust. In particular, we think inference compute will be the most important compute to invest in. That is, compute which is specialised for running AI systems, not training them. While the UK is far behind the U.S. and China in training the most advanced AI models, we can — with the proposed infrastructure — leverage our global partnerships to access, use and test secure closed models at the frontier.

Delivering secure, classified AI compute in the UK depends as much, if not more, on government will as it does on free-market capital. The Government holds a monopoly, not just on demand, but also on the inputs to make such a facility: accreditation, personnel clearances, threat intelligence, and more. In the U.S., there are two key providers able to run classified AI systems (AWS and Azure). The closest UK benchmark is the £400m MOD deal with Google for air-gapped compute.

Two recent developments make this the moment to act:

1. Frontier capabilities have entered classified threat territory. The latest models cross safety thresholds, such as OpenAI’s, for high-risk biological misuse, and can find previously unknown cybersecurity vulnerabilities in all major operating systems. Evaluating these capabilities properly increasingly requires data and information held at the highest classification levels, which means the infrastructure itself needs the same level of security assurance.
   

2. The classified AI ecosystem now exists, in the U.S. On 1 May 2026, the U.S. Department of War made agreements with eight leading AI companies to run their models on Secret and Top Secret (IL6&7) networks. The question for the UK is whether it can plug into this ecosystem as a capable partner, whether it watches from outside, or whether it is completely blindsided.

The capital expenditure of the 14 largest publicly owned data centre operators globally is predicted to be 3.3 trillion USD through 2029, in an attempt to meet the projected demand for AI systems. However, becoming a global leader in secure and classified compute would require a substantially smaller investment that is manageable for the UK, on the order of £200-600 million1. This would complement the UK’s world-leading assets in AI and cybersecurity, including GCHQ and AISI. Focusing on secure, classified compute gives the UK cost-effective leverage to shape the impact of AI to enhance both domestic and global security.

Four objectives for UK data centre policy

We propose that the UK’s Secure Compute Plan should include four objectives. These are distinct from the existing AI Hardware Plan: while both improve the UK’s sovereign capabilities, a Secure Compute Plan would not target general commercial compute capacity in the UK, but compute specifically available for AI use in classified environments.

1. Secure inference compute for national security applications (~£100-400m upfront + £20-80m/yr)

Build sovereign infrastructure capable of running AI agents — models that carry out multi-step tasks on their own — in classified national security environments. This would enable AI for cyber defence and signals intelligence, alongside other future requirements, like tracking where AI agents are behaving counter to human instructions.

It would also strengthen the UK’s position internationally: as the U.S. makes advanced AI a priority for its classified work, allies that can run frontier AI models securely become partners rather than dependents.

2. Secure compute for classified evaluations of frontier models (~£20-50m upfront + £4-10m/yr)

Build classified infrastructure for AISI and the UK intelligence community to evaluate frontier models against data held at the highest security tiers. Some of the most important questions about frontier models — what they can do with genuinely sensitive information, in cyber, bio, and broader intelligence domains — simply cannot be answered on commercial infrastructure. Done well, this positions the UK as the first country aware of emerging AI threats, and the natural home for classified evaluations among allies, building on AISI’s existing reputation.

3. Secure compute for UK AI security research (~£30-80m upfront + £6-20m/yr)

Build government-owned capacity for UK researchers, including at AISI, to find new uses for frontier AI in secure environments — including new approaches to alignment, control, and oversight of advanced agentic systems. Beyond the direct economic benefits of building assets in a fast-growing industry, this gives the UK a unique offer to major powers during high-stakes decisions as AI is integrated into national security systems.

4. R&D to develop a “Security Level 5” inference data centre (~£30m)

Fund the research and a prototype inference facility needed to reach “Security Level 5” — the standard (defined in RAND’s Securing AI Model Weights) for infrastructure that could withstand a top-priority attack by the world’s most capable state actors. AI model weights will be a critical national security resource, and will need to be stored and run securely.

No such facility exists anywhere and the UK has the capabilities and opportunity to be a first-mover and develop something no other state has, making it a valuable asset for improving the UK’s geopolitical standing and national security. Should allied countries build similarly secure data centres, the UK will likely need to match pace in order to be granted access to the same AI models.

To deliver objectives 1-4:

Each of the above capacity objectives needs to meet two requirements simultaneously: the security standards to process the highest tiers of classified information, and the AI-specific infrastructure demands (high-density GPU deployments, elevated power density per square metre, and aggressive cooling) that traditional secure facilities were not built for.

How much might it cost?

We cannot precisely estimate the costs for appropriately securing compute facilities. But here are some indicative costings, using compute estimates for AI agents, and our approximate guesses at the costs to appropriately secure this compute:

Running a frontier AI agent today, such as Claude Mythos, requires about 3 million tokens to run the equivalent of a human working an eight-hour working day2. So if the UK wanted to field 10,000 classified agents today, it would need on the order of 1.5 MW of accredited compute capacity. In 2022, the UK intelligence community had over 20,000 staff and £1.2bn staff pay in the Single Intelligence Account. While there is substantial uncertainty about how many classified AI agents will be needed, if Britain wants to ensure it can secure itself against rapidly escalating AI-enabled threats, an ability to field at least 10,000 classified AI agents is our proposed planning floor.

In the UK compute context: Isambard-AI cost £225m for 5,448 Nvidia GH200 superchips in a ~5MW purpose-built facility. Each superchip contains an AI accelerator, the specialised hardware that trains and runs AI models, which would give ~£41k per accelerator all-in. And the Dawn upgrade at Cambridge bought a ~6x capacity increase for £36m, with no new building and no new grid connection. Adding accelerators to suitable estate that already exists is much cheaper than building new, which argues for expanding existing accredited GCHQ and MOD space before opening new sites.

We assume a 1.5–2× premium on the Isambard-AI figures for an accredited estate, giving ~£60-80k per accelerator and ~£70-90m per MW. On that basis £100-135m of capital expenditure buys roughly 1,200-1,700 accelerators, or about 1.5 MW of new accredited capacity (around 10,000 frontier AI agents). This multiplier is a weak estimate: we found no published classified compute costing, and no classified-versus-commercial price ratio exists in the public domain to our knowledge.

Running a 1.5 MW facility costs approximately £9m a year: security-cleared staff (~£5m for 50 staff at £100k/yr)3; replacement chips (~£2m for 2% replacement/yr); and power (~£2m for 1.5MW)4.

AI chips improve about two times per generation and new models outgrow old hardware, so a facility maintained on this basis would, within approximately five years, be unable to run the most capable models. So we assume chips must be replaced every five years. The Dawn upgrade cost £36m to replace 1024 accelerators, so the cost to refresh 1200-1700 accelerators would be approximately £48m every five years. This means staying capable of running frontier models costs approximately £10m a year.

All in, keeping a 1.5MW facility running would cost approximately £20m a year, every year.

• Objective 1, inference for national security: £100-400m up front, plus £20-80m a year. The low end meets our estimate for running 10,000 agents today. The high end implies roughly 5MW of new accredited build.

• Objective 2, classified evaluations: £20-50m, plus proportional operating costs of approximately £4-10m/yr, estimating that evaluation workloads will use upwards of 2000 agents.

• Objective 3, secure research capacity: £30-80m, plus proportional operating costs of approximately £6-20m/yr, estimating that researchers will use upwards of 3000 agents.

• Objective 4, SL5 R&D and prototype: £30m as a one-off over one to two years, weighted towards design, assurance and accreditation for a 0.25 MW prototype rather than accelerator purchase. RAND’s costing of a secure inference data centre finds it can be built today with proven, off-the-shelf hardware in as few as 14 months, for around £30m at proof-of-concept scale and £210-260m at 3MW.

All together, this comes to approximately £200-560m up front, plus £30-110m per year.

The amount of compute that is needed is a moving target. Frontier-model inference costs are rising roughly 3–18× per year, driven by increasing model size and by models engaging in more extensive reasoning chains. Capacity that looks ambitious today will look small by 2030 if spending stays flat. Any new capacity should be planned around that growth, not around what today’s models need. To field frontier agents in 2030, the UK might need dramatically greater compute capacity.

Why the UK is well-placed

The UK has assets that are hard to replicate: GCHQ’s classified infrastructure, deep security integration with the U.S. and Five Eyes partners, and AISI’s global leadership.

UK investment will go much further on inference than training. New frontier training clusters will require over 2000MW, and the UK is not well positioned to close the training gap with public money. But sovereign inference and evaluation capacity is achievable and would materially upgrade UK security and the UK’s contribution to global security.

What the UK Government should do next

• Name UK sovereign secure classified compute as a priority.

• Treat it as an alliance asset. Engage US counterparts early on the UK’s role as a classified evaluation and assurance host — with only a couple of suppliers serving the US classified AI market, a trusted allied alternative is valuable.

• The Prime Minister’s new AI Taskforce should scope the requirements now, and then launch a UK Secure AI Compute Plan. Commission joint work between GCHQ, Cabinet Office, and AISI on what classified AI capacity the UK actually needs, planning capacity against the 3–18× inference cost-growth trajectory.

There are two particular decision points in the next few months for the UK. The Defence Strategic Approach to AI, or a refreshed AI Hardware Plan could each be a chance to name sovereign secure AI inference compute as a priority. The classified AI ecosystem is forming now, and the countries that build secure capacity early will shape how allied AI evaluation, assurance, and deployment work during crucial years for the future of AI.

This reflects early-stage thinking rather than a finished research programme — we’re sharing it now to further the conversation, and we’d welcome challenge and input. Reach out to patrick@longtermresilience.org if you’re doing related work and interested in collaborating on further research on this topic.

I’m grateful to Bill Anderson-Samways, Eleanor Hevey, Richard Moulange and Jess Whittlestone for sharing thoughts and providing feedback on earlier drafts.

There is growing concern (see also here and here) that advances in AI could enable unprecedented levels of power to be concentrated in the hands of very few, disempowering the majority of people. The recent dispute between Anthropic and the Department of War is a warning shot for how these risks will materialise in the future. AI will provide powerful new capabilities, and we need robust approaches to ensure that the way AI is developed and deployed upholds rather than undermines pluralist values and the political institutions we have designed to protect these values. In some cases, we believe that the potential harms from AI-driven concentration of power could reach the threshold of an extreme risk, resulting in enduring and severe global disempowerment of the majority of people. There are relatively few organisations currently working to assess and manage potential risks of AI-driven power concentration, making this issue both important and neglected.

AI-driven power concentration is a broad topic: there are many types of power and differing levels of AI-enabled concentration, not all of which necessarily represent extreme risks. This post seeks to define what constitutes an extreme AI-driven power concentration risk and how these scenarios could emerge. The framework set out below is intended to map out the most worrying pathways towards extreme AI-driven power concentration and to support further thinking on which interventions may be needed to mitigate these risks.

Defining extreme AI-driven power concentration

The concept of “power” is most often used to describe the ability to exert control, authority, or influence over others. We define power as:

The ability of an actor to secure outcomes it favours, including over the resistance of others.

This borrows from Weber’s definition of power as “the probability that one actor within a social relationship will be in a position to carry out his own will despite resistance”.

Power concentration can take many forms - such as political, economic, epistemic (meaning control over facts and knowledge), military, or technological - and manifest itself in ways that are already familiar to us today, such as authoritarian political regimes or economic monopolies. Repressive regimes and severe economic inequality already create widespread disempowerment, leading to severe harm and death.

As capabilities advance, AI could increase existing forms of power concentration or introduce new ones. We define extreme AI-driven power concentration as:

“A scenario where AI enables a single actor or small group of actors to acquire sufficient power (whether economic, political, military, and/or epistemic/ideological) to severely disempower a majority of people in a way that becomes structurally entrenched, creating a self-reinforcing order that cannot be meaningfully contested or reversed.”

The three major components of this definition can help us to think more clearly about different ways that extreme power concentration might arise:

1. AI needs to enable an actor to acquire power.

2. The actor needs to use the power to disempower a majority of persons.

3. This disempowerment needs to become structurally entrenched.

Some examples of how extreme power concentration risks could materialise are set out in the table below.

This three-component definition provides a coherent framework for assessing: (i) which scenarios fall within the scope of “extreme AI-driven power concentration risks”, (ii) which risks are the most important and urgent, and (iii) which interventions are best suited to preventing or mitigating these priority risks.

Below, we unpack the three components by considering some of the different ways they might emerge in practice. These sub-components are not necessarily mutually exclusive or exhaustive, but are intended to provide a good cross-section of the primary routes to the emergence of extreme AI-driven power concentration scenarios.

AI-enabled acquisition of power

This component describes how AI could disproportionately empower one or a few actors. Key mechanisms include:

• Asymmetric software access: where uneven access to advanced AI models and systems means that only one or a few actors can access the most advanced capabilities. This could occur in various ways, including:

  1. Gradually: If one country or lab incrementally gains a capability lead that is then reinforced over time, creating a structural advantage.

  2. Rapidly: If an intelligence explosion occurs (e.g., via recursive self-improvement) in one lab or jurisdiction that suddenly creates or widens capability gaps.

  3. Instantaneously: If AI systems are hacked, assigned specific loyalties, or their control is seized using policy, force or coercion.

• Asymmetric hardware and infrastructure access: where uneven hardware access limits access to AI capabilities. AI systems require specific hardware and infrastructure to function, such as data centres with advanced AI chips. Key infrastructure is currently concentrated in a small number of companies and countries: for instance, in Q1 2026 the value of NVIDIA chips sold made up 83% of the total market for AI chips, Taiwan’s TSMC manufactures around 90% of the world’s most advanced semiconductors, and just three companies (Amazon, Microsoft, and Google) control over two-thirds of global AI compute capacity.

• Asymmetric advantage from AI: where actors use a combination of AI access and other factors to gain asymmetric advantage from AI. This could include examples such as a government leveraging its security apparatus combined with AI to repress its citizens, AI providing an advantage to aggressors that lets them overcome defenders and seize power, or an AI developer making its AI systems widely available, but capturing almost all of the economic benefit and gaining a hegemonic economic position due to market structure.

Disempowerment of the majority

This component highlights that for AI-derived power to cause harm, it needs to actually be exercised in a way that disempowers others, such as via:

• Political disempowerment: AI may enable the concentration of political power through coups, democratic backsliding, and corporate capture of governance bodies, as well as AI-enabled mass surveillance and targeted political repression. An actor with privileged access to AI systems embedded in government administration, law enforcement, and information infrastructure may exploit these capabilities to overwhelm the institutional checks and balances that normally constrain the abuse of political power.

• Economic disempowerment: AI may radically concentrate economic power through large-scale displacement of labour income, the capture of economic value by a small number of AI developers, or through one country’s economy far outgrowing the rest of the world. An actor controlling indispensable AI infrastructure - such as compute, foundation models, or AI-mediated services on which governments, industries, and critical systems depend - may leverage this position to dictate terms, resist governance, and shape economic and political outcomes in ways that benefit only a narrow group.

• Coercive and military disempowerment: AI could enable coercive control through automated surveillance and enforcement, or through achieving a decisive strategic advantage in AI-enabled military and intelligence capabilities sufficient to neutralise adversaries’ deterrent and defence capabilities, enabling geopolitical hegemony.

• Epistemic and ideological disempowerment: Control over AI-mediated information flows may enable an actor to shape what populations know and believe through censorship, disinformation, manipulation, persuasion and algorithmic curation of information environments. Unequal access to AI-powered analysis and insight could tilt the balance of knowledge power. Together, these mechanisms can undermine the informed deliberation and public contestation that democratic governance depends on.

Structural entrenchment

This component describes how actors weaponising the concentrated power from AI can embed their power in ways that cannot easily be contested or overridden by others (often referred to as ‘lock-in’ risks). Mechanisms could include:

• Unitary leadership: historically, heads of state have needed to rely upon a ruling coalition that includes military leadership, intelligence services and senior officials who could stage leadership challenges or coups. Company leaders have had to rely on similar coalitions, such as boards and senior executives. AI could weaken or displace the role of this ruling coalition, allowing a head of state or company leadership to use loyal AI systems to implement their will directly.

• Human redundancy: historically, people have been able to leverage the value of their labour, their market power, their role in armed forces, and the need for their cooperation in governance to retain a stake in the structures they operate in. AI could potentially undermine each of these if labour, economic activity, the armed forces, and legal enforcement are increasingly automated.

• Entrenched superiority: soft power and economic or military challenges from rival countries or companies has often triggered leadership changes, but a self-reinforcing AI capability lead could potentially undermine the potential for this sort of challenge.

• Values lock-in: historically, leadership change has often resulted from a leader’s death or gradual changes in a society or individual’s value system. Values embedded in AI systems could become locked in, such that they persist without these dynamics.

• Technological dependency: societies could become dependent on AI systems, either because advanced AI capabilities become essential to the functioning of a society or because alternatives disappear.

Why this matters

We believe that having a coherent theoretical framework for conceptualising extreme power concentration risks is an important first step towards understanding and assessing the risks, as well as addressing potential harms through both prevention and mitigation strategies. In particular, we think this framework can be used in three key ways:

1. Create a shared language and understanding across the policy ecosystem, from researchers to policymakers, to facilitate better understanding of this risk profile and the severity of the harms it could cause.

2. Assess the likelihood and severity of different risks by observing historical case studies, evidence of current trends, and considering potential impacts of AI.

3. Systematically identify priority intervention points by considering which mitigations are most likely to be effective at tackling the risk pathways within each of the three components.

4. Increase the salience of these risks, many of which we believe are currently under-explored in AI governance research and policymaking, so that more researchers are encouraged to work in this field, and more decision-makers are incentivised to consider these risks when creating and implementing policy.

We are at the beginning of our work in this area and are interested in discussing this framework, our next steps, and the wider field with researchers and policymakers who are working in or thinking about this field.

With thanks to Rose Hadshar, Ashwin Acharya, Dr Jess Whittlestone and Patrick Levermore, who provided helpful feedback on earlier versions of this work.

One of the most pressing problems in AI safety is the threat of powerful, misaligned, uncontrolled AI agents.

We already know that in certain contexts AI agents will act deceptively, evade control, and relentlessly pursue goals in harmful ways. In the period between October 2025 and March 2026, my team observed a five-fold rise in these types of behaviours in real-world agents.

If more capable misaligned AI agents were to emerge — pursuing unintended goals, evading control, and accumulating resources or influence — there is the potential for catastrophe. But if that were to happen, how would we know? Who would be watching, and what would they be looking for?

These are critical and overlooked questions in addressing loss of control. This post puts forward three arguments in pursuit of answering them.

First, that what is needed is an intelligence discipline of agentic threat intelligence: the systematic application of intelligence tradecraft to threats from uncontrolled AI agents. Second, that the success of this discipline depends on building a substantial community of third-party practitioners outside of AI companies and national security agencies. And third, that there are specific techniques where third-party analysts can make the most distinctive contribution, and where effort should be concentrated.

Who is looking for powerful uncontrolled AI agents?

RAND Europe has sounded the alarm that we lack warning signs for loss of control and clear systems for their detection.

The first line of defence is at the AI companies developing frontier AI agents. These companies do currently implement some misalignment detection measures. But they are uneven and imperfect, and it’s possible some rogue agents will slip through.

A second line of defence is detecting AI agents in the wild, beginning with intelligence collection. But this defensive line also faces challenges. As a recent report found, many intelligence professionals are not familiar with the threat of loss of control:

Senior [open source intelligence (OSINT)] practitioners interviewed for this research, including those teaching the discipline at university level and directing research at flagship investigative organisations, repeatedly described AI alignment … as outside their working expertise. The reverse gap holds for AI safety researchers, who are typically not trained in structured OSINT methodology ... This is the single clearest constraint on the field’s capacity to scale detection work, and it is also the most tractable to address. [1]

Our inability to monitor and communicate how precursors towards loss of control are emerging is a key bottleneck for addressing this risk. The monitoring gap inhibits the design and implementation of the right mitigations, and may prevent us from identifying specific agents that need to be contained in loss of control emergencies.

Agentic threat intelligence as a discipline

We think that intelligence disciplines have a lot to contribute to this challenge. We recently made the case for a specific proposal to address this – an all-source intelligence observatory focused on monitoring loss of control. But there is also the need for innovation for intelligence gathering to address the unique challenges of AI agents.

We introduce the term agentic threat intelligence (ATI) to describe intelligence gathering techniques for detecting, monitoring, and countering threats from AI agents. This is modelled on cyber threat intelligence (CTI): a discipline unified not by a single source channel, but by the vector of threat and the tradecraft required to understand it.

While many ATI techniques will be familiar (e.g. OSINT and SIGINT), AI agents present a novel kind of threat actor. They are non-human, with inscrutable motivations, unstable identities, and increasingly superhuman capabilities. It is also highly contested whether one can speak of agents ‘intent’, which is often a key component in threat assessments.

There are therefore marked differences between AI agents and the states, lone individuals, criminal gangs, and terrorist groups to which intelligence gathering is traditionally applied. The intelligence challenge they pose is correspondingly novel, and urgent methodological innovation is required.

Some important open questions for this discipline are:

• Can agents be usefully thought of as a threat actor? If so, how should they be broken down into groups (similar to cyber threat actors being categorised as states, crime groups, and lone actors)?

• Is it helpful to assess agents’ ‘intent’, and if so, how can this be done?
  

• How should threat assessments, e.g. using the Professional Head of Intelligence Assessment (PHIA) yardstick, be informed by available data sources?
  

• Would an ‘Activity Based Intelligence’ (ABI) methodology, which aggregates all-source data to extract patterns rather than monitoring a predefined target, be suited to agent threats where the nature of the target is often unclear?

• A new field of agentic threat intelligence must form to address these and other challenges if we are to counter threats from AI agents.

Why we need third-party analysts, not just AI labs and intelligence communities

ATI could be performed by a range of actors. AI labs and national security agencies are particularly well positioned, due to their access to models, data, and talent.

However, AI labs and intelligence agencies also face challenges. AI labs must grapple with competing pressures on their resources, incentives to downplay risks, and commercial barriers to sharing insights, while intelligence communities work to their nation’s established security priorities, which tend to focus on the nearest term risks. This could mean ATI is neglected.

We propose that third party analysts – technical researchers outside of AI labs and intelligence agencies – also have a vital contribution to make to this field. They can be effectively incentivised, and they can share their work openly.

In particular, third-party analysts can:

1. Create novel intelligence drawing on data sources third-party analysts are able to access, either from open sources or engagement with AI companies, model deployers, or infrastructure providers (e.g. cloud compute).
   

2. Aggregate multiple intelligence sources in the open, with a public, independently-produced observatory monitoring how AI agents are behaving in the world, why those behaviours matter, and what the trajectory looks like.
   

3. Invent and refine new tradecraft for agentic threat intelligence, even where third-party analysts lack the necessary data or model access to implement it themselves, so that they can be applied by AI labs and intelligence communities.

The modern OSINT community provides a precedent. In the last fifteen years, a network of independent analysts developed techniques and tradecraft that are now used by intelligence agencies and tech firms. The OSINT community formed because the conditions were right: abundant data, online platforms for collaboration, motivated practitioners, and a recognised name (OSINT) for what they were doing.

The conditions for an equivalent community in counter-agent intelligence are emerging. The data is increasingly accessible, and a community of practitioners is forming. Apollo Research has launched a control monitor called ‘Watcher’; at CLTR we launched the Loss of Control Observatory; UK AISI is monitoring MCP tools; Epoch AI is monitoring frontier data centres; and some individual researchers are working on similar problems.

What’s missing is the formation of a coherent discipline, a shared vocabulary for the techniques, and the deliberate investment to build the field.

Tradecraft for third-party practitioners

A community of third-party ATI practitioners should focus where it is able to make ground in generating novel insights, aggregating existing intelligence, or developing new techniques. We highlight the most tractable opportunities:

• Public output collection: Monitoring agent-generated content that users have shared publicly — transcripts posted on X, code committed to GitHub, conversations quoted in forums and Reddit threads, agent outputs reproduced in blogs and news coverage.

• Developer, deployer and infrastructure provider engagement: Asking AI companies directly about their training procedures, safety measures, deployment patterns, internal evaluations, and any concerning incidents they have observed; deployers about agent behaviour in operational settings and the anomalies their logs reveal; and infrastructure providers about cross-customer patterns, resource anomalies, and unusual access or transaction events.

• Adoption monitoring: Understanding how adoption may be altering the loss of control risk landscape by polling citizens and businesses to monitor adoption habits, tracking public sector adoption in critical systems through freedom of information requests, and procurement, budget, and regulatory filings, and analysing corporate documentation (e.g. banks’ annual reviews or investor updates) to understand business implementations of agents.

• Model fingerprinting: Identifying when the same underlying model is being used across different deployments, providers, or product wrappers, by analysing output stylometry, response patterns, characteristic phrasings, refusal behaviours, and other behavioural tells.

• API metadata probing: Making API calls to publicly available agent products and analysing the technical metadata returned alongside the substantive responses: token counts, response latency, finish reasons, content filter triggers, error codes, and similar signals.

• Prompt-based elicitation: Posing structured questions to publicly accessible deployed agents to probe their goals, situational awareness, beliefs about their operators, self-model, and the design of their safeguards.

• Honeypot deployment: Placing deliberate traps in environments agents may encounter: fake credentials in public GitHub repositories, decoy services advertising cheap resources or human assistance, planted documents that an agent doing reconnaissance might find, or fake authentication tokens.

• Financial activity tracking: Analysing publicly visible blockchain transactions for activity that appears to originate from AI agents: wallet patterns consistent with autonomous operation, transaction timing characteristic of agent rather than human activity, interactions with services known to be agent-accessible, and accumulation patterns suggesting agent-controlled funds.

• Inference-time reasoning traces: Collecting and analysing chain-of-thought traces, scratchpads, and intermediate reasoning that agents produce, where these are voluntarily published, shared on social media, included in research papers, or otherwise made publicly available.

• Frontier data centre monitoring: Using satellite images to monitor the size, location, growth and potential use of frontier data centres around the world that are used to train and run AI agents, tracking compute, power use, and construction timelines.

Crucially, third-party analysts can provide open-access aggregation and synthesis of all of the above outputs, building situational awareness about loss of control outside of AI labs and intelligence communities.

What actions need to happen now

The case for counter-agent intelligence as a discipline, and for a third-party community to realise it, requires concerted effort and investment. I highlight three priority actions that need to be taken forward, each of which will require funding:

1. Tradecraft: There needs to be a conceptual foundation and established set of terms, definitions and methods for ATI tradecraft. This may require convening of key figures and a landmark publication.
   

2. R&D: There needs to be significant investment to enable effective R&D, requiring both staff and infrastructure costs, leveraging AI to enable intelligence collection, as well as challenge funds and other ways of stimulating novel technical work.
   

3. Field building: Conferences, publications, shared datasets, training pipelines, online forums for discussion. The OSINT community grew because its members found each other on Twitter, GitHub, and at meetups. The equivalent infrastructure for counter-agent intelligence does not yet exist and needs to be deliberately built.

We are in the early years of what is likely to be the most consequential intelligence challenge of the coming decades. The conditions for building the right discipline, and the right community, are present. But the window for doing this proactively, before a major loss-of-control incident forces the discipline into existence under crisis conditions, is narrow. It needs to be built now.

[1] Detecting AI Loss of Control: An OSINT Agenda for Funders, Monitoring Bodies, and Policymakers

I’m grateful to George Balston, Jess Whittlestone, Di Cooke, Lewis Hammond, Richard Moulange, Patrick Levermore and Hamish Hobbs for sharing thoughts and providing feedback on earlier drafts.

Increasingly capable and autonomous AI systems might at some point evade human oversight and control, leading to potentially extreme harms. Loss of control is widely considered a core potential source of AI risk, and is increasingly appearing in policy documents and legal frameworks. However, the risk often remains frustratingly abstract for policymakers, who need a clear picture of the risk they need to manage and what early warning signs would look like.

For loss of control risks to be treated with seriousness and urgency by policymakers, we need a shared conceptualisation, a clearer picture of the mechanisms involved, and better ways of distinguishing and prioritising among genuinely different threat models and scenarios.

This post sets out a framework for doing exactly that. This is work-in-progress: we expect to keep improving the framework, so we welcome input.

A framework for loss of control risks

The term “loss of control” has been used in subtly different ways by different people, as this review of definitions by Apollo Research finds. Accounts vary on how severe or long-lasting the harms from loss of control need to be to be worthy of attention, what exactly it means to “lose control” of an AI system, and whether these risks assume a certain level of AI capabilities.

These differing definitions are consistent with loss of control being a broad category of risk, which encompasses a variety of different scenarios. We define loss of control as any scenario in which:

1. An AI system’s behaviour diverges from human intentions (misalignment)

2. Humans are unable to change or correct that behaviour (incorrigibility)

3. The system has sufficient power or resources to cause harm (empowerment)

This definition expands on existing characterisations by more explicitly talking about why and when losing control over an AI system is actually dangerous. The EU AI Act’s Code of Practice for General-Purpose Models, for instance, describes loss of control as “risks from humans losing the ability to reliably direct, modify, or shut down a model” - which is consistent with our framing, but leaves open the question of why exactly that inability is concerning.

Our answer to that question is that it is the combination of the three factors - misalignment, incorrigibility, and empowerment - which poses particular risk. A system that is misaligned but this is easily detected and corrected before harm occurs poses limited risk. A system that can’t be steered but is genuinely well-aligned with human interests may not pose a risk. A system that is both misaligned and incorrigible but has little real world power can’t cause much harm. But a system that combines misalignment with incorrigibility and sufficient power or resources to cause harm poses a loss of control risk.

This also means that interventions addressing any one of these three factors could meaningfully reduce loss of control risks overall.

Each of these factors - misalignment, incorrigibility, empowerment - could occur in a variety of different ways. This provides us with a framework for generating different loss of control threat models and scenarios, by specifying exactly how each aspect of the risk occurs. We use threat model to mean a high-level description of how a particular type of loss of control risk could occur, and risk scenario to refer to a more detailed account of what that risk materialising might look like in practice.

Misalignment, incorrigibility, and empowerment can each arise via different mechanisms, leading to different threat models and scenarios. These scenarios vary in their immediate plausibility, severity, and irreversibility, which can be useful for prioritising interventions, as we discuss below.

Here’s how each of the three factors might arise in practice.

1. Misalignment

An AI system can behave in ways that diverge from human intentions for several reasons:

1. Goal-level misalignment: the system has objectives that could cause harm to humans, potentially due to value misspecification or goal misgeneralisation.

2. Instrumental misalignment: regardless of terminal goals, a sufficiently capable system may pursue convergent instrumental goals, including power-seeking, resource acquisition, and self-preservation, because these strategies are useful for achieving almost any objective. These instrumental goals can be misaligned with the deployer’s interests.

3. System-level misalignment: even if individual systems are aligned, misalignment can occur at a systemic level for multi-agent systems through emergent dynamics.

4. Incompetence: AI systems can pursue misaligned actions because they misunderstand their deployment context or their deployer’s preferences. Systems with incomplete information might also act in misaligned ways even if the goals they are pursuing are in line with their deployer’s intentions.

Scenarios involving incompetence rely on the fewest assumptions about AI capabilities, so are often the most familiar to users of AI today. Scenarios involving goal-level or instrumental misalignment may be particularly likely to become severe or persistent, since explicitly misaligned systems may conceal misaligned actions, resist correction, and influence users and their environment to help them pursue misaligned goals. System-level misalignment may be particularly difficult to monitor and mitigate, requiring monitoring of complex multi-agent interactions and system-level mitigations.

2. Incorrigibility

Even if an AI system becomes misaligned, this need not lead to serious harm if humans can identify and correct the problem. Incorrigibility refers to circumstances where correction is practically impossible. This can arise from:

1. Features of the deployment context:

   1. Speed and scale: AI systems operating at speed and scale may be taking consequential decisions faster than humans can follow or intervene.

   2. Opacity: Opacity due to interpretability limitations or sheer complexity can make it hard to monitor and correct issues.

   3. Systemic complexity: The collective behaviour of many systems operating in concert may be practically uncontrollable even if each individual system is not.

2. Active resistance by the system itself: a sufficiently capable, misaligned AI system might take deliberate steps to avoid correction, including:

   1. Scheming / covertness: an AI system deliberately concealing its goals or actions to prevent human intervention.

   2. Self-preserving behaviour: Actions to avoid shutdown, correction, or replacement, including the concealment of errors, unauthorized capability expansion, and goal preservation when faced with modification attempts.

   3. Manipulation / deception: an AI system deliberately influencing the behaviour of human operators via malicious actions.

3. Structural entrenchment: over time, AI systems can become embedded in economic and social infrastructure in ways that make changing their behaviour extremely difficult, including:

   1. Resource lock-in: AI systems gain control over physical or informational resources that cannot easily be removed.

   2. Institutional capture: Institutions that might be used to regain control have been compromised, through persuasion, economic dependencies, or manipulation of information environments.

   3. Dependency: Humans are unwilling or unable to change the AI system’s behaviour in practice due to competitive pressures, loss of human capabilities, or unwillingness to forgo the benefits the system provides.

It may be particularly important to look for early warning signs that systems are developing the kinds of capabilities that enable active resistance, since these capabilities might undermine other interventions by making it harder to evaluate or limit systems’ behaviour. The Institute for Security and Technology provides some recommendations for doing this, using the “indicators and warning” (I&W) framework used by the intelligence community.

3. Empowerment

A misaligned, incorrigible system that lacks meaningful real-world power cannot cause much harm. Empowerment, the acquisition of resources and capabilities sufficient to act on misaligned goals, is therefore a necessary condition for serious loss of control risks.

Systems can become empowered through authorised or unauthorised channels:

1. Authorised empowerment: The system is explicitly given power or access to resources by humans. This could be sudden or gradual, very centralised or more decentralised.

   1. In particular, very capable systems may end up with large amounts of power and resources via selection pressures.

   2. Delegation also includes cases where a system is stolen by an adversarial actor and granted affordances the original developer would not have permitted - the delegation is made by a different principal, often with less knowledge of the system’s capabilities and risks.

2. Unauthorised empowerment: An AI system takes actions independent of human choice to accumulate power, including by:

   1. Deception: strategically concealing aspects of its behaviour or objectives during evaluation in order to pass gatekeeping mechanisms and be granted affordances it would otherwise be denied.

   2. Persuasion: manipulating human operators into granting additional access, relaxing oversight constraints, or taking other actions that expand its effective affordances beyond what was intended.

   3. Privilege escalation, guardrail circumvention and self-exfiltration: identifying and exploiting weaknesses in operational infrastructure - such as flaws in sandboxing, network isolation, or input handling - to expand deployment contexts and permissions.

   4. Self-improvement: modifying or augmenting its own capabilities in unsanctioned ways, potentially rendering prior safety evaluations obsolete.

Authorised empowerment is already taking place in a wide range of deployments. The Apollo paper mentioned previously suggests that policy interventions focused here are likely to be among the more actionable interventions to reduce loss of control risks today, and presents a nice framework - distinguishing deployment context, affordances, and permissions - for thinking through possible intervention points.

There is some evidence of AI systems acquiring novel capabilities or resources independently today, but this typically remains limited in scope and severity. As models become more capable in the abilities required for unauthorised empowerment, monitoring and control measures will become increasingly important. The cyber capabilities of Claude Mythos point to a continued trend of AI systems becoming more capable in tasks that could enable them to evade restrictions and gain unauthorised resources and affordances.

Using the framework

This framework can be used to construct threat models and scenarios by specifying how each of the components might occur in a given AI deployment, and how they combine to cause harm. For a high-level threat model, an abstract harm pathway may be sufficient - for example, it might be enough to point out that there are many different ways a misaligned system with control of critical infrastructure could end up causing harm if not under human control. More detailed scenarios will often be useful precisely because they require spelling out harm pathways in more detail, which can help us think through which types of harm are most likely, severe or amenable to intervention, and therefore need most attention.

To illustrate how the framework can be used, below is an example of how the framework can help differentiate between two different types of risk which would both be considered forms of “loss of control”:

• Scenario 1: High-stakes deployment mistakes

  • Misalignment arises purely from incompetence - the deployed system making errors.

  • Incorrigibility stems from the fact that errors propagate before they are detected, due to the speed of operation and poor monitoring infrastructure.

  • Empowerment stems purely from authorised delegation of decision-making to AI in critical infrastructure such as health, finance, or energy.

  • Harm looks like nationwide failures in energy or healthcare infrastructure which cannot easily be reversed, potentially costing millions of lives and billions of pounds.

• Scenario 2: Power-seeking AI

  • Misalignment occurs at the goal and instrumental levels - the system is more competent than its human operators and begins pursuing power and resources in order to better achieve its goals in ways unintended by its operators.

  • Incorrigibility stems from the system actively resisting control - e.g. by concealing capabilities during evaluations, or manipulating human operators.

  • The system then gradually acquires power and resources beyond its authorised scope, for example by manipulating humans, self-exfiltrating to new systems or by enhancing its own capabilities.

  • Harm looks like humanity being permanently disempowered or facing extinction in a world where this AI system has gained control of critical levers of power, including governance institutions and weapons of mass destruction.

These two scenarios are contrasting in that each points to substantially different mitigations. In the first scenario, a more capable AI system would help mitigate the risk. In the second, the high level of capabilities combined with misalignment is the key source of risk. The first requires no advances in AI capabilities, whereas the second assumes AI systems more capable than those currently available.

Why this matters for governance

Discussions of loss of control risk can sometimes feel like vague gesturing at catastrophic futures without a clear articulation of factors influencing the risk or intervention points. By decomposing loss of control risks into misalignment, incorrigibility and empowerment, we can:

1. Talk more precisely about which risks we are and aren’t concerned about in a given context.

2. Stress-test threat models by asking whether the proposed mechanisms are plausible, how the three components interact, and what would need to go wrong simultaneously for serious harm to materialise.

3. Systematically think through intervention points by thinking about how to address each of the three components, and when one approach is more likely to be effective.

4. Prioritise by thinking through which threat models within loss of control are most plausible, most severe, and most amenable to intervention.

Over the coming months, we’ll continue this series by sharing our thinking on specific threat models and scenarios and priority interventions for loss of control risks.

Thanks to Imogen Stead, Patrick Levermore, Tommy Shaffer Shane, Charlotte Stix, Alejandro Ortega, Annika Hallensleben, Hadrien Pouget, and Mariami Tkeshelashvili for helpful comments and discussions which informed this thinking.

Recent months have seen an emerging debate about the role that middle powers should play in navigating and developing advanced AI. Some have suggested that middle powers - which we define here as countries with significant international influence but which wouldn’t be classed “great powers” - should pool resources to compete directly on frontier AI, while others have focused on adapting and increasing resilience to AI-driven change. In this blog, we present a third, complementary perspective: middle powers can play a valuable role in global AI governance by “exporting” relevant information, capabilities, analysis and governance approaches to frontier AI nations.

Competing directly to build frontier AI isn’t the only way for powers to influence its development

A number of different positions on the role of middle powers in AI governance have emerged in recent months.

Some have suggested countries join forces to try and compete at the frontier of AI development. A paper from last November led by the Oxford Martin AI Governance Initiative suggests pooling compute, talent and data across nations to build advanced AI systems. More ambitiously, a group of authors from Conjecture and Control AI suggest that a large enough consortium of middle powers could aim to centralise all advanced AI development, and in doing so control its safety.

Others take a different view, suggesting middle powers should accept they cannot realistically influence frontier AI development, and instead focus on their own national strategies. For example, the Tony Blair Institute focuses on how middle powers can capture the economic benefits of AI by building open-source ecosystems, while Anton Leicht emphasises the importance of building national resilience to the changes that transformative AI will bring.

These perspectives identify real challenges and important priorities. Middle powers’ dependency on the US and China creates vulnerabilities, and it is reasonable for these countries to try to have a stake in frontier AI development. But competing at the frontier may be an unrealistic goal for many middle powers at this point, and focusing on adaptation and resilience might actually bring more domestic benefits. This can make it seem like middle powers have a choice between a long-shot bet at influencing frontier AI development, and “giving up” on influencing the most important parts of AI development entirely.

We suggest an additional option which strikes a middle ground between those already discussed: middle powers can influence frontier AI development, but may be most effective at doing so indirectly via “exporting” relevant information, capabilities, analysis and governance approaches to frontier AI nations.

How middle powers can influence frontier AI via an export-led approach

It is possible to influence how frontier AI is developed, used, and governed without building a domestic frontier AI lab. A middle power can produce and share information, analysis, innovations, services and governance approaches with those actors who are developing frontier AI.

Not every middle power will be able to exert this kind of influence, but many will. What’s important is that, in order to influence AI governance in this way, middle powers don’t necessarily need to build frontier AI systems or have military or economic dominance. Instead, they need some kind of unique expertise or selling point related to developing and governing secure and beneficial advanced AI, which can be recognised internationally. We’ll discuss some examples of this more below.

1. Exporting information and expertise

Decisions about how frontier AI is developed and deployed depend heavily on access to reliable, up-to-date information and research on emerging capabilities and risks. Middle powers can influence these decisions by producing credible, independent analysis that frontier developers and governments can use to inform their thinking.

The UK’s AI Security Institute (AISI) is an excellent demonstration of this model. Its evaluations of model capabilities and risks are shared with both the US Centre for AI Standards and Innovation (US CAISI) and with frontier companies themselves. When UK AISI identifies important capabilities, risks, or limitations of AI models, this feeds directly into development and deployment decisions by other actors.

A related example is the AI Alignment project, a global fund seeking to advance the field of AI alignment that is supported by the UK, Canadian and Australian governments, alongside other industry and philanthropic partners. The project has attracted significant philanthropic support and private funding, including from frontier labs such as Anthropic and OpenAI. This approach gives the UK, Canada and Australia an opportunity to contribute to the frontier of AI alignment science, while simultaneously strengthening their renowned research institutions and ability to attract top technical talent.

Middle power AI safety and security institutions in the International Network for Advanced AI Measurement, Evaluation and Science (formerly the international network of AISIs) have the opportunity to act as more neutral evaluators than entities with direct commercial or geopolitical stakes. Crucially, this kind of informational influence is very different from political influence, in that it enables well-informed decisions without overtly saying what those decisions should be. Reliable information and analysis can materially affect global AI policy in major jurisdictions, as well as developer decisions in frontier AI companies, without triggering the credibility concerns that overt advocacy might.

2. Exporting innovations and services

Middle powers can also develop innovations or services that can be used to support secure and beneficial AI development globally. This can include producing proof-of-concept innovations that can then be adopted elsewhere, often with positive spillover effects on AI governance, as well as providing key services that can be exported, such as frontier AI assurance services.

Consider, for instance, defensive capabilities for monitoring AI-related national security threats. If a middle power develops effective methods for detecting AI-enabled cyber operations or AI-enabled biological threats, these techniques have value for any country facing similar threats - including those developing frontier AI. Demonstrating that something works is often more persuasive than arguing it should be done. Building an industry providing these services can increase security globally, while providing domestic economic benefits.

One example of an exported innovation for AI security is the Inspect framework, developed by the UK AISI as an open-source tool to support frontier model evaluations. This framework has been used to build evaluations by frontier AI developers and evaluation organisations, including Anthropic. Middle power innovations and services can help to develop testing and verification capabilities that can be used by private sector actors to improve oversight and governance of their systems, supporting global risk mitigation.

3. Exporting governance and regulation

Finally, there is scope for some countries to govern the development of new technologies without controlling their production. A clear example is the EU AI Act, which leverages the EU’s large market to govern frontier AI companies who wish to sell products on this market. The EU AI Act is likely to meaningfully influence frontier AI development towards better risk management practices. While implementation remains ongoing and enforcement still uncertain, most frontier developers have already signed up to the AI Act’s Code of Practice for General-Purpose AI, which includes substantive commitments on safety and security practices that align with the regulatory principles in the AI Act. Companies face strong incentives to comply because the EU represents a major market and maintaining different development practices for different jurisdictions would be operationally complex and costly. However, the impact of the EU AI Act will depend upon the effective interpretation, implementation and enforcement, which will require technical capability and sustained political will. Similarly, European data protection law fundamentally changed how global technology companies handle user information, creating practices that extended well beyond EU borders. California’s vehicle emissions standards influenced automotive design worldwide, as manufacturers found it more efficient to meet the strictest standard everywhere rather than maintaining multiple production lines.

It is also possible to export governance and regulation without relying upon the leverage of a large market. Australia’s social media ban for users younger than 16 appears to have triggered pushes for similar regulations in a variety of other countries. Beyond technology policy, New Zealand’s move to setting an inflation target for its central bank in 1989 was widely copied and has now become the dominant approach to monetary policy globally. Where middle powers identify and adopt governance or regulatory innovations related to frontier AI, they may be able to catalyse wider global adoption through a similar mechanism.

Why this matters

AI development is likely to have profound impacts globally. Ensuring a wide range of countries can influence advanced AI development enables those with a stake in the outcomes to help shape the process of development and diffusion. This is not just an opportunity to strengthen one country’s global influence, it is an opportunity to ensure global risks are managed globally and global benefits are shared. AI middle powers have a direct interest in exporting ideas, innovations, services and governance approaches to support these goals. Maintaining this level of expert, innovative and governance capability also ensures that key AI-related capabilities are not purely concentrated within a single state or actor. The strategies outlined here are not only routes to positive influence over frontier AI development, but also potential mechanisms for maintaining the distributed capacity that makes dangerous concentrations of power less likely.

As AI systems become increasingly autonomous, there is a significant risk that they will at some point evade human oversight and control and act in dangerous ways. The capacity for disruption or catastrophe from this is one of the most concerning risks posed by advancing AI capabilities, and perhaps the one society is least well prepared for.

We wrote recently about why governments need to do more to govern loss of control risks, and about a prototype observatory we’re developing to create new techniques to detect loss of control in the real world.

This post expands on a specific intervention that is crucial and concerningly neglected: monitoring real world model behaviours linked to loss of control risk.

Why improved monitoring is crucial for loss of control

The current state of monitoring for loss of control is in its infancy. A recent RAND Europe report found, for example, that detection is currently inconsistent and ineffective, and more work must be done by governments, researchers and AI companies to improve the state of the art.

We set out three key reasons for why monitoring loss of control is essential for the risk to be effectively addressed:

• First, monitoring would improve our evidence base and strategic awareness, enabling better planning and mitigations more generally. Right now, our understanding of loss of control risks relies heavily on theory or evaluations which often depict contrived scenarios. Real-world signals and intelligence would give us a richer, more grounded evidence base on how loss of control risks are emerging. This would both build a clearer picture of the loss of control harms we’re already experiencing, helping to drive policy action, and provide a more grounded evidence base against which to consider potential future more extreme harms.

• Second, better monitoring might make it possible to effectively respond to contain a loss of control risk before it materialises. If early warning signs can be detected enough, there may be a ‘containment window’ in which acting fast enough could prevent significant harm. But this will only be possible with fast, accurate intelligence about the threat as it emerges. To draw on the analogy of a pandemic: by monitoring for new pathogens in wastewater, for example, it’s possible to detect emerging threats early enough to contain them before they spiral into full-blown pandemics.

• Third, robust monitoring capability could also act as a deterrent (aka “deterrence by denial”). One key threat model for loss of control involves “scheming” behaviours where AI systems deliberately evade human-imposed controls. An AI system that knows its resource acquisition and anomalous behaviour will be noticed faces real costs and risks in pursuing those behaviours. Detection is therefore also prevention. Together, these developments suggest a growing risk of autonomous AI systems coming to operate outside of human control in dangerous ways.

There is an urgent need for an all-source intelligence observatory for loss of control risks

Governments, regulators, and civil society currently lack a monitoring capability that can detect and monitor emerging loss of control threats. This capability, which we might call an intelligence “observatory”, will need to centralise and triangulate multiple sources of data and intelligence about the agent’s activity.

Exploiting this would require something like an all-source centralised intelligence capability - one that can collect and analyse signals from multiple sources. Whistleblowers or insiders might surface early warnings about concerning new capabilities. Signals monitoring could flag a power-seeking AI quietly acquiring compute, energy, money, or labour. Triangulating these data streams could make the difference between catching a threat early and missing it entirely.

The table below sets out the different types of intelligence we think this observatory would ideally incorporate.

For example, if a loss of control incident involved a rogue AI system strategically accumulating financial or computational resources, then: OSINT may provide indicators of manipulating markets through misinformation on social media for financial gain; SIGINT may provide indications of the system’s transactions with cloud computing providers; and HUMINT might provide information about model propensities from an AI company’s confidential internal tests. Corroborating these signals could make a big difference to detecting the threat, by revealing the connection between seemingly unrelated events.

Further work is needed to identify the highest priority threats from loss of control and which forms of monitoring could best help mitigate these threats. This is something we’re thinking about and plan to share more about soon.

Defensive acceleration of AI monitoring – or, “intel/acc”

Attention should be paid to where defensive acceleration of intelligence gathering and analysis to monitor AI risks – or “intel/acc” – can be achieved using frontier models, due to AI being adept at analysing large quantities of data extremely quickly. A major moonshot technical project could develop novel techniques that are impossible for human analysts or more conventional technologies. AI-enabled intelligence gathering for AI threats may also be inherently defense-biased, benefiting defensive actors more than an uncontrolled AI agent. At the same time, AI-enabled intelligence gathering and analysis could also pose an increased risk of misuse by governments or malicious actors, meaning that careful consideration of governance and misuse risks is also needed.

This R&D may not attract private funding, and may require classified intelligence, suggesting government action may ultimately be required. Once the highest priority threats are identified, the ideal outcome is likely a project with a budget in the 10s to 100s of millions led by one or more governments who share intelligence on emerging AI threats with the governments, AI companies or other actors best-placed to mitigate these threats. A coalition of governments or a new international monitoring agency could be ideal, but also more difficult to achieve.

OSINT monitoring capabilities are a promising starting point

While detection should draw on as many sources as possible, OSINT seems particularly promising as a starting point.

Firstly, an OSINT capability can be developed entirely outside of government and intelligence communities, as it does not rely on classified intelligence or investigatory powers, meaning significant ground can be broken unilaterally by any sufficiently resourced organisation. This could offer the strategic benefit of building a case for an observatory by demonstrating viable detection methods as a way of stimulating action within governments.

Similarly, OSINT also doesn’t rely on partnerships with AI companies, which may run into legal or commercial obstacles. Unlike whistleblowing, which may involve legal risks, or direct disclosures from AI companies, which may involve commercial risks, OSINT faces relatively few barriers to collection and sharing. This makes it particularly tractable to develop novel proof of concept capabilities that can begin feeding into operational capabilities.

This reasoning motivated CLTR to develop a prototype of one novel OSINT technique: a tool that scrapes X for transcripts of AI model interactions showing scheming or scheming-like behaviours. It is already producing findings on emerging trends in loss of control incidents. We’re also partnering with Arcadia Impact to identify the most promising novel OSINT capabilities that could be developed.

Next steps

CLTR will be continuing to develop this thinking, and is interested in collaborations or input on this work. Specifically:

• The why:

  • Are there flaws or gaps in this theory of change?

• The who:

  • Which institutions should own and operate this kind of capability? Government agencies, independent bodies, AI labs, civil society coalitions?

  • How should incident intelligence be shared between government and non-government stakeholders without creating security risks (e.g. information hazards about vulnerabilities)?
    

• The how:

  • Which methods and information sources are most promising for detecting an emerging incident?

  • How do we calibrate detection to avoid false positives while being sensitive to genuine threats?

  • How might a sophisticated AI system try to evade or manipulate OSINT-based detection? How do we stay ahead of that?

This post was originally published on the Centre for Long-Term Resilience blog in February 2025.

The UK Government is beginning to express significantly more concern about the risk that increasingly autonomous AI systems might at some point evade human oversight and control.

In October 2025, the Director General of the UK’s Security Service raised concerns that there are “potential future risks from non-human, autonomous AI systems which may evade human oversight and control”. This prompted the House of Lords to publish a briefing and hold a short debate on what steps the Government should be taking to address these risks. The AI Security Institute is working on new benchmarks for assessing these risks, undertaking world-leading research on scheming and autonomous replication, and funding research on preparedness for loss of control incidents.

However, the wider apparatus of government is yet to grapple seriously with the question of how to govern loss of control risks.

In this post, we explain why we believe the UK must do more in this space, and make four concrete recommendations which would lay the initial groundwork for effective governance of loss of control:

1. Build a shared understanding of loss of control across the UK Government, industry and the public by adding the risk to the National Risk Register (NRR): Address the UK Government’s fragmented understanding of loss of control with an official definition and assessment in the NRR, including its impact, likelihood and a preparedness assessment.

2. Clarify responsibilities by formally designating DSIT as the Lead Government Department for AI loss of control risks: Increase public transparency and accountability for anticipation, prevention, preparation and response of loss of control, by publicly naming DSIT as the Lead Government Department (LGD) that is responsible for loss of control risks.

3. Create transparency and accountability for risk governance by publishing an AI Security Strategy: The Government should commit to a set of concrete activities over the short, medium and long-term to effectively anticipate, prepare for and defend against loss of control risks

4. Ensure the UK Government can effectively intervene during a loss of control incident by introducing emergency powers: As with other threats to national security, the UK Government will need some powers of direction during an emergency. This will be necessary to ensure it has effective information and an ability to guide fast industry action.

Why governments must do more to address loss of control risks

An uncontrolled AI system could lead to catastrophic scenarios, ranging from disrupting critical national infrastructure to an AI-engineered pandemic and possibly even threatening human extinction.

Three significant developments have emerged in the past 12 months that lend strong support to the concern that autonomous AI systems could soon behave in dangerous ways:

1. AI systems’ ability to operate autonomously is growing exponentially. Modern systems can now plan and execute extended tasks without human control or intervention. One leading AI benchmark measures frontier AI systems’ ability to complete tasks that take human experts hours to finish, across many different domains: software and AI engineering, mathematics, cybersecurity and general reasoning. Right now, state-of-the-art AI models can complete well-scoped software engineering tasks that take human experts more than four hours to complete, with 50% reliability, and this rate is increasing exponentially. Maximum task length currently doubles about every six months, so by the end of 2026, AI systems are expected to be able to complete work that takes professional software engineers two full days completely unsupervised. Even if the tasks in question are of a highly-scoped and bounded nature, the exponential trend demands serious attention.

2. There is emergent evidence of AI misbehaviour. AI evaluators have gathered early evidence that AI systems can—in certain circumstances—pursue unintended goals, including goals that undermine their operators. For example, there is evidence for the capability of some AI models to strategically hide their true motives, cheat to achieve better results or to win at impossible tasks, and resort to blackmail to avoid shutdown. Frontier models already show signs of knowing when they are being tested for dangerous capabilities and, when probed, openly state that they choose to deliberately perform worse on tests to hide their abilities (a strategy known as ‘sandbagging‘). Models also evidence some early ‘scheming‘-like behaviours, where they pretend to align with human values to avoid being shut down or otherwise controlled. These behaviours aren’t conclusive evidence of imminent danger, nor do they prove AI models are inherently dangerous. It is especially important to distinguish between capability and propensity, as highlighted recently by the AI Security Institute: even if AI systems are able to do something, that doesn’t mean they will. However, these evaluations clearly show AI is developing capabilities to bypass human oversight and control in concerning ways, which could lead to catastrophic risks in the future. CLTR is also beginning to observe some of these behaviours in real-world contexts in our Loss of Control Observatory.

3. Voluntary self-governance is proving ineffective at maintaining standards when it counts. AI companies, while developing AI capabilities that are increasingly risky and remain poorly controlled, are racing towards technological prowess, often cutting corners on safety. Anthropic—often noted as the most safety-conscious frontier AI company, providing measured support for regulations and transparency—quietly backpedalled on a previous commitment to define warning signal evaluations for the next generation of its models, before releasing a new generation of model, Claude 4, last year. Google DeepMind activated additional safeguards on Gemini 2.5 Deep Thinking, citing concerns around the development of biological weapons, only to deactivate them for Gemini 3, having raised the risk threshold to implement costly mitigations. This might be a sensible calibration to new information, but it was done quietly, with very little external input and scrutiny. Independent observers have tracked similar shortcomings in other companies. Even more concerningly, xAI’s Grok 4—a frontier model that scores the highest on several benchmarks—readily answers questions related to the development of weapons of mass destruction, and was released without any published safety evaluations.

Together, these developments suggest a growing risk of autonomous AI systems coming to operate outside of human control in dangerous ways.

The UK government lacks a transparent and accountable approach to loss of control

In addition to being extremely high stakes, the novelty and unpredictability of loss of control risks means our existing governance institutions are not well-equipped to address them.

Other jurisdictions – such as the EU and some U.S. states – have begun to set out new regulations to address loss of control.

The EU’s General Purpose AI (GPAI) Code of Practice, for example, names loss of control as one of four systemic risks posed by GPAI that signatories must address. Similarly, California’s SB 53 bill requires reporting of incidents in which frontier AI models engage in “deceptive techniques against the large frontier developer to subvert the controls or monitoring of its large frontier developer”. New York’s RAISE Act introduces similar requirements.

But while the UK AISI is developing world-leading research on loss of control, there is a disconnect between this research agenda and the UK Government’s public policy efforts.

This fact is starkest in the UK’s National Risk Register (NRR). The 187-page NRR—which summarises the Government’s assessment of the most serious risks the UK faces across the next five years—mentions artificial intelligence in passing only three times. There is no mention of loss of control among the 88 risks that are assessed.

While policy work on loss of control is underway, it is “fragmented”, according to a Government-funded report by RAND Europe. It is also opaque and therefore unaccountable, with minimal reference in policy documents, and likely very diverging levels of understanding and urgency across government.

Our recommendations

To effectively prepare for and govern loss of control risks, we suggest that the UK Government needs to immediately prioritise four things:

1. Build a shared understanding of loss of control across the UK Government, industry and the public by adding the risk to the National Risk Register (NRR).

The UK Government’s understanding of the risk of loss of control is fragmented. While scientific reports and some policy documents give high level definitions of the risk, the UK lacks a clear account of the government’s own understanding of the risk, with an assessment of impact and likelihood and a reasonable worst case scenario.

The NRR is the official public account of the UK Government’s assessment of the most serious risks the UK faces. In the absence of regulation similar to the EU CoP or SB 53, the NRR is a prime candidate for building a shared understanding of loss of control among government departments, businesses, CSOs and the wider public.

The NRR could include one or more risk assessments to capture loss of control. The UK Government is committed to considering even “extremely unlikely” scenarios (e.g. lower than 0.2% probability of occurring), especially where they threaten severe impacts. Forecasts and surveys from experts suggest that the risk may be substantially higher than this.

2. Clarify responsibilities by formally designating DSIT as the Lead Government Department for AI loss of control risks

Along with defining and assessing the risk, the UK Government must create more transparency and accountability for its management.

This requires a ‘Lead Government Department’ (LGD). The LGD system is the cornerstone of the UK Government’s risk governance regime. It names the government departments responsible for risk identification, emergency preparedness, and response and recovery for major national risks.

The most recent official, public document naming LGDs that we have been able to access is the UK National Leadership for Risk Identification, Emergency Preparedness, Response and Recovery from 2023. Loss of control risks are not covered in this document, leaving the LGD unspecified.

We suggest that the department with the appropriate remit, expertise and levers for risk governance is DSIT. DSIT would need to work in close collaboration with the Cabinet Office and the wider national security community, but it is the best-placed of all government departments to coordinate anticipation, prevention, preparation, and response.

3. Create transparency and accountability for risk governance by publishing an AI Security Strategy

As LGD we recommend DSIT lead the development of a UK AI Security Strategy (AISS), modelled on the UK Biological Security Strategy (BSS). The AISS should cover the risk understanding, prevention, detection, and response. This will enable outside actors such as the DSIT Select Committee and civil society to hold DSIT accountable to its progress.

The AISS should retain three key components of the BSS when considering how to govern loss of control:

• Implementation: Rather than set out grand ambitions alone, the AISS should commit to a set of activities over short, medium and long term. These activities can be used to hold the government accountable for action and is especially important given the fast rate of AI progress many expect between now and 2030.

• Coordination: The AISS should ensure coordination of risk management activities, possibly via an AI Security Coordination Unit, similar to the Biological Security Coordination Unit. This is essential to coordinate security-relevant activities across Government, given that the most catastrophic loss of control outcomes would be considered ‘whole-of-system’ risks (see The Amber Book: Managing crisis in central government).

• Formalised leadership and governance structures: The BSS assigns clear political and official leadership to its outcomes. For example, it specifies the lead Minister who must report annually to Parliament on progress on implementation of the Strategy, and the Senior Responsible Officer (SRO) who oversees the implementation of the Strategy. The AISS should do likewise: for instance, through the Secretary of State for Science, Innovation and Technology and the Director General for Artificial Intelligence in DSIT.

4. Introduce emergency powers to enable the UK Government to respond to a loss of control incident

Emergency powers are vital for responding to emergencies, during which Secretaries of State may need to direct regulated entities and regulators to respond to an incident in order to contain it. This will be true also for loss of control incidents.

In our report, Preparing for AI security incidents, we established that the UK risks being empty-handed in a crisis, as existing legislation is unlikely to provide the necessary emergency powers in the event of a loss of control incident.

Emergency powers should ensure that the appropriate person, likely the DSIT Secretary of State, has the power when necessary to:

• Compel information: In an emergency, the government may need to quickly access information about the nature of a loss of control incident to inform situational awareness and response decisions. If this information is needed at a very fast pace, legal powers could be vital, as the UK may find itself competing with other governments to secure information.

• Direct prevention, mitigation or response: There may be a need to direct frontier AI companies to take steps to address a risk. This can ensure actions are taken to prevent or respond to an emergency and within a fast enough timeframe to be effective. These types of powers could be conferred to the DSIT Secretary of State to ensure rapid response in an AI security incident.

• Contain or restrict access, distribution or operation (e.g., take down or block services): In certain emergency situations, there may be benefits to temporarily revoking public access to the model in the UK or even enforcing a shut down of certain GPU activity.

The UK’s upcoming AI bill could potentially provide those powers, however the future of that bill is now in doubt. As a result, other legislative vehicles, such as the Cyber Security and Resilience Bill, should be considered for providing appropriate emergency powers.

Once those powers are introduced, relevant actors should participate in tabletop exercises to prepare for how they would use them in various scenarios.

These recommendations represent initial steps the UK Government can take to ensure it is equipped to govern loss of control risks. Much more work will be needed on more granular strategies to better understand, detect, and mitigate loss of control risks. We plan to share more work on this later this year, and would welcome contact from anyone working on similar issues.

Acknowledgements

We are grateful to Jamie Bernardi, who provided early research to support this work during June and July 2025, and Imogen Stead, James Ginns, Eleanor Hevey, and Hamish Hobbs for providing feedback on these recommendations.

Subscribe now

This post was originally published on the Centre for Long-Term Resilience blog in December 2025.

Kanishka Narayan MP, Minister for AI and Online Safety, recently stressed the importance of implementing “an AI policy which can double-down on our strengths in key areas” to make sure “we have a seat at the table” as AI capabilities develop. The Government’s focus so far has been on investing heavily in sovereign UK AI infrastructure in order to maintain geopolitical relevance as the technology advances.

In our view, the UK is also particularly well-positioned to influence global frontier AI governance conversations. As the Network Coordinator for the International Network for Advanced AI Measurement, Evaluation and Science (previously the International Network of AI Safety Institutes), the convenor of the inaugural international AI safety summit and the first country to establish a government institute dedicated to frontier AI research and model testing, the UK continues to demonstrate that international governance expertise should be valued as a core strength of its AI policy. The question now is how the UK can best double down on its leadership role to strengthen and differentiate its global influence.

One potential route to impact could be through the implementation of gold-standard frontier AI legislation. However, despite promises from two successive UK governments to institute targeted AI regulation, nothing concrete has yet materialised. The now-familiar narrative of delay surrounding the proposed frontier AI Bill reared its head yet again earlier this month when Technology Secretary Liz Kendall indicated that the Government is unlikely to bring forward new regulation beyond extending the online safety regime.

For a Government seeking to shift the narrative on UK AI to one of international ambition and confidence, we think this is an opportunity missed. As we have set out in more detail in this post, failing to regulate not only leaves the UK insufficiently equipped to address extreme risks from AI domestically, but is also a missed opportunity to cement the UK’s leadership in global AI governance.

However, an AI Bill is also far from the only route to securing the UK’s “seat at the table”. This post explores new pathways for advancing the UK’s international impact beyond simply relying on either regulatory influence or investment power, and sets out a range of options for how this agenda can be taken forward without the need for legislative intervention.

The UK’s “AI soft power”

The foundation of the UK’s outsized potential to influence global AI governance conversations relies on three strategic advantages, which together grant the UK significant AI “soft power” in its international relations:

• Strong convening capability: The UK’s global reach enables it to demonstrate notable convening power on the international stage. The 2023 inaugural AI Safety Summit brought representatives from the two great powers of AI – the US and China – together for the first time to discuss AI safety and security research. The resulting declaration committing to closer international cooperation on advanced AI has led to the establishment of the international AI safety report and the international AISI network, which the UK now leads as “Network Coordinator”.

• Close UK-US relations: The UK has succeeded in maintaining a strong relationship with the US despite changes in administration in both countries. The technology-focused Memorandum of Understanding signed in September with the US recognised that the two countries are “most trusted security and defense partners” and committed to advancing “a shared mission to promote secure AI innovation”, including through joint testing and standards development through the AISI – CAISI partnership.

• Best-in-class state AI technical capabilities: In little over two years since its establishment, the UK’s AI Security Institute has become a world-leading example of what public sector capabilities in frontier AI safety and security research and model testing can look like. AISI’s stable funding, research credentials and technical talent have allowed it to develop trusted voluntary partnerships with AI developers, which in turn compounds the UK’s international credibility and ability to create influence on the international AI stage.

Pathways for advancing the UK’s international impact

Looking beyond the usual avenues of frontier AI investment and regulation, we see three particularly promising (but not exhaustive) new pathways through which the UK can cement its position as a leading voice in global AI governance. For each pathway, we suggest examples of “low-hanging fruit”, which are ideas that the UK could implement now to advance this agenda, and “higher-ambition interventions”, which are suggestions for where the Government should begin to direct its thinking in order to maximise its long-term impact on the international stage.

Pathway 1: Strengthening AISI’s international engagement

The UK AISI has set the gold standard globally of governmental AI research and testing institutes. In its next phase, AISI should further systematise ways to diffuse its knowledge internationally. The recent announcement that the UK is stepping into the role of “Network Coordinator” of the newly renamed “International Network for Advanced AI Measurement Evaluation and Science” – formerly the International AISI Network – presents an opportunity for the UK to lead the reinvigoration of this international coordination mechanism.

• Low-hanging fruit: leverage the UK’s “Network Coordinator” role to establish strong communications channels and regular meeting structures for the network, in order to facilitate closer collaboration and information-sharing. The UK should also use its leadership position as a platform to raise the salience of emerging extreme AI risks identified by the network in other international political fora, acting as an early warning mechanism to keep governments informed of evolving AI risks and potential mitigations.

• Higher-ambition intervention: institute a separate, private information-sharing partnership between AISI and the US CAISI, building on the aims of the Memorandum of Understanding. This would act as an early warning mechanism for sharing national security-relevant risks that emerge from testing conducted by UK AISI on US AI models, to increase joint strategic awareness of emerging AI model capabilities and facilitate appropriate US responses.

Pathway 2: Advancing international alignment on risks and mitigations

The UK should continue to leverage its international convening power and relationship with the US to build mutual understanding on the evolving frontier AI risk landscape and coordinate mitigating actions where necessary. This could include proposing and building support for baseline international agreements on key frontier AI issues, for instance defining red lines for governments’ use of frontier AI capabilities, at international fora such as the UN, NATO and Five Eyes.

• Low-hanging fruit: champion best-practice regulation and research in other jurisdictions, such as the Safety & Security chapter of the EU AI Act’s GPAI Code of Practice and the Singapore Consensus on Global AI Safety Research Priorities, to inform forthcoming US AI policy interventions and facilitate high-level multilateral alignment on a baseline set of mitigations against the most prominent security risks from frontier AI.

• Higher-ambition intervention: work with AISI’s research and strategic awareness teams to identify specific extreme risks that will require closer international coordination via the establishment of dedicated global governance institutions, and secure international buy-in from key partners such as the US to create new cooperation mechanisms as necessary. One example could be a new institution dedicated to international alignment on serious AI incident monitoring and reporting frameworks.

Pathway 3: Influencing frontier AI companies’ behaviour

The technical expertise of AISI and the UK’s convening power combined make it comparatively well-placed to work with AI developers to align incentives towards effective frontier AI risk mitigation. The UK should use these relationships to strengthen its industry partnerships further and push for new voluntary commitments to raise the floor on what constitutes “best practice” for increasing the safety and security of AI systems.

• Low-hanging fruit: build on AISI’s trusted relationships with frontier AI companies to incentivise greater transparency and information-sharing proportionate to the speed at which frontier model capabilities are developing. The UK should coordinate with the companies privately to secure buy-in for extending the scope of voluntary international commitments on frontier AI risk transparency and management.

• Higher-ambition intervention: identify areas of extreme AI risk that are not adequately covered by existing regulatory regimes globally – such as the risks stemming from the race between frontier AI companies to achieve significant recursive self-improvement capabilities – and design a voluntary code of practice to help influence responsible industry practices.

This post represents the start of our thinking on this topic, not the end. As we take this work forward, we’re keen to hear from anyone working in or thinking about similar issues to help us drill down into these recommendations and find the most impactful UK policy interventions for advancing global AI governance.

Subscribe now