Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

China is quietly developing one of the world’s most sophisticated infrastructures for agentic AI governance. This article outlines what China’s agentic AI policy position is, how this could evolve into a framework of agentic AI regulations and standards, and what enterprise AI governance practitioners can learn from these developments.

Disclaimer: this article is not intended as legal advice and should not be read, interpreted, or relied on as such. It is for educational and informational purposes only.

On 8 May 2026, China published one of the world’s first agentic AI governance frameworks: the Implementation Opinions on the Standardized Application and Innovative Development of Intelligent Agents (henceforth ‘Implementation Opinions on Agents’).

Although not legally binding, the Implementation Opinions on Agents, jointly released by the Cyberspace Administration of China (CAC), the National Development and Reform Commission (NDRC), and the Ministry of Industry and Information Technology (MIIT), provides a detailed and influential policy position on agentic AI governance. It reveals how China’s AI regulatory and standards framework will evolve to address agentic AI, whilst also providing useful guidance and inspiration for AI governance practitioners and enterprises in China and further afield.

Credit: this article would not have been possible without the English language translation of the ‘Implementation Opinions on Agents’, published by Geopolitechs on 8 May 2026. Read the original article here.

My book, Fundamentals of AI Governance, will be released soon. It features a comprehensive, visual guide to China AI Law and Policy and agentic AI governance. Pre-order today to secure a 25% discount.

Pre-order my book

The key thing to understand about AI governance in China is that innovation and regulation are not viewed as opposing forces. The government has long championed the importance of regulation and standards to promote responsible and trustworthy AI. Although the Chinese government often has different fundamental AI policy objectives to jurisdictions like the EU, there is consensus between these regulatory superpowers on the notion that innovation and governance go hand in hand.

For organisations operating in mainland China, there exists a comprehensive suite of AI-specific regulations and standards that impose mandatory requirements and shape best practice. These requirements have become embedded into everyday business practices. This has happened in parallel to notable technological progress in China’s AI ecosystem. The Stanford AI Index Report 2026, for example, argued that the performance gap between Chinese and U.S. AI models “has effectively closed”.

China’s AI regulatory framework

China’s AI regulatory framework centres on the following binding instruments:

• Provisional Measures on the Administration of Human-Like Interactive AI Services (‘Human-Like Interactive AI Measures’)

  • Purpose: governs AI services that are designed to interact with individuals and simulate human traits, with a focus on AI companionship and emotional dependence.

  • Effective date: 15 July 2026

• Measures for Labelling of AI-Generated Synthetic Content (‘AI-Generated Content Labelling Measures’)

  • Purpose: mandates the use of ‘implicit’ and ‘explicit’ labels to promote transparency regarding AI-generated content (e.g., deepfakes).

  • Effective date: 1 September 2025

• Interim Measures for the Management of Generative AI Services (‘Generative AI Services Measures’)

  • Purpose: imposes stringent requirements, including model evaluations and registration, for the development and release of public-facing generative AI services.

  • Effective date: 15 August 2023

• Provisions on the Administration of Deep Synthesis Internet Information Services (‘Deep Synthesis Provisions’)

  • Purpose: governs the development and use of AI services to create and share synthetic content (e.g., image, audio, video, and text).

  • Effective date: 10 January 2023

• Provisions on the Management of Algorithmic Recommendations in Internet Information Services (‘Algorithmic Recommendation Provisions’)

  • Purpose: regulates providers of online services that use AI to recommend, distribute, and share content to users in a targeted way, with a focus on social media, online news, and e-commerce platforms.

  • Effective date: 1 March 2022

These instruments are ‘departmental regulations’ rather than national laws. The foundational national laws that are relevant for AI include:

• Cybersecurity Law (updated in October 2025 to incorporate AI-related provisions)

• Copyright Law

• Data Security Law

• Personal Information Protection Law

• Law on the Progress of Science and Technology

The prospect of a national AI law has been actively discussed in Chinese policy circles for several years. Such a law could merge and integrate the core requirements from the AI-specific regulations listed above, whilst imposing a horizontal and comprehensive framework for AI governance (akin to other national laws). The government has previously suggested such a law is on the cards. In 2026, the State Council’s legislative plan stated China will “accelerate comprehensive legislation” on AI. To date, no concrete plans have been announced and no official draft AI law has been published.

From traditional ML and generative AI to the agentic frontier

Traditional machine learning (ML), generative AI, and agentic AI represent the three core waves for enterprise AI. Analysis of China’s AI regulatory and policy framework highlights the way it has been at the forefront of anticipating and responding to this evolution of AI technology development. The widespread societal impact of traditional ML (i.e., AI recommending content) in shaping the information individuals consume, the way they are profiled and targeted, and how this can influence behaviour at scale was addressed with the Algorithmic Recommendation Provisions in 2022. Subsequent regulations, including the Generative AI Services Measures and the AI-Generated Content Labelling Measures, largely focused on generative AI (i.e., AI producing content). These generative AI-era regulations are designed to ensure that prohibited information sources are not used to train public-facing AI models or generated as output by those models, and that the public is not misled or deceived by AI-generated content.

China has been the most active jurisdiction worldwide in terms of the number of different AI-specific regulations it has enacted. A common thread is that these regulations focus on governing AI technologies and services that are available to the public, as well as AI-generated content that the public can access and share. Several of these regulations require algorithms and models that can impact public opinion or mobilise public action to be registered with the authorities. Although the government’s primary focus has been the impact of AI on the information ecosystem, it is now increasingly working on agentic AI security and governance.

How will China govern agentic AI?

Since 2025, there has been increasing concern about the security and safety risks of agentic AI and how these can be managed. This concern ramped up with the release of OpenClaw, which was met with a flurry of official responses and warnings. A comprehensive report on agentic AI security was released by TC260, China’s cybersecurity standards committee, in March 2026. It highlighted 11 agentic AI threats and countermeasures (summarised here by Concordia AI), from agent hijacking to tool abuse. Most of these threats are equivalent or similar to those highlighted by OWASP in its Top 10 for Agentic Applications for 2026 framework.

In recent months, the focus on agentic AI governance has gained further momentum. The most significant development was in May 2026, when three central government departments jointly released the ‘Implementation Opinions on Agents’. Although not legally binding, the Opinions provide a detailed official position on China’s agentic AI governance and policy ambitions. They reveal how China’s AI regulatory framework will evolve to address agentic AI, emphasising the importance of technical standards, open-source frameworks, risk-based classification, and industry self-governance, whilst also providing useful inspiration for enterprise AI governance practitioners grappling with agentic AI, in China and beyond.

The Opinions advocate for developing decision authority rules to codify what agents can and cannot do, what type of human oversight is required, and how human-on-the-loop approaches can be supported with continuous monitoring and anomaly detection. The fundamental principles underpinning this governance approach are safety, controllability, reliability, and trustworthiness. These fundamental principles should be operationalised into baseline requirements that organisations adhere to across the agentic AI lifecycle.

The Opinions outline the contours of an agentic AI governance framework, whilst also championing the importance of agentic AI for economic development and highlighting priority use cases for agents across all major sectors. This follows naturally from the government’s ‘AI+ Plan’, announced in 2025, which imbued a sense of urgency by setting ambitious targets for widespread diffusion of AI systems and agents across Chinese society. The policy vision is for agents to become embedded in all parts of the economy, which includes embodiment in physical AI applications.

Here is a snapshot of five of the most interesting agentic AI use cases encouraged in the Opinions. Although there is no guarantee these use cases will be directly supported by the government, their inclusion highlights that officials think there are promising opportunities to explore:

• Financial services: credit card fraud interception and anti-money laundering agents.

• Commercial services: embodied agents for cleaning, warehousing, and distribution to improve operational efficiency (e.g., in restaurants, hospitality, retail, and logistics).

• Education: agents for personalised learning, intelligent tutoring, and virtual teaching assistance.

• Public safety: embodied agents in disaster rescue and hazardous materials handling.

• Information services: agents for online content governance, including recommendation distribution and intelligent moderation, as well as emotional counselling.

The Opinions recommend comprehensive risk mitigation strategies whilst also promoting priority agent use cases and innovation opportunities. This duality is distinctive to the Chinese approach. It illustrates, through the prism of agentic AI, that innovation and governance are not deemed mutually exclusive. Perhaps the sharpest example of this is the recommendation to explore the use of agents for “emotional counselling”, which was issued just weeks before the Human-Like Interactive AI Measures (which are designed to address the risks of such use cases) became effective.

Agentic AI standards

The Opinions state that agentic AI development shall be “standardized and orderly”. Establishing a framework of agentic AI standards is positioned as a priority action area. To turn this vision into a reality, a host of agentic AI standards are currently under development. This is consistent with the approach taken to generative AI governance. There exists a comprehensive package of mandatory standards that facilitate compliance with the Generative AI Services Measures and AI-Generated Content Labelling Measures. These are highly detailed and prescriptive documents for organisations to follow, and a core part of the AI regulatory framework.

Concordia AI, a social enterprise that works on AI governance in China, notes that ‘agent safety’ is a priority theme for AI standardisation in China. Its 2026 report on the ‘State of AI Safety in China’ highlights seven agentic AI standards under development, as well as an additional seven-part series:

• Basic Specification for Agent Safety (drafting announced January 2026)

• General Security Requirements for AI Agent Applications (drafting announced April 2026)

• General Technical Requirements for Agent Safety (drafting announced July 2025)

• AI Agent Safety Protection Guide (drafting announced July 2025)

• Agent Collaboration Safety Framework and Safety Requirements (drafting announced July 2025)

• Technical Requirements for Agent Development Platform Safety (drafting announced November 2025)

• Model Context Protocol (MCP) Application Safety Requirements (drafting announced September 2025)

In June 2026, the National Standardization Administration of China (SAC) published seven guiding technical documents under the ‘AI – Agent Interconnection’ series. These documents, part of the evolving standards framework, address frontier issues including agent identity, multi-agent interaction, and tool invocation.

This means fourteen different agentic AI governance standards and technical documents are under development or published. Several of these could become mandatory standards in future.

Three key takeaways for AI governance practitioners

For AI governance professionals building agentic AI frameworks, tracking these developments is prudent. Irrespective of whether compliance with China’s AI regulatory framework is of direct relevance, the agentic AI standards referenced above represent some of the most advanced policy and governance work on agentic AI risk management worldwide. The AI governance frameworks and regulations that most organisations align to, including the NIST AI RMF, ISO/IEC 42001, and the EU AI Act, do not cover agent-specific considerations in any detail. Therefore, organisations have little choice but to consult wider sources to inform their approach.

Here are the top three takeaways for practitioners grappling with agentic AI governance:

1. Establish decision authority rules

The Implementation Opinions on Agents emphasise the importance of clarifying “decision-making authority”, including boundaries and required permissions, for actions executed by agents. A high-level, three-tier framework is proposed:

• Decisions reserved exclusively for users (i.e., humans)

• Decisions requiring user authorisation

• Autonomous agent decisions

As agentic AI capabilities improve rapidly, each enterprise should agree and codify internal decision governance and authority rules for agents, covering their highest priority domains. This includes defining where agents can act autonomously, where human review and approval is essential, what the thresholds for human intervention should be, and everything in between. Without such a framework, the technical controls and guardrails that bound and restrict agent action will lack a coherent policy foundation. It’s worth noting that the Chinese framework proposes that users should retain “ultimate decision-making authority over autonomous agent actions”. Human oversight is thus integral to agent governance.

2. Prioritise agent identity and traceability

Agent identity and traceability are core themes of China’s emerging framework of agentic AI standards and technical documents. These considerations are particularly pertinent for multi-agent systems, where agent behaviour, action, and impact are shaped by the myriad ways in which different agents interact, communicate, and coordinate. Oversight and governance of such systems is virtually impossible without each agent having a unique identity, with inputs/outputs and actions logged in a manner that is attributable to each agent. This is a prerequisite for most key aspects of agent governance, including operationalising human oversight and decision authority.

3. Take a risk-based approach to agent governance

Finally, the Opinions signal that the precise governance approach, including whether mandatory requirements will be imposed via new regulations, will depend on the level of risk. Sensitive sectors will likely be subject to new requirements, whereas ‘self-governance’ (e.g., compliance self-assessment and internal evaluations) will be expected in lower-risk domains.

Agentic AI presents distinct risks, but the fundamental approach to AI governance need not change. Not all agents and agentic systems can or should be governed in the same way. Controls and risk management measures should be targeted and proportionate relative to risk. This means organisations should update risk classification and tiering frameworks to adequately incorporate agentic AI use cases.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

This article is Part 2 of a two-part series that explains the ways in which the EU AI Act will be amended. Last week, following months of trilogue negotiations, the European Parliament voted to approve the regulation to amend the EU AI Act. This series analyses the ten most significant amendments that enterprises need to prepare for.

Part 1, published last week, covered the timeline changes and postponements, the prohibited AI expansion, the softening of the AI literacy obligation, and the registration obligation. Part 2 covers the processing of sensitive personal data for bias mitigation, the expansion in scope of the AI Office’s regulatory powers, the extension of proportionality to small mid-cap (SMC) enterprises, and the changes for AI systems in scope of the Machinery Regulation and other EU product safety laws. This article also features a free, high-res EU AI Act Amendments Regulation Cheat Sheet, which you can download via the link below.

It is worth repeating that although the new amending regulation has not yet been formally adopted and entered into force, enterprises should already be reviewing and updating their AI governance and compliance approaches, as the three co-legislating EU institutions have reached a consensus on the substance of the amendments. Nonetheless, the core purpose, logic, and structure of the EU AI Act remain mostly intact.

My book, Fundamentals of AI Governance, will be released in September 2026. It features a comprehensive, visual guide to the EU AI Act, which will incorporate all of the amendments which are formally adopted. Pre-order today to secure a 25% discount.

Get the discount

For each of the ten most significant amendments, this series explains what the law says today, how this is changing, and the practical implications for enterprises. Items 1-5 were covered in Part 1 and items 6-10 are covered in Part 2 (below).

The 10 most significant EU AI Act amendments

1. Expanding the list of prohibited AI practices to ban “nudifier apps”.

2. Timeline changes for high-risk AI system compliance.

3. Timeline changes for transparency-requiring AI system compliance.

4. Limiting registration in the EU public database for the “exempted” AI systems.

5. Softening of the AI literacy obligation.

6. Processing sensitive personal data for bias detection and correction.

7. Expanding the scope of the European AI Office’s regulatory powers.

8. Proportionality for small mid-cap (SMC) enterprises.

9. Addressing potential duplication for high-risk AI systems which are regulated by EU product safety laws listed in Annex I.

10. Addressing potential duplication for high-risk AI systems which are regulated by the EU Machinery Regulation.

Disclaimer: this article is not legal advice and should not be used, relied on, or interpreted as such. Always consult a qualified legal professional. Furthermore, this article does not cover all amendments outlined in the amending regulation. Finally, these amendments have not yet been enacted as law and are not in force.

Download the pdf version (link below) for the highest resolution version :)

EU AI Act Amendments Regulation Cheat Sheet (by Oliver Patel)

281KB ∙ PDF file

Download

Download the high-res PDF cheat sheet for free. You can use this for any purpose, including commercially, but please provide appropriate attribution.

Download

6. Processing sensitive personal data for bias detection and correction

What is in law today?

The GDPR stipulates that the processing of ‘special categories’ of personal data (henceforth sensitive personal data) is prohibited, apart from in limited circumstances. Sensitive personal data includes:

• Data revealing racial or ethnic origin

• Data revealing political opinions

• Data revealing religious or philosophical beliefs

• Data revealing trade union membership

• Genetic data

• Biometric data (for the purpose of uniquely identifying a natural person)

• Health data

• Sex life or sexual orientation

Article 9(2) of the GDPR outlines ten exceptions to this prohibition. These exceptions represent distinct scenarios where sensitive personal data can be processed. This includes, but is not limited to, when explicit consent has been obtained, when doing so is necessary to “protect the vital interests of the data subject” where they are “physically or legally incapable of giving consent” (e.g., in a medical emergency), or when the processing is necessary and proportionate for a “substantial public interest”.

The EU AI Act augments this by providing an additional legal basis for the processing of sensitive personal data. It does so because providers of high-risk AI systems are required to “detect, prevent, and mitigate possible biases” that are likely to affect the “health and safety” of individuals, have a “negative impact on fundamental rights”, or lead to unlawful discrimination. To enable this, Article 10(5) of the EU AI Act stipulates that providers of high-risk AI systems can process sensitive personal data for the purpose of “ensuring bias detection and correction” in relation to their high-risk AI systems.

However, it must be “strictly necessary” to use the sensitive personal data to achieve this purpose, and the bias detection and correction cannot be effectively achieved by other means. The EU AI Act outlines a range of additional mandatory safeguards which must be applied alongside all the safeguards already required by the GDPR.

How is this changing?

There are two key changes. Firstly, providers and deployers of AI systems and AI models which are not high-risk will also be permitted to process sensitive personal data when it is “strictly necessary” to ensure “bias detection and correction”. When doing so, they must apply the same safeguards referenced above. Secondly, deployers of high-risk AI systems will also be permitted to process sensitive personal data, under the same conditions as providers.

The substantive changes are therefore the expansion in scope of which organisations can process sensitive personal data for bias detection and correction (i.e., providers and deployers), as well as in relation to which AI archetypes (i.e., high-risk AI systems, as well as AI systems which are not high-risk and AI models). However, what is not changing are the types of biases this processing must be targeted at addressing, the “strictly necessary” condition, and the safeguards that must be applied for the processing to be lawful.

Enterprise implications

These changes are useful for enterprises which develop and deploy AI systems which are not high-risk, but which nonetheless support decisions or actions that directly impact individuals or groups and can therefore result in bias. They are also useful for deployers using high-risk AI systems, in domains like recruitment, education, and healthcare. However, the strict conditions and safeguards stipulated in both the EU AI Act and the GDPR ensure that there will continue to be a relatively high bar for when such processing (of sensitive personal data) is lawful. The notion of strict necessity, for example, is well established in EU case law, and it means that organisations must be prepared to demonstrate that they effectively had no other option but to process the data in order to mitigate the risk of bias.

7. Expanding the scope of the European AI Office’s regulatory powers

What is in law today?

The EU AI Act has established a decentralised governance and enforcement regime for AI systems. The European AI Office, part of the European Commission, is responsible for overseeing and enforcing the provisions relating to general-purpose AI (GPAI) models (e.g., the frontier models released by the likes of OpenAI and Anthropic). The national market surveillance authorities (i.e., member state regulators) are responsible for overseeing and enforcing the provisions relating to AI systems (e.g., high-risk and transparency-requiring AI systems), as well as most other EU AI Act provisions.

Article 75(1) provides the AI Office with powers to monitor and supervise AI systems based on GPAI models. This only applies when the AI model and AI system are developed by the same provider. For example, ChatGPT is an AI system which is powered by AI models like GPT-5.5. OpenAI is the provider of both the AI model and the AI system. However, this regulatory power is not exclusively held by the AI Office. Member state regulators can also oversee and enforce EU AI Act provisions relating to these AI systems. This creates potential situations of overlapping competence and providers (like OpenAI) being investigated and enforced against by multiple regulators simultaneously.

How is this changing?

Considering that there are, in many cases, multiple designated regulators per member state, as well as an increasingly large number of AI systems based on GPAI models, governance and enforcement of the EU AI Act has the potential to get messy.

Confronting this challenge head on, the EU has opted to simplify the governance regime by granting the AI Office with “exclusive competence” to oversee and govern AI systems based on GPAI models, where both the AI system and the GPAI model are developed by the same provider (or by providers that are part of the same organisation or corporate group). This also applies where the deployer of the AI system is also the provider of that system (as well as the GPAI model(s) on which it is based).

However, there are several exceptions to the AI Office’s exclusive competence. These are:

• AI systems developed and used by EU institutions, agencies, and bodies.

• AI systems which are safety components in the management and operation of critical infrastructure.

• AI systems intended to be used in the administration of justice.

• AI systems provided by law enforcement authorities, border management authorities, and (certain) financial institutions.

• AI systems which are products, or safety components of products, covered by an EU product safety law listed in Annex I of the EU AI Act.

In these scenarios, the AI Office will not have exclusive competence, and member state regulators will continue to have an important role to play.

Enterprise implications

For organisations that develop GPAI models and then integrate those models into AI systems which they both provide to other organisations and use themselves—like OpenAI and Anthropic—their interactions with EU regulators should become more streamlined. This is because the AI Office will serve as their key point of contact (e.g., for reporting serious incidents), and there will be a reduced likelihood of multiple regulators simultaneously investigating them or pursuing enforcement action in relation to the same issue.

However, these changes do not affect organisations that are merely deployers of AI systems which they did not develop—even if those AI systems and their GPAI models were developed by the same provider. Furthermore, certain AI systems may always be governed by both the AI Office and member state regulators, no matter their technical architecture and model composition. Therefore, these changes only directly affect a small number of organisations and are not particularly relevant for any enterprise which doesn’t develop GPAI models.

8. Proportionality for small mid-cap (SMC) enterprises

What is in law today?

The EU AI Act provides targeted flexibility and proportionality for micro, small, and medium-size enterprises (SMEs), including start-ups. For example, the compliance penalties which SMEs can be issued are capped as follows:

• Prohibited AI breaches: up to €35 million or 7% of total worldwide annual turnover for the preceding financial year (whichever is lower).

• Most other provisions: up to €15 million or 3% of total worldwide annual turnover for the preceding financial year (whichever is lower).

• Supplying incorrect, incomplete, or misleading information to regulators: up to €7.5 million or 1% of total worldwide annual turnover for the preceding financial year (whichever is lower).

This is the inverse to the “whichever is higher” penalty logic that applies for all other businesses (i.e., those which are not SMEs). This means, for example, that even if 7% of an SME’s total annual turnover far exceeds €35 million, it would not pay more than this larger amount if penalised for engaging in a prohibited AI practice. However, an organisation which is not an SME could pay a substantially higher penalty for an equivalent breach, perhaps in the hundreds of millions or billions of euros.

How is this changing?

The most significant change is to extend the proportionate, inverse penalty regime for SMEs to small mid-caps (SMCs) as well. This would significantly reduce the total potential penalty exposure of SMCs in certain circumstances, meaning that many more companies benefit from EU AI Act proportionality.

SMCs that are providers of high-risk AI systems would also be able to provide the required technical documentation in a simplified manner. Again, this is an extension of a proportionality mechanism already afforded to SMEs.

The other key change is to include the formal legal definitions of SME and SMC in the amending regulation. These are:

• SME: an enterprise that employs fewer than 250 people and which has an annual turnover not exceeding €50 million, and/or an annual balance sheet total not exceeding €43 million.

• SMC: an enterprise, which is not an SME, that employs fewer than 750 people and which has an annual turnover not exceeding €150 million or an annual balance sheet total not exceeding €129 million.

Enterprise implications

The influence of the Draghi report can be detected in these amendments. The “Draghi report” on The Future of European Competitiveness, published in 2024, argued that the EU’s complex digital regulatory environment impedes innovation and growth. It positioned “reducing the regulatory burden” as a core priority for transforming the EU’s economic prospects. By significantly lowering the potential regulatory penalties which both start-ups and scale-ups could face, the EU will be hoping to encourage a broader range of firms to take a proactive, innovative, and ambitious approach to AI adoption, to the benefit of European growth and competitiveness.

9. Addressing potential duplication for high-risk AI systems which are regulated by EU product safety laws listed in Annex I

What is in law today?

Broadly speaking, there are two categories of high-risk AI system under the EU AI Act:

• AI systems listed in Annex III; and

• AI systems which are products, or safety components of products, regulated by an EU product safety law listed in Annex I.

  • For these AI systems to be classified as high-risk, there must be an existing requirement (in the applicable product safety law) for the product, or the safety component of the product, to undergo a third-party conformity assessment.

For these ‘Annex I AI systems’, the most important consideration is the EU product safety law under which it is already regulated, and how this interacts with any new EU AI Act requirements. Annex I is split into two sections: Section A and Section B. Each section lists EU product safety laws. For AI systems regulated by an EU product safety law listed in Section A, the requirements of the EU AI Act directly apply, in addition to all existing requirements from the other law. However, for high-risk AI systems regulated by an EU product safety law listed in Section B, most EU AI Act requirements do not directly apply. Instead, the European Commission is supposed to “take into account” the EU AI Act’s high-risk AI system requirements when adopting future implementing acts that concern AI systems that are safety components (or products themselves) in these domains.

Section A includes laws that regulate machinery, medical devices, and radio equipment. Section B includes laws that regulate aviation, vehicles, and marine equipment.

How is this changing?

The main concern has been potential overlap between the requirements stipulated in the existing product safety laws and the EU AI Act. Although there were already some mechanisms to address this (e.g., not requiring multiple conformity assessments), these are being expanded and strengthened.

Specifically, to reduce “compliance burdens” and address potential overlap, for high-risk AI systems listed in Annex I (Section A), the Commission will be able to limit the application of certain substantive high-risk AI system requirements, so that providers need not comply with them where an equivalent or higher safeguard already exists (in the applicable EU product safety law).

The Commission will do so by adopting delegated acts which outline which AI systems, product safety laws, and domains are impacted. It must do this by 2 August 2027. However, it will only be able to restrict the application and mandatory nature of specific requirements when the following two conditions apply:

• the EU product safety law (listed in Annex I (Section A)) has requirements or obligations that provide an “equivalent or higher level of protection of health, safety or fundamental rights as the requirement or obligation concerned"; and

• limiting the requirements does not “reduce the overall level of protection” which the EU AI Act provides.

Simply put, the Commission will have the power to determine that certain EU AI Act requirements do not have to be adhered to by providers of certain AI systems. However, it can only do so where the applicable product safety law already delivers an equivalent or higher level of protection for the specific requirement(s) in question, and where lifting that requirement does not weaken the overall level of protection the EU AI Act provides.

Enterprise implications

We do not yet know the precise significance of this change, as we do not know which high-risk AI systems, EU AI Act requirements, and EU product safety laws will be impacted—aside from AI systems regulated by the EU Machinery Regulation, which is being moved from Section A to Section B (see below). However, this could have a major impact on certain organisations, especially if they are providers of high-risk AI systems which the Commission deems to be covered by an existing product safety law which overlaps with EU AI Act requirements in a meaningful way.

Given the perceived and actual reduction in compliance workload, there will likely be intense industry lobbying over the coming months to persuade the Commission to relax certain EU AI Act restrictions, for certain sectors and AI systems, on this basis. However, this change is less significant than earlier proposals to move much of the Section A list into Section B, which would have drastically reduced the overall scope of the EU AI Act’s requirements across many industries. Finally, this change does not impact obligations for deployers of high-risk AI systems covered by Annex I (Section A).

10. Addressing potential duplication for high-risk AI systems which are regulated by the EU Machinery Regulation

What is in law today?

As explained above, the EU Machinery Regulation (Regulation (EU) 2023/1230) is currently listed in Section A of Annex I. This means that, today, high-risk AI systems which are products, or safety components of products, regulated by the Machinery Regulation are directly subject to the EU AI Act's high-risk AI system requirements, in addition to all requirements under the Machinery Regulation itself.

How is this changing?

The amending regulation moves the Machinery Regulation out of Section A and into Section B of Annex I. This is the only change to the structure and content of Annex I itself. Once moved to Section B, most substantive EU AI Act requirements for high-risk AI systems will no longer directly apply to any providers of AI systems which are regulated by the Machinery Regulation. Only a limited set of provisions will apply, none of which are particularly material or noteworthy, for providers, from a compliance perspective.

Enterprise implications

This is a meaningful simplification for providers of AI-enabled machinery, who will increasingly look to the Machinery Regulation, rather than the EU AI Act, as the primary source of their high-risk AI obligations and requirements. However, as per the amending regulation itself, this is a change in regulatory and compliance architecture to promote simplicity and avoid duplication, not deregulation. Indeed, it is important to note that AI systems covered by Annex I (Section B) are still classified as high-risk AI systems under the EU AI Act—and this includes AI-enabled machinery. The level of protection is intended to remain consistent. Providers should track how the Machinery Regulation, and associated artefacts, evolve to absorb these requirements, as that is where the key detail will now sit.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

On Tuesday 16 June, the European Parliament voted in support of the regulation to amend the EU AI Act. The vote was decisive, with 423 MEPs in favour, 57 against, and 174 abstentions. The Parliament’s approval follows the provisional agreement, announced on 7 May, reached between the Council (EU member states) and the lead MEP negotiators. We now know (almost) exactly how the EU AI Act will be amended (unless there is a political earthquake). Although the new regulation has not yet been formally adopted and entered into force, it is a good time for enterprises to review their AI governance and compliance approaches, in light of the imminent changes.

This two-part series provides a comprehensive summary and explanation of the ten most significant EU AI Act amendments, drawing directly from the regulatory text recently approved by MEPs. The goal is to enable enterprise practitioners to prepare for and understand these changes. Part 2 (coming soon) will also feature a free EU AI Act Amendments Cheat Sheet visual.

Background: In November 2025, the European Commission proposed various amendments to the EU AI Act, as part of its Digital Omnibus package. The stated purpose of this was to simplify and streamline key elements of the law, to reduce compliance burdens, support businesses, and boost European competitiveness. The trilogue negotiations that occurred over the past few months have resulted in the text of the regulation we can read and analyse today, which is now endorsed by all three EU institutions. Although the timeline changes and postponements are headline-grabbing, the core purpose, logic, and structure of the EU AI Act remain mostly intact. The EU is not walking away from comprehensive and stringent AI regulation.

The text still requires formal adoption and publication in the Official Journal of the EU before it enters into force. The substance, however, is now settled. Given the dependency on getting this done before the 2 August 2026 cut-off date, we can expect the EU to move quickly.

I will be presenting on Agentic AI Governance and enterprise implementation at the Responsible AI Summit North America next week (on 23 and 24 June). Check out this page for more info and reach out if you will be there!

My book, Fundamentals of AI Governance, will be released in September. It features a comprehensive, visual guide to the EU AI Act, which will incorporate all of the amendments which are formally adopted. Pre-order today to secure a 25% discount.

Pre-order my book

For each of the ten most significant amendments I explain what the law says today, how this is changing, and the practical implications for enterprises. Items 1-5 are covered in Part 1 (this article) and items 6-10 are covered in Part 2.

The 10 Most Significant EU AI Act Amendments

1. Expanding the list of prohibited AI practices to ban “nudifier apps”.

2. Timeline changes for high-risk AI system compliance.

3. Timeline changes for transparency-requiring AI system compliance.

4. Limiting registration in the EU public database for the “exempted” AI systems.

5. Softening of the AI literacy obligation.

6. Processing sensitive personal data for bias detection and correction.

7. Expanding the scope of the European AI Office’s regulatory powers.

8. Proportionality for small mid-cap (SMC) enterprises.

9. Addressing potential duplication for high-risk AI systems regulated by the EU Machinery Regulation.

10. Addressing potential duplication for AI systems which are regulated by other existing EU product safety laws.

Disclaimer: this article is not legal advice and should not be used, relied on, or interpreted as such. Always consult a qualified legal professional. Furthermore, this article does not cover all amendments outlined in the forthcoming regulation. Finally, these amendments have not yet been enacted as law.

1. Expanding the list of prohibited AI practices to ban “nudifier apps”

What is in law today?

Article 5 of the EU AI Act lists eight distinct prohibited AI practices. These provisions have been applicable since February 2025 and they carry the largest enforcement penalties under the AI Act, of up to 7% of global annual turnover for the most serious violations. The Commission’s original proposal in November 2025 did not feature any changes to the prohibited AI practices.

How is this changing?

A new category of prohibited AI practice is being added to Article 5 of the EU AI Act. Specifically, the “placing on the market, the putting into service or the use of an AI system that generates or manipulates” non-consensual intimate imagery or CSAM will be prohibited. This prohibition covers two broad scenarios:

• The intended purpose of the AI system is the generation or manipulation of non-consensual intimate imagery or CSAM.

• The way the AI system has been developed, trained, and/or configured means that the generation or manipulation of non-consensual intimate imagery or CSAM is a “reasonably foreseeable and reproducible outcome, without requiring significant technical modification, and the system does not have reasonable and adequate technical safety measures” to prevent this.

These two scenarios are most relevant for providers. However, deployers can also fall foul of this prohibition if they use an AI system to generate or manipulate non-consensual intimate imagery or CSAM. This can be done through use or misuse of a prohibited AI system, or potentially even through circumventing the preventive measures and guardrails in an AI system that is not prohibited.

Enterprise implications

This new prohibited AI practice will be applicable from 2 December 2026. The most practical implication is that providers of general-purpose AI (GPAI) systems will need to ensure that they implement robust, state-of-the-art guardrails and technical safeguards to mitigate these risks. According to the recital text, this can include:

• Data cleaning

• Refusal training

• Safe prompt design

• Output controls

• Runtime prompt guardrails

• Content classification and filtering mechanisms

• Usage restrictions

• Abuse detection

• Notice and action mechanisms
  

2. Timeline changes for high-risk AI system compliance

What is in law today?

In law today, the applicable date for high-risk AI system compliance is 2 August 2026 for high-risk AI systems listed in Annex III (e.g., education, employment, and law enforcement) and 2 August 2027 for high-risk AI systems that are products, or safety components of products, regulated by an EU product safety law listed in Annex I. Some parts of the EU AI Act are already in force and applicable—this is not changing. For example, the provisions on prohibited AI practices, AI literacy, and GPAI models are applicable today.

How is this changing?

The EU has agreed to postpone the applicable date for high-risk AI system compliance. The new dates are 2 December 2027 for high-risk AI systems listed in Annex III and 2 August 2028 for high-risk AI systems covered by Annex I. These new dates will apply irrespective of the availability of harmonised standards and associated support tools. However, even though there is no new legal dependency, the regulatory text itself strongly emphasises the importance of the standards being ready and available.

Enterprise implications

The 16-month delay to the high-risk AI system compliance and enforcement date is arguably the most significant amendment for enterprises. It provides some breathing room and ensures that there will not be any investigation or enforcement action relating to high-risk AI system compliance until then. However, anyone who works in this field can appreciate that the time will fly by. Organisations would be unwise to use the delays as a reason to deprioritise.

3. Timeline changes for transparency-requiring AI system compliance

What is in law today?

Article 50 of the EU AI Act outlines the obligations for providers and deployers of transparency-requiring systems, covering topics like AI content labelling and disclosure.

Article 50(2) requires providers of AI systems that generate “synthetic audio, image, video, or text content” to ensure that their AI system outputs are “marked in a machine-readable format and detectable as artificially generated or manipulated”. This can be broadly referred to as the AI output detection and watermarking obligation. Currently, this specific obligation applies from 2 August 2026, alongside all other Article 50 obligations. This compliance date applies to all AI systems, irrespective of whether they are placed on the market or put into service before or after 2 August 2026.

How is this changing?

The compliance date for the AI output detection and watermarking obligation is being postponed by 4 months, to 2 December 2026. However, this postponement only applies to providers who have “already placed their systems on the market before 2 August 2026”, i.e., legacy AI systems that can generate this type of output. For AI systems placed on the market after 2 August 2026, they must be compliant from this date onwards.

Enterprise implications

This is a shorter delay than was initially proposed by the Commission and Council, which both landed on 2 February 2027. It also solidifies the expectation that generative AI providers implement output detection and watermarking capabilities in all new AI systems. The recently published Code of Practice on Transparency of AI-Generated Content can support with this.

4. Limiting registration in the EU public database for the “exempted” AI systems.

What is in law today?

Annex III of the EU AI Act lists eight categories of high-risk AI system, including law enforcement (#6), education and vocational training (#3), and employment, workers’ management and access to self-employment (#5).

However, AI systems listed in Annex III are not high-risk if it can be demonstrated that they do not pose a significant risk of harm to health, safety, or fundamental rights. The parameters of this derogation are outlined in Article 6(3). One of these four conditions must apply for an Annex III AI system to be “exempt” from high-risk classification:

• The AI system is intended to perform a narrow procedural task.

• The AI system is intended to improve the result of a previously completed human activity.

• The AI system is intended to detect decision-making patterns or deviations from prior decision-making patterns and is not meant to replace or influence the previously completed human assessment, without proper human review

• The AI system is intended to perform a preparatory task to an assessment relevant for an Annex III use case.

Providers must register their Annex III high-risk AI systems in the EU’s public database for high-risk AI systems. This registration obligation also includes the “exempted” AI systems that the provider has concluded are not high-risk, via the derogation procedure outlined in Article 6(3). Therefore, if the provider follows the mandatory process of evaluating their AI system and documenting that it is not high-risk, they still need to register it.

How is this changing?

These “exempted” AI systems, which are considered not to be high-risk, will still need to be registered in the EU’s public database for high-risk AI systems. However, less information about these AI systems will need to be submitted as part of the registration. Specifically, the following information attributes will no longer need to be submitted:

• A short summary of the grounds on which the AI system is considered to be not-high-risk in application of the procedure under Article 6(3).

• Any Member States in which the AI system has been placed on the market, put into service or made available in the Union.

These are items 7 and 9 of Annex VIII (Section B) of the EU AI Act. Annex VIII outlines the information items that need to be registered for all high-risk AI systems and the “exempted” AI systems. The other items in Annex VIII (Section B) represent information which providers will have to submit about these “exempted” AI systems in the future.

Enterprise implications

The EU is streamlining the content submission requirement without removing the registration obligation. This could be a tricky obligation to navigate for many enterprises. Although there is an element of simplification (as less information needs to be registered), the fundamental challenge remains: a large number of AI systems which are not high-risk will have to be registered. Any enterprise developing and deploying AI systems in one of the Annex III high-risk AI domains will have to evaluate, classify, track, and govern their AI systems, to keep on top of what needs to be registered and under which pathway. However, for everything that is registered, there will be less public transparency about why it is not considered high-risk.

5. Softening of the AI literacy obligation

What is in law today?

Article 4 of the EU AI Act obliges providers and deployers of AI systems to implement “AI literacy”. The AI literacy obligation has been applicable since February 2025. However, there are no direct enforcement penalties for non-compliance with it.

Specifically, Article 4 requires organisations to “take measures to ensure, to their best extent, a sufficient level of AI literacy of their staff and other persons dealing with the operation and use of AI systems on their behalf”.

How is this changing?

Here is the new text for Article 4(1):

”Providers and deployers of AI systems shall take measures to support the development of AI literacy of their staff and other persons dealing with the operation and use of AI systems on their behalf, taking into account their technical knowledge, experience, education and training and the context the AI systems are to be used in, and considering the persons or groups of persons on whom the AI systems are to be used. This obligation does not require providers or deployers to guarantee any specific level of AI literacy of any individual”.

The key difference is that organisations will no longer be obliged to take measures to “ensure” sufficient levels of AI literacy. Rather, they are obliged to “support the development” of AI literacy. Also, they will not be required to achieve a “sufficient level of AI literacy”. And it is clarified, with a new, additional clause, that organisations are not expected to guarantee the AI literacy level of a specific individual.

Enterprise implications

The change to the AI literacy provision seems fairly moderate on first reading, but it does represent a substantive softening of the legal obligation. Moving from “ensure” to “support” means that although organisations should still take meaningful steps to promote AI literacy, they are less likely to be held accountable for the lack of AI literacy in a particular individual and cannot be held to a measurable standard.

However, nothing in this change should be interpreted as a reason to abandon AI literacy and training which is focused on responsible AI. Furthermore, deployers must continue to assign human oversight of high-risk AI systems to staff with the “necessary competence, training and authority”—this separate provision is not changing. Moreover, actually complying with the EU AI Act in practice will not be possible without organisation-wide AI literacy.

Interestingly, both the Council and the Commission proposed removing the AI literacy obligation for providers and deployers. Ultimately, the Parliament stood its ground and the provision remains, albeit with a softer and more gentle flavour.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

Some organisations still respond to AI with severe or disproportionate restrictions, up to and including outright bans on the use of AI. This article sets out five reasons why that approach backfires, from driving AI use into the shadows to discouraging high-value innovation. The core message is that the purpose of enterprise AI governance is not to stop people from using AI in order to mitigate risk, but to empower the workforce to use AI in a safe, responsible, and informed way.

If you value my work and want to read a comprehensive guide to enterprise AI governance and regulatory compliance, sign up to secure a 25% discount for my forthcoming book, Fundamentals of AI Governance. It will be published in September 2026.

Pre-order my book

Some organisations have implemented relatively cautious AI usage policies. At the more extreme end of the spectrum, this includes bans on the use of AI, either in general or in relation to specific activity areas. This was especially common in the earlier days of generative AI adoption, when there was less comfort and experience with the technology. Such restrictions have been more common in certain sectors, such as universities (e.g., bans on students using AI for coursework), law firms (e.g., bans on using AI to support client advisory work), and healthcare institutions (e.g., bans on using AI in patient-facing services).

Although wholesale AI bans are now less common, many organisations maintain highly restrictive AI policies. This can include only allowing the use of AI for certain use cases, requiring all AI use cases to be risk assessed via cumbersome processes, or only allowing employees to use one AI tool for their work. Anecdotally, it is often remarked that “we let our employees use Copilot, but that’s it”. Whilst the temptation for AI governance practitioners to implement similarly restrictive AI policies is understandable and well-intentioned, it is unlikely to work.

Here are five reasons why banning or significantly restricting the use of AI is not an effective AI policy. Each is explored in detail below:

1. The shadow AI risk

2. Policies must be conceptually sound and practically enforceable

3. Workforce sentiment and engagement

4. Discouraging innovation and high-value use cases

5. The purpose of AI governance is to empower
   

1. The shadow AI risk

Banning or significantly restricting the use of AI does not mean that employees will actually stop using AI. Just as the prohibition of alcohol in the U.S. in the 1920s did not stop people from purchasing and consuming alcohol, AI prohibitions do not stop AI use. Rather, it pushes it into the shadows. However, instead of a raucous Manhattan speakeasy bar, the ‘shadow’ is the publicly available version of ChatGPT, Claude, or Gemini. This is because no matter what your policy says, people will still want and need to use AI in their work, and they will find ways to do so.

Shadow AI is one of the biggest challenges for enterprise AI governance and cybersecurity teams. Irrespective of your AI usage policy and its requirements, it is extremely difficult to deter and prevent the use of publicly available AI tools which are not approved. This shadow AI use poses a range of data and compliance risks. For example, it is challenging to protect any confidential data which is uploaded. Depending on the AI tool, such data may be shared with third parties or used to train publicly available AI models, increasing the risk of data leakage. It is also difficult to have meaningful oversight of potentially high-risk or regulated use cases and to ensure risks are effectively mitigated. Consider the law firm example provided above. Exposing client data via publicly available AI could be highly damaging. This is because it could breach the firm’s duty of confidentiality and, depending on the precise circumstances and AI tool used, risk waiving legal professional privilege. Therefore, whilst banning AI does not stop its use, it does mean you have less visibility and control.

It may seem counterintuitive, but one of the most effective AI governance ‘controls’ is to provide your workforce with access to a wide range of cutting-edge AI tools, covering a multitude of capabilities and models. These AI tools should be complemented by proportionate and actionable usage policies, as well as clear guidance explaining which tools are available and approved for use, how to access them, and what their capabilities and limitations are. Finally, these tools should be configured with enterprise-grade guardrails and security controls, to protect your data and intellectual property.

The easier it is for employees to access and use approved and secure internal AI tools to solve their problems, the less incentivised they will be to use shadow AI, and the more visibility and control you will have.

2. Policies must be conceptually sound and practically enforceable

AI is a general-purpose technology which is increasingly being embedded in all software and devices that people use. When you use Google Maps, AI models help you navigate from A to B. When you order an Uber, AI models connect you with your driver. And when you use a search engine, AI models generate AI overviews for you to read. In this context, if you are going to ban or significantly restrict the use of AI, it will be challenging to define what the scope of the prohibition is and how it should be interpreted and complied with in practice. Consider a student that is told they cannot use AI as part of the essay-writing process. Does this also mean that when doing their research, they cannot read and factor in the AI overviews generated by the search engine? And why is this different from reading the content contained in the search results directly underneath, which are ranked and filtered by AI, and may also feature AI-generated content? As AI becomes ubiquitous, wholesale prohibitions are conceptually unsound and difficult to justify, implement, and enforce.

However, this does not mean that you should not restrict certain types of AI or data use. It is important to articulate and codify what your prohibited AI practices are, to ensure regulatory compliance and to uphold ethical standards. Also, certain types of data use may be inappropriate in certain AI tools, contexts, or jurisdictions. For example, AI overviews are not generated by Google in response to every search query, and are deliberately disabled in certain circumstances (e.g., certain queries during election campaigns due to the risks of AI-generated misinformation). Also, there are many reasons why universities will want to prohibit students from using AI to generate essays and submitting them as if they were their own work. However, this is more targeted and practical to enforce against than “do not use AI at all for your coursework”.

When you do implement an AI-related prohibition or restriction, it should be tightly defined, well-scoped, and supported by a rationale grounded in commercial, legal, or ethical considerations. If you do not take this approach, you risk your AI policies being perceived as disconnected from reality or even illegitimate. Furthermore, there will be lower levels of compliance and highly uneven or limited enforcement. All of this would have damaging implications for the credibility and effectiveness of the AI governance function.

3. Workforce sentiment and engagement

It is not groundbreaking to claim that AI poses challenges for the workforce and for young people in particular. There is widespread concern and fear about what AI means for the future of work. This is leading to increased recognition of the importance of AI literacy and upskilling. Employees and students alike recognise that developing and honing AI skills, knowledge, and experience is one of the best things they can do for their career. If your organisation is perceived as actively blocking or undermining this—via overly restrictive AI usage policies or by failing to provide advanced AI tools for general use—then this is bound to have a negative impact. Merely saying “we let our employees use Copilot” and restricting everything else is no longer going to cut it. Such a rigid approach could also lead to difficulties in attracting and retaining talent.

Students, for example, may feel that they are not being adequately prepared for their future career if their university discourages them from using AI. A doctor may feel they are letting patients down by not leveraging the latest tech. And knowledge workers may feel that their long-term career aspirations are being undermined by an excessively cautious employer which is failing to keep up with the pace of technological change. This sentiment will be particularly acute amongst technical employees working in engineering, data science, and IT roles, as well as AI enthusiasts that want to vibe code prototypes and take advantage of all that agentic AI has to offer. Negative workforce sentiment and engagement is therefore a likely unintended consequence of banning or significantly restricting AI use. Do not underestimate the impact of sending the wrong message at the wrong time.

4. Discouraging innovation and high-value use cases

This negative impact on employee engagement is likely to be exacerbated by broader opportunity costs relating to value generation. Although the precise level of impact is impossible to predict or quantify, it is highly likely that any organisation that takes an overly cautious approach will miss out on the efficiency, productivity, and innovation benefits that AI brings and the value that this can generate. This potential “loss” could be transformative or moderate, and will occur even with widespread shadow AI use. The precise scale of the opportunity cost depends on many factors. However, even in the best case scenario, over time, an organisation which is overly restrictive and discourages the use of AI will struggle to keep pace with competitors that take a more permissive and pro-innovation approach.

Simply put, by discouraging the use of AI, you will miss out on the use cases and innovations that could have positively transformed your business and its position in the market. Therefore, in one fell swoop you may be creating a disgruntled workforce and a less innovative and competitive organisation.

For those employees that circumvent the rules and leverage shadow AI, this activity is far less likely to result in impactful AI use cases and innovative ways of working that can be reused at scale across different teams and functions. If individuals and teams are knowingly breaching policies, they are more likely to keep quiet and perhaps only use AI for their personal productivity. In such an environment, an organisation will miss out on the benefits of employees exchanging and sharing information, best practice, and learnings about their AI use cases. Such knowledge exchange is valuable for promoting a culture of experimentation, innovation, and AI fluency, all of which is integral to successful AI adoption at scale.

5. The purpose of AI governance is to empower

Ultimately, the purpose of enterprise AI governance is not to discourage, deter, or stop people from using AI. Wholesale prohibition is not an effective way of mitigating risk. Such an approach is characterised by a lack of nuance on a profoundly complex issue. Enterprise AI governance must be aligned with the broader strategic objectives of the organisation. Therefore, its aim should be to encourage and enable safe and responsible AI adoption. This can be done by implementing proportionate guardrails that promote governance by design and embedding a culture of openness and informed risk-taking. AI usage policies can support this by providing actionable dos and don’ts, making it clear which AI tools are approved for use, promoting a culture of experimentation and innovation, and clarifying which types of activities are prohibited, higher-risk, or subject to additional governance.

More broadly, forward-thinking organisations are thinking deeply about the role of the human in the AI era and making efforts to ensure their policies and guidance are empowering and ahead of the curve.

For example, many universities are no longer telling students to not use AI. Rather, they are highlighting, with actionable frameworks, guidelines, and case studies, the types of AI use that are appropriate, inappropriate, and borderline for different types of activities. This may differ from module to module and assignment to assignment. More broadly, they are educating students on why an informed and intentional approach to AI use matters for preserving academic integrity—and why academic integrity itself is important. They are teaching students how to use AI in a manner that advances critical thinking, and why the fastest route is not always the best route. Just telling a student to not use AI does not tell them why there is value in deepening their own expertise through structured engagement with primary literature, as well as what long-term benefits they may attain from refining their own thinking through independent writing. However, they are also teaching students how AI can be used to augment and accelerate various aspects of the research, writing, review, and editing processes, in a manner that does not undermine any of the aforementioned objectives.

By empowering your organisation with access to cutting-edge AI tools and capabilities, reinforcing this with high-quality AI literacy training, and encouraging your workforce to use AI to solve problems they face, you can achieve a transformative level of organisational AI fluency. Rather than telling people to not use AI, make it easy for everyone to understand what safe, responsible, and compliant AI use entails. And, if necessary, clarify where there are specific prohibitions or restrictions and why these exist. This is what effective and balanced AI governance looks like in practice.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

Apologies for the brief hiatus over the past couple of weeks. I’ve been hard at work finishing my book, Fundamentals of AI Governance, which I’m writing in my spare time alongside a rather busy day job. It will be published on 7 September 2026 and I can’t wait to share it with the community! Until then, I will post updates on Enterprise AI Governance every two weeks. You can pre-order the book here, to secure a 25% discount:

Pre-order my book

This week’s newsletter presents the updated definitive agentic AI governance resource guide. This article (best viewed on browser) collates 80+ authoritative resources. Most of these resources have been published in the past 18 months and are grouped into 9 categories:

1️⃣ Agentic AI: technology foundations and capabilities
2️⃣ How enterprises are developing and deploying AI agents
3️⃣ Agentic AI risks and challenges
4️⃣ Risk mitigation and safeguards (security and safety)
5️⃣ Agentic AI evaluations and observability
6️⃣ Human oversight, accountability, and liability
7️⃣ Agentic AI governance frameworks
8️⃣ How EU law governs agentic AI: EU AI Act and GDPR
9️⃣ AI provider guidance and agentic AI usage policies

This is a substantial update to my original resource guide, published in February 2026. I've added dozens of new resources, reflecting an exceptionally active period for agentic AI governance. This includes landmark outputs from the OECD, NIST, UK AISI, Spanish DPA, European Commission, and Dutch DPA, alongside major new academic and industry contributions—particularly on agentic AI and the EU AI Act. I've also introduced a new category—AI provider guidance and agentic AI usage policies—to capture the wave of usage policies and guidance that companies like Anthropic, OpenAI, Microsoft, and Google have published.

Whether you are working on the frontline of enterprise AI governance, advising clients in a legal or consulting capacity, or building, deploying, and using agentic AI systems, this resource guide enables you to keep track of and understand this rapidly evolving field.

The Ultimate Agentic AI Governance Resource Guide

80+ resources to master agentic AI governance

1) Agentic AI: technology foundations and capabilities

New resources

• The agentic AI landscape and its conceptual foundations - OECD, (2026)

• Technical Performance (Chapter 2) - Stanford AI Index Report, (2026)

Featured in previous resource guide

• The 2026 Guide to AI Agents - IBM (2026)

• A practical guide to building agents - OpenAI (2025)

• Building Effective AI Agents - Anthropic (2024)

• Agents - Chip Huyen (2025)

• Delegation Nation: Advanced AI Assistants and why they matter - Harry Farmer and Julia Smakman, Ada Lovelace Institute (2025)

• Large Language Model Agents: A Comprehensive Survey on Architectures, Capabilities, and Applications - Yiming Lei et al., (2025)

• ReAct: Synergizing Reasoning and Acting in Language Models - Shunyu Yao et al., (2023)

• Toolformer: Language Models Can Teach Themselves to Use Tools - Timo Schick et al., (2023)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - Jason Wei et al., (2022)

• What is the Model Context Protocol (MCP)? (2025)

• What is the MCP and how does it work? - Google (2025)

• Announcing the Agent2Agent Protocol (A2A): A new era of Agent Interoperability - Google (2025)

2) How enterprises are developing and deploying AI agents

Featured in previous resource guide

• The 2026 State of AI Agents Report: How enterprises are building and deploying AI in production - Anthropic (2026)

• 2026 Agentic Coding Trends Report - Anthropic (2026)

• Measuring Agents in Production - Melissa Z. Pan et al. (2025)

• State of Agent Engineering - LangChain (2026)

• The state of AI in 2025: Agents, innovation, and transformation - Alex Singla et al., McKinsey QuantumBlack (2025)

• Frontier AI Trends Report - UK AI Security Institute (2025)

3) Agentic AI risks and challenges

New resources

• Tech Futures: Agentic AI and Data Protection - UK Information Commissioner’s Office, (2026)

• International AI Safety Report 2026 - Yoshua Bengio et al., Department for Science, Innovation and Technology, (2026)

• Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt Injections - David Schmotz, Sahar Abdelnabi, and Maksym Andriushchenko, (2025)

Featured in previous resource guide

• OWASP Top 10 for Agentic Applications for 2026 - OWASP (2025)

• Agentic AI - Threats and Mitigations - OWASP (2025)

• Multi-Agentic system Threat Modelling Guide v1.0 - OWASP (2025)

• Initial reflections on agentic AI governance - Oliver Patel, Enterprise AI Governance (2025)

• Agentic AI risks and challenges enterprises must tackle - Domino Data Lab (2025)

4) Risk mitigation and safeguards (security and safety)

New resources

• The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems - Leon Staufer et al., (2025)

• Secure AI Framework (SAIF) 2.0: Focus on Agents - Google, (2025)

Featured in previous resource guide

• Practices for Governing Agentic AI Systems - Yonadav Shavit et al., OpenAI (2023)

• Infrastructure for AI Agents - Alan Chan et al., Centre for the Governance of AI (2025)

• AI Agents: Governing Autonomy in the Digital Age - Joe Kwon, Center for AI Policy (2025)

• Securing Agentic AI: An Addendum to the Guidelines and Companion Guide on Securing AI Systems - Cyber Security Agency of Singapore (2025)

• Deploying agentic AI with safety and security: A playbook for technology leaders - McKinsey (2025)

• Securing Agentic AI: A Discussion Paper - Cyber Security Agency of Singapore and Far.AI (2025)

• Google’s Approach for Secure AI Agents - Christoph Kern and Kara Olive, Google (2025)

• Agents Rule of Two: A Practical Approach to AI Agent Security - Meta (2025)

  Thanks for reading Enterprise AI Governance! Subscribe for free to receive new posts and support my work.

5) Agentic AI evaluations and observability

New resources

• Agents of Chaos - Natalie Shapira et al., (2026)

• The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis - Peiran Wang et al., (2026)

• AI Agents Break Rules Under Everyday Pressure - Matthew Hutson, IEEE Spectrum, (2025)

Featured in previous resource guide

• Strengthening AI Agent Hijacking Evaluations - U.S. AI Safety Institute (2025)

• How to Evaluate Control Measures for LLM Agents? A Trajectory from Today to Superintelligence - Tomek Korbak et al., UK AI Security Institute (2025)

• Demystifying evals for AI agents - Anthropic (2026)

• Prioritizing Real-Time Failure Detection in AI Agents - Madhulika Srikumar et al., Partnership on AI (2025)

• Agent Factory: Top 5 agent observability best practices for reliable AI - Yina Arenas, Microsoft (2025)

6) Human oversight, accountability, and liability

New resources

• Agentic AI: The liability gap your contracts may not cover - Clifford Chance, (2026)

• Intelligent AI Delegation - Nenad Tomasev et al., Google DeepMind, (2026)

• Levels of Autonomy for AI Agents - K. J. Kevin Feng et al., (2025)

Featured in previous resource guide

• Governing AI Agents - Noam Kolt (2025)

• The Principal-Agent Alignment Problem in AI - Dylan Hadfield-Menell, UC Berkley (2021)

• AI and Human Oversight: A Risk-Based Framework for Alignment - Laxmiraju Kandikatla and Branislav Radeljic (2025)

• The dilemmas of delegation: an analysis of policy challenges posed by Advanced AI Assistants and natural-language AI agents - Harry Farmer, Ada Lovelace Institute (2025)

• Fully Autonomous AI Agents Should Not be Developed - Margaret Mitchell et al., Hugging Face (2025)

• Human-in-the-Loop for AI Agents: Best Practices, Frameworks, Use Cases, and Demo - Gabriel L. Manor, Permit.io (2025)

7) Agentic AI governance frameworks

New resources

• Scalable Runtime Governance for Agentic AI in Financial Services - Lukasz Szpruch et al., (2026)

Featured in previous resource guide

• Model AI Governance Framework for Agentic AI - Singapore Infocomm Media Development Authority (2026)

• AI Agents in Action: Foundations for Evaluation and Governance - World Economic Forum (2025)

• AI Agent Governance: A Field Guide - Jam Kraprayoon, Institute for AI Policy and Strategy (2025)

• Characterising AI Agents for Alignment and Governance - Atoosa Kasirzadeh & Iason Gabriel (2025)

• The State of Agentic AI Security and Governance - OWASP (2025)

• The regulation of delegation: Are AI advisers, agents and companions regulated in the UK? - Julia Smakman, Ada Lovelace Institute (2025)

• Effective legal protections from harms caused by advanced AI assistants - Lucie Audibert & Alex Lawrence-Archer, AWO Agency (2025)

8) How EU law governs agentic AI: EU AI Act and GDPR

New resources

• AI Agents Under EU Law: A Compliance Architecture for AI Providers - Luca Nannini et al., (2026)

• How are AI agents addressed within the AI Act? - European Commission FAQs, (2026)

• Agentic AI from the Perspective of Data Protection - Spanish Data Protection Agency, (2026)

• Regulating AI Agents - Kathrin Gardhouse, Amin Oueslati, and Noam Kolt, (2026)

• Human Oversight under Article 14 of the EU AI Act - Melanie Fink, (2026)

• Autoriteit Persoonsgegevens (AP) warns of major security risks with AI agents like OpenClaw - Dutch Data Protection Authority, (2026)

Featured in previous resource guide

• Ahead of the Curve: Governing AI Agents Under the EU AI Act - Amin Oueslati and Robin Staes-Polet (2025)

• Agentic Tool Sovereignty - Lloyd Jones, European Law Blog (2025)

• Engineering GDPR compliance in the age of agentic AI - Keivan Navaie, IAPP (2025)

• Minding Mindful Machines: AI Agents and Data Protection Considerations - Daniel Berrick, Future of Privacy Forum (2025)

  ---

9) AI Provider Guidance and Agentic AI Usage Policies

New resources

Anthropic

• Anthropic Usage Policy (with dedicated agentic AI provisions) (2025)

• Using Agents According to Our Usage Policy (2026)

• Use Claude Cowork Safely (2026)

OpenAI

• OpenAI Usage Policies (2025)

• Using ChatGPT agent in line with our policies (2025)

• Agent approvals & security (Codex) (2026)

Microsoft

• Microsoft Copilot Studio Application Card (2026)

• Responsible AI FAQs for Copilot Studio (2026)

• Agentic AI Adoption Maturity Model: AI Governance and Security Pillar (2026)

Google

• Gemini API Additional Terms of Service (2026)

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

This week I'm releasing a new Cheat Sheet which maps and compares the negotiating positions of the European Commission, Council of the EU, and European Parliament on AI Act amendments—across eight key domains. It takes 180+ pages of legislative complexity and distils it into one graphic. You can download it for free below (it’s much better viewed on a large screen as a high-res pdf).

If you value my work and want to read a comprehensive, visual guide to enterprise AI governance implementation—with over 150 original cheat sheets like this—sign up to pre-order my forthcoming book, Fundamentals of AI Governance (2026), for a 25% discount.

Pre-order my book

Frequent readers of Enterprise AI Governance will recall that this newsletter has covered the EU AI Act amendment process in detail. Understanding and adhering to the AI Act is integral for AI governance practitioners and the organisations they support, given the uniquely comprehensive and stringent nature of this law, as well as its extraterritorial scope and cross-sectoral coverage.

As if AI governance was not hard enough already, practitioners now find themselves in the unenviable position of having to track and make sense of a raft of proposed amendments to this fiendishly complex law. Given that the AI Act only entered into force in August 2024 and is not yet fully applicable, the fact that substantive amendments are already being advanced is telling. It signals both the challenges of moving as early to regulate AI as the EU did—something which governments like the UK warned about—as well as the increasing pressure within the EU to make life easier for businesses.

Although, perhaps every AI governance practitioner can empathise with the EU, to a certain extent. Having to constantly adapt is a familiar feeling and a defining challenge of this field, given the relentless pace at which the AI technology and risk landscape is evolving, coupled with the increasing emphasis corporate leaders are placing on the importance of AI for their organisation’s future success.

Last week was a significant milestone for the AI Act amendment process. The European Parliament formally adopted its position on AI Act simplification, with MEPs voting to adopt a set of proposed amendments on 26 March 2026. The margin of approval was decisive: 569 votes in favour, 45 against, and 23 abstentions.

Following months of work behind the scenes, we now have the official negotiating positions for each of the three co-legislating institutions. The trilogue negotiations can now begin. The clock is ticking to August 2026, when the high-risk AI compliance date is due to kick in.

Each of these documents outline the respective institutions’ position for how the AI Act should be amended (via a new regulation):

• European Commission position (published 19 November 2025)

• Council of the EU (member states) position (13 March 2026)

• European Parliament position (26 March 2026)

My intention in covering this process has been to support readers with some of the analytical heavy lifting, by breaking down the complexity of these developments in a digestible way. Indeed, there are now over 180 pages of dense legislative text to wade through, in addition to 144-page AI Act itself.

Previous editions of Enterprise AI Governance have already covered:

• Why the EU is amending the flagship digital laws, like the AI Act, and the impact of the Draghi report

• A deep-dive on the European Commission’s proposal for AI Act amendments

• A comparative analysis of the Commission and Council positions

The key contribution of this week’s edition is an original Cheat Sheet, exclusively available for free download below, which compares and contrasts the negotiating positions of the Commission, Council, and Parliament—mapped against what is in law today. Although I will not do a detailed write-up of the European Parliament’s position, the visual highlights the key aspects.

Of note, both the Council and the Parliament have somewhat curtailed some of the more ambitious simplification measures proposed by the Commission (e.g., on sensitive data processing and registration of “exempted” AI systems), and have (broadly speaking) aligned on a new prohibited AI practice. Also, the Parliament is ostensibly more supportive than the Council and Commission of retaining a broad and explicit AI literacy obligation. Crucially, all three institutions are converging on 2 December 2027 as the applicable date for the high-risk AI systems listed in Annex III. Such a delay will give enterprises 16 additional months to get compliant.

This visual is much better viewed on a large screen as a downloaded pdf :)

EU AI Act Amendments Cheat Sheet (by Oliver Patel)

365KB ∙ PDF file

Download

Download the high-res pdf here

Download

Disclaimer: nothing in this article or Cheat Sheet constitutes legal advice and this content should not be used, relied on, or interpreted as such. It is for educational purposes only. Always consult a qualified legal professional for your specific compliance or legal needs.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

On 19 November 2025, the European Commission proposed targeted amendments to the EU AI Act as part of its Digital Omnibus package. On 13 March 2026, the Council of the EU (i.e., the EU member state governments) agreed its negotiating position, broadly maintaining the Commission’s proposals whilst introducing several notable changes. Finally, on 18 March, the European Parliament’s Internal Market and Civil Liberties committees adopted their joint position by 101 votes to 9, with a plenary vote expected on 26 March. Trilogue negotiations between the three institutions will follow this vote.

This article analyses the eight most consequential proposed amendments, comparing and contrasting the positions of the Commission and the Council. For each, I explain what is in the law today, what the Commission is proposing, and what the Council is proposing. I have excluded the European Parliament’s position from this analysis, as it has not yet been formally approved by MEPs via the plenary vote.

If you value my work and want to read a comprehensive guide to the EU AI Act and enterprise AI governance, sign up to secure a 25% discount for my forthcoming book, Fundamentals of AI Governance (2026).

Pre-order my book

The key AI Act changes this article analyses are:

1. Expanding the list of prohibited AI practices.

2. Timeline changes for high-risk AI system compliance.

3. Timeline changes for transparency-requiring AI system compliance.

4. Limiting registration in the EU public database for high-risk AI systems.

5. Softening of the AI literacy obligation.

6. Processing sensitive personal data for bias mitigation.

7. Expanding the scope of the European AI Office’s regulatory powers.

8. Proportionality for small mid-cap (SMC) enterprises.

Before diving in, three caveats are needed. First, this article presents the positions of two of the three co-legislating EU institutions—neither of which represents the final text. Second, the upcoming trilogue negotiations may result in significant changes. Third, and critically, the existing AI Act remains fully in force. Unless and until these amendments are formally adopted, the current obligations and timelines apply, including the 2 August 2026 applicable date for high-risk AI system compliance.

Here are the key documents this article is based on:

• European Commission AI Act amendments and negotiating position (November 2025)

• Council of the EU AI Act amendments and negotiating position (March 2026)

• European Parliament Committee position and vote details (March 2026)

• Original text of EU AI Ac

Scope: this article focuses on the eight EU AI Act changes that are most consequential for enterprises implementing AI governance. It does not cover all EU AI Act changes, nor the proposed amendments to the GDPR or other EU digital legislation.

Disclaimer: this article is not legal advice and should not be used, relied on, or interpreted as such. Always consult a qualified legal professional.

1. Expanding the list of prohibited AI practices

What is in law today?

Article 5 of the EU AI Act lists eight distinct prohibited AI practices. These provisions have been applicable since February 2025 and they carry the largest enforcement penalties under the AI Act, of up to 7% of global annual turnover for the most serious violations.

What is the European Commission’s position

The Commission did not propose amendments or additions to the prohibited AI practices outlined in Article 5.

What is the Council’s position?

The most significant amendment proposed by the Council is to add two new prohibited AI practices to Article 5 of the AI Act. Under this proposal, AI systems capable of “generating, manipulating or reproducing” non-consensual intimate imagery (and similar content) would be prohibited, as well as AI systems capable of “generating, manipulating or reproducing” CSAM. These prohibitions cover the following scenarios:

• The intended purpose of the AI system is to generate, manipulate or reproduce non-consensual intimate imagery or CSAM.

• It is a “reasonably foreseeable” reproducible outcome that the AI system could be used in this way_, without requiring significant technical modification, due to the way the AI system has been developed and its functionality. Furthermore, the safety measures and guardrails are not effectively able to prevent this type of use.

The second point is significant, as it puts the onus on AI system providers to ensure that the safety features and guardrails they develop and configure are sufficiently robust to prevent this type of malicious and harmful use. The Council provides the following as examples of effective safety features and guardrails:

• Refusal training

• Data cleaning

• Output controls

• Content classification and filtering

• Usage restrictions

• Abuse detection

• Notice action and corrective measures
  

2. Timeline changes for high-risk AI system compliance

What is in law today?

From 2 August 2026, unless there is a change in the law, providers and deployers must adhere to the obligations and requirements for high-risk AI systems. However, this applicable date only applies to high-risk AI systems listed in Annex III (e.g., education, employment, law enforcement etc.) that are placed on the market or put into service from 2 August 2026 onwards. Annex III high-risk AI systems that predate this are only subject to these EU AI Act obligations and requirements if there is a significant change in the AI system’s design or intended purpose. For high-risk AI systems that are products, or safety components of products, regulated by the EU product safety laws listed in Annex I, the applicable date is 2 August 2027.

What is the European Commission’s position

The Commission’s position is to link the availability of standards with the applicable date for high-risk AI system compliance. To do this, two options are proposed.

Scenario 1. If harmonised standards and associated support tools for high-risk AI system compliance are finalised and approved by the Commission, then the applicable compliance date should be six months after this approval (for high-risk AI systems listed in Annex III) and 12 months after this approval (for product safety-related high-risk AI systems covered by Annex I).

Scenario 2. If harmonised standards and associated support tools for high-risk AI system compliance are not finalised or approved by the Commission in a given timeframe (i.e., before the dates below), then the applicable compliance dates should be 2 December 2027 (for high-risk AI systems listed in Annex III) and 2 August 2028 (for product safety-related high-risk AI systems covered by Annex I). These effectively serve as backstop dates.

What is the Council’s position?

The Council’s position is more straightforward. Its proposal is that the applicable date for high-risk AI system compliance should be 2 December 2027 (for high-risk AI systems listed in Annex III) and 2 August 2028 (for product safety-related high-risk AI systems covered by Annex I). These dates would apply irrespective of the availability of harmonised standards and associated support tools.

3. Timeline changes for transparency-requiring AI system compliance

What is in law today?

Article 50(2) requires providers of AI systems that generate “synthetic audio, image, video, or text content” to ensure that their AI system outputs are “marked in a machine-readable format and detectable as artificially generated or manipulated”. Currently, this specific obligation applies from 2 August 2026. This compliance date applies to all AI systems, irrespective of whether they are placed on the market or put into service before or after 2 August 2026.

What is the European Commission’s position?

The Commission proposes to postpone the applicable date for this specific transparency obligation to 2 February 2027 for providers of AI systems that have been placed on the market before 2 August 2026. This proposed six-month postponement only applies to obligation stipulated in Article 50(2) and not the other transparency obligations outlined in Article 50.

What is the Council’s position?

The Council’s position is the same as the Commission’s. No changes were proposed to the Commission’s original proposal.

4. Limiting registration in the EU public database for high-risk AI systems

What is in law today?

Annex III of the EU AI Act lists eight categories of high-risk AI system, including law enforcement (#6), education and vocational training (#3), and employment, workers’ management and access to self-employment (#5).

However, AI systems listed in Annex III are not considered high-risk if it is demonstrated that they do not pose significant risk of harm to health, safety, or fundamental rights. For example, if the AI system does not materially influence decisions or is only used for a narrow procedural task, the provider is entitled to demonstrate, based on a documented assessment, that it is not a high-risk AI system. The parameters of this derogation are outlined in Article 6(3). This derogation procedure only applies to AI systems listed in Annex III.

Providers must register high-risk AI systems listed in Annex III in the EU public database for high-risk AI systems. This registration obligation also includes “exempted” AI systems that the provider has concluded are not high-risk via the derogation procedure outlined in Article 6(3).

What is European Commission’s position?

The Commission proposes to limit the scope of this registration obligation so that it no longer applies to AI systems that providers have concluded are not high-risk via the Article 6(3) derogation procedure. Simply put, where a provider has assessed and documented that an AI system used in an Annex III domain is not high-risk, the provider would not have to register that AI system in the EU public database.

What is the Council’s position?

The Council is proposing a more moderate amendment. Rather than completely removing the registration obligation for these exempted AI systems, registration would still be mandatory. However, less information would be required to be submitted as part of the registration. Annex VIII of the EU AI Act lists 9 information attributes that must be registered in relation to these exempted AI systems. The Council’s position is to remove 2 out of 9 attributes:

• The summary of why the AI system is not high-risk.

• The EU member states in which the AI system is placed on the market, put into service, or made available.

The purpose of this is to make the registration process more simplified and streamlined in these scenarios, whilst retaining the transparency and accountability registration provides.

5. Softening of the AI literacy obligation

What is in law today?

Article 4 of the EU AI Act obliges providers and deployers of AI systems to implement “AI literacy”. Specifically, Article 4 requires organisations to ensure that “staff and other persons dealing with the operation and use of AI systems” have a “sufficient level of AI literacy”. The AI literacy obligation has been applicable since February 2025. However, there are no enforcement penalties for non-compliance with it.

What is the European Commission’s position?

The Commission proposes to remove the obligation for providers and deployers to implement AI literacy. Rather than providers and deployers being legally obliged to ensure their staff operating and using AI systems have sufficient levels of AI literacy, the Commission and member states will be required to foster and encourage AI literacy.

What is the Council’s position?

The Council’s position is largely aligned with the Commission’s. There would no longer be a broad and widely applicable AI literacy obligation for providers and deployers of AI systems. Rather, the Commission and member states will be obliged to encourage AI literacy across the EU.

However, the Council’s proposal is to clarify, in Article 4, that providers and deployers of high-risk AI systems have specific AI literacy and training-related obligations, stipulated or implied in different parts of the EU AI Act. For example, deployers must assign human oversight (of high-risk AI systems) to individuals with the “necessary competence, training and authority”. Also, as part of the provider’s quality management framework, roles and responsibilities must be assigned to management and other staff, and these individuals must be adequately trained and competent to fulfil these responsibilities.

Nonetheless, this clarification does not introduce substantive new AI literacy or training-related obligations for organisations. It merely reinforces what is already required for high-risk AI system compliance.

6. Processing sensitive personal data for bias mitigation

What is in law today?

The GDPR stipulates that the processing of “special categories” of personal data (i.e., sensitive personal data) is prohibited, apart from in limited circumstances. Sensitive personal data includes:

• Racial or ethnic origin

• Political opinions

• Religious or philosophical beliefs

• Trade union membership

• Genetic data

• Biometric data for the purpose of uniquely identifying a natural person

• Health data

• Sex life or sexual orientation

Sensitive data can only be processed in limited circumstances, where a specific GDPR Article 9(2) exception applies. This includes, but is not limited to, where explicit consent has been obtained or where processing the data is necessary to “protect the vital interests of the data subject” where they are “physically or legally incapable of giving consent”.

Article 10(5) of the AI Act provides another way in which sensitive personal data can be processed. It stipulates that providers of high-risk AI systems can process sensitive personal data for the purpose of “ensuring bias detection and correction” in relation to their high-risk AI systems. However, it must be “strictly necessary” to use the sensitive personal data in this way, and the bias detection and correction cannot be effectively achieved by other means. These conditions create a high practical bar for providers. Furthermore, this exception is subject to additional strict conditions, such as implementing robust safeguards relating to data transfer, reuse, and deletion.

What is the European Commission’s position?

The Commission’s proposal is to broaden the scope of when organisations can process sensitive personal data for bias detection and correction. Rather than limiting this exclusively to providers of high-risk AI systems, it is expanded to providers and deployers of other AI systems and AI models, including deployers of high-risk AI systems. The rationale for this is because harmful biases could also arise from the use of AI in other contexts.

This means that sensitive personal data could be lawfully processed for bias detection and correction in a much broader range of circumstances and by a wider range of organisations. For this reason, the proposal is to move this provision from Article 10 to Article 4a of the AI Act, as it applies more broadly than to just high-risk AI systems.

Additionally, rather than this type of data processing only being permitted when it is “strictly necessary”, sensitive data could be processed when it is merely “necessary” (to ensure bias detection and correction). This subtle change lowers the bar for when organisations can use sensitive personal data in this way.

What is the Council’s position?

The Council largely agrees with broadening the scope—beyond providers of high-risk AI systems—for when sensitive personal data could be processed for bias detection and correction. This means that providers and deployers of other AI systems, including deployers of high-risk AI systems, would be permitted to process sensitive personal data for this bias detection and correction. However, the Council’s proposed changes are more limited than the Commission’s, in two important ways:

• Processing sensitive personal data would still have to be “strictly necessary” for ensuring bias detection and correction.

• Providers and deployers of other AI systems and deployers of high-risk AI systems would only be permitted to do so to address specific types of biases impacting health, safety, fundamental rights, or discrimination.

Therefore, although a wider range of organisations could process sensitive personal data for bias detection and correction in a broader range of circumstances than what the AI Act currently allows for, it would nonetheless be more limited than what the Commission is proposing.

7. Expanding the scope of the European AI Office’s regulatory powers

What is in law today?

The AI Act enforcement architecture for AI systems is decentralised.

The AI Office, which is part of the European Commission, is responsible for overseeing and enforcing the provisions on general-purpose AI (GPAI) models. At the member state level, the national market surveillance authorities (there are typically several per member state) are responsible for overseeing and enforcing the provisions on AI systems (e.g., high-risk and transparency-requiring AI systems), as well as most other AI Act provisions.

Article 75(1) provides the AI Office with powers to monitor and supervise AI systems based on GPAI models. This applies when the AI model and AI system are developed by the same provider. However, this is not an exclusive power. National market surveillance authorities can also oversee and enforce applicable provisions relating to these AI systems. This creates potential situations of overlapping competence and providers being regulated by multiple regulators simultaneously.

What is the European Commission’s position?

The Commission proposes to “centralise oversight over a large number of AI systems built on general-purpose AI models” when the same provider develops both the GPAI model and the AI system.

The proposed amendments to Article 75 would render the AI Office as the sole body responsible for monitoring and supervising compliance of AI systems that leverage GPAI models. However, this would only apply when the GPAI model and the AI system are developed and placed on the market or put into service by the same provider. In such scenarios, the AI Office would be “exclusively competent”, which means that the market surveillance authorities in the respective EU member states would no longer have a supervisory role. The AI Office would also have “all the powers of a market surveillance authority”. In a nutshell, the key change to Article 75 is the shift from shared to exclusive competence.

This change primarily affects high-risk AI systems listed in Annex III and transparency-requiring AI systems regulated by Article 50 (where such AI systems leverage general-purpose AI models). It does not apply to high-risk AI systems covered by an EU product safety law listed in Annex I.

What is the Council’s position?

Whilst the Council broadly aligns with the Commission’s proposals, it has introduced specific exceptions to this expansion in scope of the AI Office’s regulatory powers.

Under the Council’s proposal, the AI Office would be “exclusively competent” for the supervision and enforcement of AI Act provisions for AI systems based on GPAI models where the AI model and AI system are developed by the same provider— which includes different entities in the same corporate group (a subtle change to the Commission’s position). However, the exceptions to this AI Office exclusive competence are:

• Product safety-related high-risk AI systems covered by Annex I.

• AI systems used for critical infrastructure management and operation (Annex III, point 2).

• AI systems provided by law enforcement authorities and border management authorities.

• AI systems provided by certain financial institutions.

• AI systems used for the administration of justice and democratic processes (Annex III, point 8).

This proposal means that, for the above exceptions, the AI Office would not have exclusive competence. The Council is signalling that these areas (i.e., critical infrastructure, law enforcement, border management, financial services etc.) should be the domain of member state regulators, not the European Commission.

8. Proportionality for small mid-cap (SMC) enterprises

The AI Act provides an element of flexibility and proportionality for micro, small, and medium-size enterprises (SMEs), including start-ups. For example, the compliance penalties which SMEs can face are capped as follows:

• 35 million EUR or 7% of total worldwide annual turnover (whichever is lower).

• 15 million EUR or 3% of total worldwide annual turnover (whichever is lower).

• 7.5 million EUR or 1% of total worldwide annual turnover (whichever is lower).

This contrasts with the “whichever is higher” penalty logic that applies for all other businesses (i.e., those which are not SMEs).

What is the European Commission’s position?

The first proposed change is to include legal definitions of SME and small mid-cap enterprise (SMC) to the AI Act. These are:

• SME: an enterprise which employs fewer than 250 people and which has an annual turnover not exceeding 50 million EUR, and/or an annual balance sheet total not exceeding 43 million EUR.

• SMC: an enterprise which employs fewer than 750 people and which has an annual turnover not exceeding 150m EUR or an annual balance sheet total not exceeding 129m EUR.

The second and more significant proposed change is to extend the proportionate, inverse penalty regime for SMEs to SMCs also. This would significantly reduce the total potential penalty exposure of SMCs (i.e., many more companies) in certain circumstances. SMCs that are providers of high-risk AI systems would also be able to provide the required technical documentation in a simplified manner.

What is the Council’s position?

The Council has adopted the same position as the European Commission. This means that penalties for SMCs would also be capped at the lower amount, just as they are for SMEs today.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

AI risk assessments are one of the most important, challenging, and contentious activities in enterprise AI governance. Designing effective assessments from scratch is difficult, especially as the regulatory and technology landscape keeps changing. What your risk assessment consists of, and what type of AI-related initiatives are in scope of it, requires constant refinement and adaptation.

This week’s edition of Enterprise AI Governance spotlights 12 practical resources—from governments, regulators, standards bodies, and leading companies—that AI governance, compliance, and security teams can use to design, evaluate, and strengthen their AI risk assessment processes and frameworks.

If you haven’t done so already, consider signing up to pre-order my forthcoming book, Fundamentals of AI Governance. It’s a comprehensive, visual guide to enterprise AI governance implementation, with dedicated chapters on the EU AI Act, AI risks and mitigations, and agentic AI governance. It also includes my AI risk assessment template. Pre-order here for a 25% discount:

Secure your 25% discount

12 Practical Resources for AI Risk Assessments

AI risk assessment templates and checklists

1. AI Verify Testing Framework for Traditional and Generative AI (IMDA and Singapore AI Verify Foundation)

2. AI Impact Assessment Tool and Supporting Guidance (Australian Government)

3. Microsoft Responsible AI Impact Assessment Template (Microsoft)

4. ISO/IEC 42005 — AI System Impact Assessment (ISO/IEC Standard)

EU AI Act and GDPR-specific resources

5. Risk Management - EU AI Act Guidelines, Part 5 (The Spanish Agency for the Supervision of AI)

6. Conformity Assessments under the EU AI Act: A step-by step guide (Future of Privacy Forum and OneTrust)

7. Guidance for Risk Management of AI Systems (European Data Protection Supervisor)

AI security risks

8. OWASP Top 10:

   1. OWASP Top 10 for LLM Applications (OWASP)

   2. OWASP Top 10 for Agentic Applications (OWASP)

9. Google SAIF - Secure AI Framework (including SAIF - 2.0 Secure Agents)

For public authorities

10. AI Testing and Assurance Framework for Public Sector (UK Government)

11. Algorithmic Impact Assessment tool (Government of Canada)

12. Responsible AI Innovation in Law Enforcement: Risk Assessment Questionnaire (Interpol and United Nations)
    

AI Risk Assessment Templates and Checklists

1. AI Verify Testing Framework for Traditional and Generative AI

Publisher: Infocomm Media Development Authority (IMDA) and Singapore AI Verify Foundation

Date: 2025

Purpose: The AI Verify Testing Framework (AIVTF) is an extensive, practical resource that companies can use to assess and evaluate AI systems across the core dimensions of responsible AI. It outlines key steps that organisations should take for responsible AI system development and deployment.

Features: The AIVTF is structured around 11 key principles (e.g., transparency, explainability, safety, and accountability) and it follows an evaluation questionnare format. For each principle, there are several specific outcomes organisations should aim to achieve and a series of practical steps for achieving them. Each of these steps can be integrated into AI risk assessments and mitigation and control plans. For example, under the ‘safety’ principle, a key outcome to achieve is ‘carry out regular tests to evaluate for safety and possible harms (e.g., hallucination and general toxicity)’. This outcome is achieved by implementing the following (non-exhaustive) steps:

• Identify relevant and/or use-case appropriate benchmarks

• Run benchmarking and red teaming

• Testing to be performed at several checkpoints throughout the AI lifecycle

• Where applicable, share results with relevant stakeholders

Key takeaways: This 118-page document is one of the most comprehensive and useful AI governance resources—and it is openly available for organisations to use. AI governance practitioners can use it to design, implement, evaluate, and strengthen their AI risk assessment questionnaires and AI lifecycle control frameworks.

2. AI Impact Assessment Tool and Supporting Guidance

Publisher: Australian Government (Digital Transformation Agency)

Date: December 2025

Purpose: The AI Impact Assessment Tool is a comprehensive risk assessment template that is used by Australian Government departments. For certain AI use cases, it is mandatory for agencies to conduct AI impact assessments that align to this framework. This includes AI systems that the public will directly interact with or where the use of AI will materially influence administrative decisions that affect individuals.

Features: The AI Impact Assessment Tool provides a structured template for assessing and scoring the risk of AI use cases, as well as mapping impacted stakeholders It is aligned to Australia’s AI Ethics Principles, is supported by implementation guidance (covering each question), and contains the following sections:

• Basic information

• Purpose and expected benefits

• Inherent risk assessment (e.g., unfair discrimination, harm, privacy, security etc.)

• Fairness

• Reliability and safety

• Privacy protection and security

• Transparency and explainability

• Contestability

• Human centred-values

• Accountability

Key takeaways: Similar to Singapore, the Australian Government has also favoured soft law over AI-specific legislation, and has been proactive in publishing various guidelines and resources that organisations can use. This is a great example of a practical artefact that you can use to identify gaps and improvement opportunities for your internal AI risk assessment.

3. Microsoft Responsible AI Impact Assessment Template

Publisher: Microsoft

Date: June 2022

Purpose: Microsoft’s Responsible AI Impact Assessment Template provides a structured set of questions that AI governance teams can use to assess the impact and risks of an AI system. It is well-suited for higher-risk AI systems. The Template is accompanied by the Responsible AI Impact Assessment Guide. These documents were released externally by Microsoft to provide transparency regarding internal responsible AI practices. However, it is inevitable that these internal processes have changed significantly since 2022.

Features: The Responsible AI Impact Assessment Template consists of a set of technology-neutral risk assessment questions, grouped in the following sections:

• System information

• Intended uses

• Adverse impact

• Data requirements

• Summary of impact

Key takeaways: This resource is older than others on this list and is probably not sufficiently up-to-date to use as a standalone AI risk assessment framework in 2026. However, it contains useful guidance for AI governance practitioners and is aligned to Microsoft’s Responsible AI Standard. Furthermore, it is part of a broader collection of Responsible AI Tools and Practices released by Microsoft, which includes information about technical controls for data loss mitigation and AI content safety.

4. ISO/IEC 42005 — AI System Impact Assessment

Publisher: International Organisation for Standardisation (ISO) and International Electrotechnical Commission (IEC)

Date: May 2025

Purpose: ISO/IEC 42005 is an international standard that provides guidance for organisations conducting AI system impact assessments. The purpose of such assessments is to evaluate how AI may impact individuals, groups, or society throughout the AI system lifecycle.

Features: The standard outlines a structured process covering:

• Timing of the AI system impact assessment

• Scope

• Allocation of responsibilities

• Establishing thresholds for sensitive and restricted uses

• Impact scales

• Performing and analysing the AI system impact assessment

• Approval and ongoing monitoring

Annexes provide guidance on alignment with ISO/IEC 42001, the relationship with risk management under ISO/IEC 23894, and a taxonomy for analysing potential harms and benefits of AI systems.

Key takeaways: This standard is complementary to ISO/IEC 42001 — AI Management System and is part of the same series of standards. It is also the first international standard dedicated to AI system impact assessments and it contains holistic guidance about every aspect of how to design, operate, and manage such processes. It is therefore a valuable resource for organisations that are conducting AI assessments, although it is behind a paywall.

EU AI Act and GDPR resources

5. Risk Management - EU AI Act Guide, Part 5

Publisher: The Spanish Agency for the Supervision of AI (AESIA)

Date: December 2025

Purpose: This is one of 16 practical guides recently published by AESIA to support EU AI Act compliance for high-risk AI systems. These resource guides, which are available in English, can be used by providers and deployers of high-risk AI systems. This Guide (Part 5), focuses on the risk management obligation for providers, outlined in Article 9 of the EU AI Act.

Features: The Risk Management Guide provides information on how providers can fulfil their obligation to implement a continuous and iterative risk management system for high-risk AI systems. This includes, but is not limited to, risk identification and analysis, risk evaluation, and risk management, mitigation, and elimination measures. The Guide is accompanied by an Excel checklist template with an illustrative risk management process that providers can leverage for different use cases. The broader collection of 16 guides covers key obligations including conformity assessments, quality management, human oversight, data governance, transparency, and accuracy, robustness, and cybersecurity.

Key takeaways: This is the most comprehensive set of EU AI Act compliance guidance published by a national regulator to date. With harmonised standards still in development and the European Commission's guidelines on high-risk AI still being finalised, the AESIA guides serve as a valuable resource for organisations seeking practical, requirement-by-requirement implementation guidance. However, following such guidance does not guarantee compliance.

6. Conformity Assessments under the EU AI Act: A step-by step guide

Publisher: Future of Privacy Forum and OneTrust

Date: April 2025.

Purpose: This White Paper provides a step-by-step roadmap and detailed guidance for organisations to conduct and oversee conformity assessments under the EU AI Act. This is targeted primarily at providers of high-risk AI systems, as these organisations are obliged to ensure that the appropriate type of conformity assessment is performed, before their AI system is placed on the market or put into service.

Features: The guide includes flowcharts and checklists for determining whether an AI system qualifies as high-risk, what type of high-risk AI system it is, what type of conformity assessment is required, and who is responsible for performing the required conformity assessment. It covers both internal conformity assessments (i.e., self-assessment) and third-party conformity assessments (i.e., those performed by notified bodies).

Key takeaways: This resource is the most extensive and practical guidance available on conformity assessments under the EU AI Act. It is important to understand that the type of conformity assessment required—and who should perform it—depends on the type of high-risk AI system, as well as the availability of harmonised standards. See the flowchart below, which is provided in the guide, for more information (full credits to Future of Privacy Forum and OneTrust).

7. Guidance for Risk Management of AI Systems

Publisher: European Data Protection Supervisor (EDPS)

Date: November 2025

Purpose: This 55-page guidance document supports EU Institutions, Bodies, Offices and Agencies in identifying and mitigating data protection risks when developing and deploying AI systems. It is particularly relevant for scenarios where EU Institutions are data controllers. The EDPS is the data protection supervisory authority for the EU Institutions. This guidance is issued in this context of this EDPS role and not in the context of its role as an AI Act supervisory authority.

Features: The guidance outlines the most significant AI-related risks associated with each of the main data protection principles, including:

• Fairness

• Accuracy

• Data minimisation

• Security

It also outlines risks relating to data subject rights, including in incomplete rectification or erasure and the challenge of machine unlearning.

Key takeaways: This is one of the most useful resources for practitioners seeking to understand the data protection and privacy-related risks and challenges of AI development and deployment. Although it is primarily intended for EU institutions, it serves as a valuable reference for any organisation seeking a structured methodology for AI-related data protection risk assessment and management.

AI security risks

8. OWASP Top 10 (AI Threats and Mitigations)

• OWASP Top 10 for LLM Applications (OWASP)

• OWASP Top 10 for Agentic Applications (OWASP)

Publisher: Open Worldwide Application Security Project (OWASP)

Date: November 2024 (Top 10 for LLM Applications) and December 2025 (Top 10 for Agentic Applications)

Purpose: These are two open-source, community driven resources that identify the top 10 most critical security risks and threats, as well as potential mitigations, for LLM applications and agentic AI applications respectively. They were developed collaboratively by many security researchers, practitioners, and AI experts.

Features: The Top 10 for LLM Applications covers:

• Prompt injection

• Sensitive information disclosure

• Supply chain risks

• Data and model poisoning,

• Improper output handling

• Excessive agency

• System prompt leakage

• Vector and embedding weaknesses

• Misinformation

• Unbounded consumption

The Top 10 for Agentic Applications covers:

• Agent goal hijack

• Tool misuse and exploitation

• Identity and privilege abuse

• Agentic supply chain vulnerabilities

• Unexpected code execution

• Memory and context poisoning

• Insecure inter-agent communication

• Cascading failures

• Human-agent trust exploitation

• Rogue agents

Key takeaways: Both resources adopt a compelling format. The top ten risks are presented, with each accompanied by an explanation, examples, and technical mitigations organisations can implement. Therefore, these are essential references for any organisation deploying LLMs or agentic AI applications at scale. These are security-focused lists, not comprehensive AI governance frameworks. However, the security-related controls should inform the broader AI risk assessment and control frameworks.

9. Google SAIF - Secure AI Framework (including SAIF - 2.0 Secure Agents)

Publisher: Google

Date: Published in 2023 and periodically updated since then.

Purpose: Google’s Secure AI Framework (SAIF) is designed to enable organisations to manage the security-related risks of AI. It maps risks and controls across four layers of the AI system layers: data, infrastructure, model, and application.

Core features: SAIF is built around six core elements:

• Expand strong security foundations to the AI ecosystem

• Extend detection and response to bring AI into an organisation’s threat universe

• Automate defences to keep pace with existing and new threats

• Harmonise platform level controls to ensure consistent security

  across the organisation

• Adapt controls to adjust mitigations and create faster feedback loops for AI deployment

• Contextualise AI system risks in surrounding business processes

The Framework covers 15 AI security risk areas, including data poisoning, prompt injection, model exfiltration, model reverse engineering, and model evasion. For each risk, guidance is provided regarding how it emerges and is exposed in the AI lifecycle and what mitigations can be implemented.

Google recently published SAIF 2.0 Secure Agents, which focuses on agentic AI security and governance. Agent specific risk themes include rogue actions and sensitive data disclosure, with mitigations covering agent permissions, observability, and agent user control (e.g., human approval).

Key takeaways: SAIF is a a useful high-level framework for integrating AI-specific security assessments and controls into existing programmes. Organisations can use SAIF alongside the OWASP Top 10 lists to build a more complete picture of AI security risks and controls.

For public authorities

10. AI Testing and Assurance Framework for Public Sector (UK Government)

Publisher: UK Government, Cross-Government Testing Community

Date: Beta version published June 2025

Purpose: A structured and technically rigorous approach to testing, evaluating, and assuring AI systems deployed in the UK public sector, covering every phase of the AI development lifecycle, from planning to ongoing monitoring.

Core features: The framework identifies four core AI quality attributes, which should be the focus of testing and assurance activities:

1. Safety and Ethics

2. Openness & Trust

3. Performance & Resilience

4. User & Context Fit

It distinguishes between testing (producing evidence), evaluation (interpreting evidence in context), and assurance (providing confidence the system is ready). It includes a modular strategy for selecting testing activities based on AI system type, use case, and risk level. It aligns with the UK Government AI Playbook and supports proportionate and risk-based assurance of impactful AI systems.

Key takeaways: Although primarily for UK government, the testing-evaluation-assurance distinction is a useful framing that any organisation can adopt. This resource is part of the broader trend of the UK government producing practical, non-binding AI governance resources, rather than pursuing national AI legislation. Departments are encouraged to adapt it for their specific context and any organisation could to the same. It is currently in beta, openly available, and accepts community contributions as a living tool.

11. Algorithmic Impact Assessment tool

Publisher: Government of Canada

Date: Published in 2019 and periodically updated since then.

Purpose: The Algorithmic Impact Assessment (AIA) tool is a mandatory risk assessment tool supporting Canada's Directive on Automated Decision-Making. It is used by Canadian federal government departments to assess and evaluate automated decision systems, prior to use in specific contexts. It must be completed at the beginning of the design phase of a project and its outputs should guide the project thereafter.

Core features: The AIA can be accessed as an online questionnaire with 65 risk questions and 41 mitigation questions. It classifies systems used for automating of administrative decisions into four impact levels:

• Level I: Little to no impact

• Level II: Moderate impact

• Level III: High impact

• Level IV: Very high impact

Each impact level corresponds to different requirements for human oversight, transparency, peer review, and monitoring. Completed assessments must be made publicly available.

Key takeaways: The Directive on Automated Decision-Making that the AIA supports is one of the world's first AI-specific legally binding instruments. Although it applies solely to Canadian federal government use of automated decision systems, its open-access nature makes it a practical reference for any organisation designing AI risk assessments for high-risk use cases.

12. Responsible AI Innovation in Law Enforcement: Risk Assessment Questionnaire

Publisher: Interpol and United Nations Interregional Crime and Justice Research Institute (UNICRI)

Date: February 2024

Purpose: This is a practical tool for law enforcement agencies to evaluate the risks of AI system deployment to individuals and communities. It is part of a series of resources within the broader Toolkit for Responsible AI Innovation in Law Enforcement.

Core features: The questionnaire consists of 24 questions and aligns to five core principles:

• Lawfulness

• Minimisation of harm

• Human autonomy

• Fairness

• Good governance.

Key takeaways: This resource is designed to complement existing AI risk assessment artefacts and practices. Although designed for law enforcement entities, it provides a useful approach for structuring AI risk assessments in high-stakes contexts where AI systems can directly impact individuals’ fundamental rights.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

It is surprisingly easy for a deployer of an AI system to become the provider of a high-risk AI system under the EU AI Act. This article explains the three ways this can happen, why one of them is particularly difficult to detect and prevent, and what enterprises should do about it. In light of the widespread democratised access to general-purpose AI systems, this is one of the least discussed yet most consequential topics in enterprise AI governance today. The key takeaway is that when using AI systems for activities or use cases that overlap with the AI Act’s high-risk categories, it is advisable to use AI systems specifically designed, intended, and marketed for that purpose.

Disclaimer: This article is for informational purposes only and does not constitute legal advice. Organisations should seek professional legal counsel for guidance on their specific circumstances and obligations under the EU AI Act.

If you haven’t done so already, consider signing up to pre-order my forthcoming book, Fundamentals of AI Governance. It’s a comprehensive and practical guide to enterprise AI governance implementation, with a dedicated chapter on the EU AI Act. In the book, I break down every aspect of the law in 35+ visual cheat sheets. Pre-order at the link below for a 25% discount.

Pre-order my book

What are “providers” and “deployers”?

The concepts of “provider” and “deployer” are integral to the EU AI Act.

Simply put, providers are entities that develop AI systems—either for their own use or to make available for others to use—and deployers are entities that use AI systems developed and made available to them by providers. The legal definitions, supported by brief explanations, are provided below.

Provider: “a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge”.

An organisation can either be a provider of an AI system or a provider of a general-purpose AI (GPAI) model. However, they do not become a provider unless and until they place an AI system or GPAI model on the market or put an AI system into service. “Putting into service” means that the provider uses the AI system (e.g., for internal purposes) or directly supplies it to a deployer for first use.

Deployer: “a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity”.

Deployers are typically organisations that use AI systems developed and made available to them by providers. An organisation can only be a deployer of an AI system; it cannot be a deployer of a GPAI model, as this scenario is not possible under the AI Act.

Organisations routinely use GPAI models by integrating them into AI systems that they develop or by using externally provided AI systems powered by GPAI models (e.g., ChatGPT). In the former scenario, organisations typically become the “downstream provider” of an AI system. In the latter scenario, organisations typically become the deployer of an AI system. In neither scenario do they become the deployer of a GPAI model, despite the AI system being powered by a GPAI model. Indeed, the deployer never has any obligations which directly relate to the GPAI model itself. In such scenarios, deployer obligations only relate to the AI system which the GPAI model is part of.

However, an organisation can become the provider of a GPAI model by making significant modifications to an existing GPAI model (e.g., by amending a pre-trained external GPAI model via fine-tuning or similar techniques). This scenario is beyond the scope of this article and was covered extensively in a previous edition of Enterprise AI Governance.

Why does the provider and deployer distinction matter?

This article explains the scenarios in which a deployer of an AI system can become a provider of a high-risk AI system.

The key takeaway is that the obligations for providers and deployers are highly distinct in nature. The purpose of having these two separate roles is to appropriately and transparently allocate the key accountabilities, responsibilities, and liabilities to different entities. The allocation of these responsibilities directly relates to their respective relationship to the applicable AI system; it also has important implications for their relationship with each other.

Therefore, it is crucial for organisations to understand, as early as possible, whether they are a provider or a deployer (or both), and to take steps to adhere to the applicable obligations.

Furthermore, it is reasonable to argue that the obligations and requirements for providers of high-risk AI systems are more extensive and onerous than the corresponding deployer obligations. Therefore, not realising that you are a provider could pose significant regulatory and legal risk.

With respect to AI systems, the compliance obligations for providers and deployers only kick in for high-risk AI systems (see Article 6 and Annexes I and III) and transparency-requiring AI systems (see Article 50). An AI system can be high-risk, transparency-requiring, both, or neither. If it is neither, then it is not directly regulated by the AI Act (at least not with respect to the obligations for providers and deployers).

The precise compliance obligations and requirements that apply will always depend on the type of AI system. For example, there are baseline obligations and requirements for providers of high-risk AI systems covered by Annex I (Section A) and high-risk AI systems listed Annex III. However, there are also important differences depending on the type of high-risk AI system. For example, a third-party conformity assessment is required for some high-risk AI systems, whereas only an internal conformity assessment is required for others.

The same point can also be made for deployers. There are baseline obligations for deployers of high-risk AI systems, as well as obligations that only apply for deployers of specific types of high-risk AI systems (e.g., the obligation to conduct a fundamental rights impact assessment).

Although the scope of what is a high-risk AI system is relatively narrow—at least when considering the broad spectrum of enterprise AI use cases—many AI systems will be caught by the Article 50 transparency obligations. These obligations also differ, depending on both whether you are a provider or a deployer of a transparency-requiring AI system, as well as the exact type of AI system.

Although it is beyond the scope of this article to outline in detail the obligations for providers and deployers of AI systems, the key point is that they are highly distinct in nature and vary depending on the type of AI system.

How can deployers become providers of high-risk AI systems?

Despite these important differences, it is surprisingly easy for a deployer of an AI system to become the provider of a high-risk AI system. This is one of the least discussed yet most consequential topics in AI governance today.

Article 25(1) of the AI Act outlines the three main ways in which an entity that is a deployer of an AI system can become the provider of a high-risk AI system. In each of these three scenarios, the deployer would become the provider of a high-risk AI system and “therefore assume all the relevant obligations” (i.e., all the obligations for a provider of that type of high-risk AI system).

The 3 ways in which deployers can become providers of high-risk AI systems

1. The deployer puts its name or trademark on a high-risk AI system that has already been placed on the market or put into service.

In this scenario, the deployer becomes the provider of the high-risk AI system, unless the agreement with the original provider stipulates otherwise.

2. The deployer makes a ‘substantial modification’ to a high-risk AI system that has already been placed on the market or put into service, and it remains a high-risk AI system.

Understanding this scenario requires unpacking the meaning of “substantial modification”. This concept is defined as:

“a change to an AI system after its placing on the market or putting into service which is not foreseen or planned in the initial conformity assessment carried out by the provider and as a result of which the compliance of the AI system with the requirements set out in Chapter III, Section 2 is affected or results in a modification to the intended purpose for which the AI system has been assessed”.

Simply put, if the deployer changes a high-risk AI system in a manner that alters the intended purpose determined and communicated by the original provider, or in a manner that means it is no longer compliant with the applicable requirements for that type of high-risk AI system, then that deployer could become the provider of a new high-risk AI system.

3. The deployer modifies the ‘intended purpose’ of an AI system, including a general-purpose AI system, which has not been classified as high-risk and has already been placed on the market or put into service in such a way that the AI system becomes a high-risk AI system.

In this scenario, the deployer would have to modify the intended purpose of an AI system that is not high-risk in a manner that results in it becoming a high-risk AI system.

“Intended purpose” is defined as: “the use for which an AI system is intended by the provider, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation”.

However, it is important to note that the AI Act does not explain exactly how the concept of “modify” should be interpreted in this context. For example, we do not know exactly how much active intervention is required for a “modification” to occur or whether mere usage of an AI system could equate to modification of it.

Nonetheless, in this scenario, the deployer would also become the provider of a new high-risk AI system.

Indeed, in all three scenarios, the original provider would no longer be the provider of the new AI system, which itself would, as explained above, have a new provider (i.e., the organisation that went from being a deployer to being a provider).

Could you become the provider of a high-risk AI system by merely using a general-purpose AI system?

The caveat to the below analysis is that the European Commission was due to publish guidelines on high-risk AI systems in February 2026, but these guidelines are delayed. The Commission is also due to publish guidelines on the concept of “substantial modification”. At present, we have little indication as to how this aspect of the AI Act will be interpreted and enforced by the regulators. Once published, these guidelines will bring greater clarity to this knotty topic. This analysis is therefore based primarily on interpretation of the text in the AI Act itself.

Scenario 3 is arguably the most important for organisations to consider, as it is ostensibly surprisingly easy to do and could be quite difficult to detect and prevent.

With Scenarios 1 and 2, the organisation is perhaps more likely to know they are doing something which could make them a provider of a high-risk AI system. However, with Scenario 3, it is easy to anticipate ways in which this could happen inadvertently and/or without the organisation realising or appreciating.

In most enterprises, employees have democratised access to general-purpose AI systems (GPAI systems) that are usually provided by external organisations. This ranges from mainstream generative AI chatbots to no- and low-code AI agent builder platforms. Such GPAI systems can be used for all aspects of enterprise work. And across large swathes of the corporate world, employees are being strongly encouraged to adopt AI as much as possible.

A key point to note is that these GPAI systems increasingly enable customisation and modification. For example, even non-technical users can connect knowledge sources, set system prompts, adjust ‘temperature’, or build bespoke features. Therefore, although Scenario 3 requires the deployer to “modify” the AI system’s intended purpose, it does not require a “substantial modification”, and there are various more subtle types of modification which are easy to implement and could potentially be captured by this somewhat nebulous term.

In many cases, the ‘intended purpose’ of these GPAI systems will not cover the AI Act’s high-risk AI system categories. Sometimes, a GPAI system provider may even explicitly prohibit certain categories of high-risk AI usage.

This is not hypothetical. Below are some brief excerpts from GPAI provider terms of use:

Gemini API Additional Terms of Service (Google): “You may not use the Services in clinical practice, to provide medical advice, or in any manner that is overseen by or requires clearance or approval from a medical device regulatory agency”.

Microsoft Enterprise AI Services Code of Conduct: “Customers must not use the services [...] to make decisions or take actions without appropriate human oversight as part of an application that may have a consequential impact on any individual’s legal position, financial position, life opportunities, employment opportunities, or human rights, or may result in physical or psychological harm to an individual”.

It is also worth noting that where a provider has clearly specified that its AI system should not be used as a (particular type of) high-risk AI system—as the above terms indicate—Article 25(2) of the AI Act clarifies the original provider may have no obligation to closely cooperate with, or provide documentation to, the deployer that, in doing so, becomes a provider.

This means that if an employee or team in your organisation modifies—or potentially even just uses—a GPAI system in a manner that is both a) high-risk under the AI Act and b) not aligned with that GPAI system’s intended purpose, then your organisation is at risk of becoming both a deployer and a provider of a high-risk AI system—potentially without even realising it. This could happen inadvertently or unintentionally.

Consider the following example. A university admissions team uses a mainstream generative AI chatbot to upload, evaluate, and score submissions from prospective students. Depending on how regulators interpret the AI Act, this in itself could be considered “modification” and the university could become a provider of a high-risk AI system without even realising it.

Alternatively, perhaps the university creates relatively detailed system prompts and knowledge graphs, which it leverages alongside the GPAI system to optimise performance for this use case. In doing so, this could be interpreted as “modifying” the GPAI system, as this entails active intervention. However, the university may still not realise that in doing so it has become a provider.

If the chatbot (i.e., the GPAI system) was not intended to be used (by deployers) in this way, and this use is directly informing and impacting the admissions and decision-making process, then the university could potentially become both the provider and deployer of a high-risk AI system—even if its interventions or changes were minimal. As explained above, this entails significant and novel compliance obligations that require planning, investment, and capacity-building.

The key takeaway is that when modifying, or even using, an AI system in a way that is directly related to, or overlaps with, one of the AI Act’s high-risk categories, it is advisable to use an AI system specifically designed, intended, and marketed for that purpose.

This is important for both regulatory compliance and legal reasons, as well as for performance-related reasons. Such purpose-built and domain-specific AI systems—even if powered by GPAI models—are likely to be better suited for the use case, and using them will also provide you with greater legal clarity on your responsibilities and liabilities (as a deployer).

The logic underpinning this is that the EU deems it preferable for bespoke and tailor-made AI systems—designed, developed, and deployed according to required specifications—to be used for high-risk AI activities.

To put it even more simply, make sure your entire workforce is aware of what is a high-risk AI system under the AI Act, and implement both training and technical controls that decrease the likelihood of GPAI systems being used for high-risk AI use cases, apart from in scenarios where the provider confirms that this is part of the GPAI system’s intended purpose.

If you are a deployer that does not want to assume the legal responsibilities of a provider, incorporating such controls into your AI governance framework is essential.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance. Welcome to the 40th edition of this newsletter! In this edition, I dissect the ‘AI policies versus guardrails’ debate and argue that pitting the two against each other is a false dichotomy. For AI governance to be effective, robust policies and technical guardrails need to work together and evolve in harmony.

If you haven’t done so already, consider signing up to pre-order my forthcoming book, Fundamentals of AI Governance. It’s a comprehensive, visual guide to enterprise AI governance implementation, with dedicated chapters on the EU AI Act, AI risks and mitigations, and agentic AI governance. Pre-order here for a 25% discount:

Pre-order my book

The field of AI governance is evolving

When I first noticed the flurry of new AI governance roles appearing in the corporate world in 2022, I immediately took interest. My fascination with the field of AI began during my postgraduate studies in Philosophy and Public Policy, where I focused on philosophy of mind, consciousness, and the ethical and societal implications of AI and emerging technologies. Since then, although I continued to cultivate this (formerly esoteric) set of interests, my career was primarily focused on privacy and data, as this is where much of the tech policy action was.

Fast forward to today and AI governance has quickly become a standalone profession, growing in both size and significance, and encompassing many distinct sub-fields and career pathways. Although the topic of AI ethics and governance is nothing new, it was only in the early 2020s that it evolved from being a relatively niche research and policy issue, to a sizeable professional field and capability across the corporate world—akin to established fields like privacy, cyber security, and data governance.

The more mainstream and important the field of AI governance becomes, the more divergent perspectives there are about what good looks like. And now more than ever, there is a robust and healthy debate about the best way organisations should approach AI governance.

The ‘policies versus guardrails’ debate

A major point of contention in 2026 is what I call the ‘AI policies versus guardrails’ debate. This short essay outlines what this debate is, why I think pitting policies versus technical guardrails is a false dichotomy, and that traditional governance mechanisms—like policies and training—and engineered guardrails working in harmony is the only way to effectively govern AI.

My core argument is that the organisational culture and enabling environment fostered by AI governance leaders is just as important as the safety and security guardrails engineered into the AI systems and models that are the target of governance.

Having led enterprise AI governance at AstraZeneca for over three years now, whilst also delivering countless AI governance trainings for organisations and professional cohorts worldwide, I have had a front-row seat to watch the profession of AI governance mature into what it is today. In my view, although the current debates and challenges make perfect sense, it would be damaging for AI governance to split into fractured silos.

Over the past few years, there has undoubtedly been more focus across the enterprise AI governance world on codifying responsible AI principles, implementing policy and risk management frameworks, and working towards AI regulatory compliance, than there has been on engineering guardrails into AI systems, embedding policy-as-code, and building AI governance by design.

aI’ll be the first to admit that much of my work has focused on the former rather than the latter. However, as I have written about at length, agentic AI takes the human out of the loop and poses fundamental challenges to legacy AI governance frameworks, which were not designed with such advanced and increasingly autonomous AI capabilities in mind.

So, what are the contours of the ‘AI policies versus guardrails’ debate? And what is a viable path forward?

On one hand, many are increasingly dismissive about the policy and compliance aspects of AI governance work. I frequently read commentary along the lines of “AI governance is just compliance theatre“ or “your policies are just PDFs and not real governance“. The proposed solution invariably revolves around technological, engineering, and architectural controls and guardrails.

On the other hand, there are others who focus solely on AI regulatory compliance (e.g., EU AI Act) and standardisation (e.g., ISO 42001), and enabling mechanisms such as policy and process documentation, risk assessments, governance committees, and ethical principles—and providing training on all of the above.

Why policies versus guardrails is a false dichotomy

Although both positions are well-intentioned, framing AI governance as a choice between policies and processes on one hand, and technical guardrails on the other, is a false dichotomy. In isolation, both are necessary but neither are sufficient for effective AI governance. It’s when you intentionally and seamlessly fuse both worlds together that AI governance starts to get real.

There is undoubtedly plenty of compliance theatre and performative governance to go around. Too often, for example, AI policy and compliance work does not adequately address how requirements can actually be implemented and operationalised in practice by technical teams working at the AI frontier, or where flexibility and proportionality should be afforded. And we are all drowning in a never-ending stream of AI frameworks and standards—many of which have high degrees of overlap.

However, this policy, compliance, and risk management work—and the AI literacy initiatives that support it—is valuable because it shapes how people think and behave, directly influencing how problems are solved, risks and trade-offs are weighed, and decisions are made.

Policies have an important signalling effect. Risk assessment processes and oversight and accountability structures moderate and temper risky behaviour, giving decision-makers pause for thought in high-stakes scenarios, and ensuring the buck stops with a specific, senior individual. And AI literacy and training promotes understanding of the limitations and risks of AI, which is crucial for ensuring responsible use of the technology.

Ultimately, AI governance is change management.

Rather than thinking of it solely as a control framework, think of it as a set of activities designed to shape and steer how people think and act. In this respect, the broader objective of AI governance is to foster an organisational culture and workforce that genuinely understands and prioritises responsible AI.

However, if AI governance goes no further than policies, frameworks, and training, its impact will remain limited. Governance by PDF is a genuine problem.

Instead, as a profession, we should strive for AI governance by design and by default. Technical guardrails—integrated into the AI models, systems, and platforms being developed and deployed—are integral for ensuring AI governance is automated, enforceable, and scalable.

For example, we will not meaningfully govern agentic AI without technical restrictions on the data agents can access, the tools they can use, and the actions they can execute. Runtime guardrails that keep semi-autonomous agentic AI systems in check and bound their autonomy are essential, especially as the human is taken out of the loop.

More broadly, although the days of reviewing and approving every AI-generated output are ending, agentic AI deployment and governance must be underpinned by orchestration and observability, facilitated at the platform level. This should entail technical assurance mechanisms and automated monitoring and flagging of deviations, drift, and incidents.

Simply put, as well as restricting and constraining what AI agents can do via policy-as-code mechanism, we need to be able to continuously monitor and track how they are behaving. In other words, we will be unable to govern agentic AI without AI.

Another instructive example is how to combat the prevalence and risk of shadow AI use. Merely publishing a strict policy and delivering training that tells people not to use unapproved AI tools is insufficient. Technical controls are required to mitigate the risk of data loss and flag suspicious activity. Moreover, employees need to have access to best-in-class AI capabilities internally, to reduce incentives to use unapproved public AI tools.

Nonetheless, dismissing AI policy-focused work as “compliance theatre” fails to acknowledge that technical guardrails depend on codified policy. It’s called “policy-as-code” for a reason. Technical guardrails have to be based on agreed policies and standards—rooted in organisational values, culture, business strategy, and risk appetite. Simply put, you need to know what guardrails to prioritise and why, before you build them.

This means that, sequentially, it makes sense that, historically, the AI governance profession has predominantly focused on codification of policies and standards. However, for the long-term impact of this to be meaningful, urgent focus is now required on how to operationalise and engineer these policies by design. These two things can be true simultaneously.

Consider this analogy. You don’t have the right to live in your property because of technical security systems and features (e.g., locks, cameras, and alarms). You have the right to live there because a piece of paper (i.e., your mortgage or rental agreement), backed up by a law, backed up by a fundamental principle, codified in a constitution says that you do. However, technical security systems play a crucial role in ensuring that you are safe in your home and that your right to live in peace and security is enforced and protected. And the physical safety of the property itself depends on building codes and standards—policies that directly influence the property’s structure and design.

For the most part, the interplay of these elements (norms, laws, policies, technical controls, safety features etc.) is well understood. Do you think of your mortgage agreement as mere “compliance theatre” or “just a PDF”?

AI governance is no different. Policies matter. Compliance matters. Risk assessments and committee reviews matter. But ensuring the key requirements from your policies and standards are engineered into your production AI models, systems, and platforms also matters—especially as AI becomes more advanced and harder for humans to control.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

This week’s newsletter presents the definitive resource guide to agentic AI governance. This article (best viewed on browser) collates 50+ authoritative resources on agentic AI governance—mostly published in the past 18 months—grouped into 8 categories:

✅ Agentic AI: technology foundations and capabilities
✅ How enterprises are developing and deploying AI agents
✅ Agentic AI risks and challenges
✅ Risk mitigation strategies (security and safety)
✅ Agentic AI evaluations and observability
✅ Human oversight and accountability
✅ Agentic AI governance frameworks
✅ How EU law governs agentic AI: EU AI Act and GDPR

Whether you are working on the frontline of enterprise AI governance, advising clients in a legal or consulting capacity, or building, deploying, and using agentic AI systems, this resource guide enables you to keep track of and understand this rapidly evolving field.

There has been a brief hiatus of articles on Enterprise AI Governance, as I have been diving into this literature myself.

Agentic AI governance is one of the most important topics for practitioners this year, which is why I have taken some time out from posting to research and write a new long-read on Agentic AI Governance in 2026: A Guide for Practitioners. This free guide will soon be published on Enterprise AI Governance and shared with all subscribers.

In the meantime, if you haven’t done so already, consider pre-ordering my forthcoming book, Fundamentals of AI Governance. It features a dedicated chapter on Agentic AI Governance, with actionable guidance and risk assessment templates for enterprise practitioners. Pre-order here for a 25% discount:

Pre-order my book

The Ultimate Agentic AI Governance Resource Guide

50+ resources to master agentic AI governance

1) Agentic AI: technology foundations and capabilities

• The 2026 Guide to AI Agents - IBM (2026)

• A practical guide to building agents - OpenAI (2025)

• Building Effective AI Agents - Anthropic (2024)

• Agents - Chip Huyen (2025)

• Delegation Nation: Advanced AI Assistants and why they matter - Harry Farmer and Julia Smakman, Ada Lovelace Institute (2025)

• Large Language Model Agents: A Comprehensive Survey on Architectures, Capabilities, and Applications - Yiming Lei et al., (2025)

• ReAct: Synergizing Reasoning and Acting in Language Models - Shunyu Yao et al., (2023)

• Toolformer: Language Models Can Teach Themselves to Use Tools - Timo Schick et al., (2023)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - Jason Wei et al., (2022)

• What is the Model Context Protocol (MCP)? (2025)

• What is the MCP and how does it work? - Google (2025)

• Announcing the Agent2Agent Protocol (A2A): A new era of Agent Interoperability - Google (2025)

2) How enterprises are developing and deploying AI agents

• The 2026 State of AI Agents Report: How enterprises are building and deploying AI in production - Anthropic (2026)

• 2026 Agentic Coding Trends Report - Anthropic (2026)

• Measuring Agents in Production - Melissa Z. Pan et al. (2025)

• State of Agent Engineering - LangChain (2026)

• The state of AI in 2025: Agents, innovation, and transformation - Alex Singla et al., McKinsey QuantumBlack (2025)

• Frontier AI Trends Report - UK AI Security Institute (2025)

3) Agentic AI risks and challenges

• OWASP Top 10 for Agentic Applications for 2026 - OWASP (2025)

• Agentic AI - Threats and Mitigations - OWASP (2025)

• Multi-Agentic system Threat Modelling Guide v1.0 - OWASP (2025)

• Initial reflections on agentic AI governance - Oliver Patel, Enterprise AI Governance (2025)

• Agentic AI risks and challenges enterprises must tackle - Domino Data Lab (2025)

4) Risk mitigation strategies (security and safety)

• Practices for Governing Agentic AI Systems - Yonadav Shavit et al., OpenAI (2023)

• Infrastructure for AI Agents - Alan Chan et al., Centre for the Governance of AI (2025)

• AI Agents: Governing Autonomy in the Digital Age - Joe Kwon, Center for AI Policy (2025)

• Securing Agentic AI: An Addendum to the Guidelines and Companion Guide on Securing AI Systems - Cyber Security Agency of Singapore (2025)

• Deploying agentic AI with safety and security: A playbook for technology leaders - McKinsey (2025)

• Securing Agentic AI: A Discussion Paper - Cyber Security Agency of Singapore and Far.AI (2025)

• Google’s Approach for Secure AI Agents - Christoph Kern and Kara Olive, Google (2025)

• Agents Rule of Two: A Practical Approach to AI Agent Security - Meta (2025)

  Thanks for reading Enterprise AI Governance! Subscribe for free to receive new posts and support my work.

5) Agentic AI evaluations and observability

• Strengthening AI Agent Hijacking Evaluations - U.S. AI Safety Institute (2025)

• How to Evaluate Control Measures for LLM Agents? A Trajectory from Today to Superintelligence - Tomek Korbak et al., UK AI Security Institute (2025)

• Demystifying evals for AI agents - Anthropic (2026)

• Prioritizing Real-Time Failure Detection in AI Agents - Madhulika Srikumar et al., Partnership on AI (2025)

• Agent Factory: Top 5 agent observability best practices for reliable AI - Yina Arenas, Microsoft (2025)

6) Human oversight and accountability

• Governing AI Agents - Noam Kolt (2025)

• The Principal-Agent Alignment Problem in AI - Dylan Hadfield-Menell, UC Berkley (2021)

• AI and Human Oversight: A Risk-Based Framework for Alignment - Laxmiraju Kandikatla and Branislav Radeljic (2025)

• The dilemmas of delegation: an analysis of policy challenges posed by Advanced AI Assistants and natural-language AI agents - Harry Farmer, Ada Lovelace Institute (2025)

• Fully Autonomous AI Agents Should Not be Developed - Margaret Mitchell et al., Hugging Face (2025)

• Human-in-the-Loop for AI Agents: Best Practices, Frameworks, Use Cases, and Demo - Gabriel L. Manor, Permit.io (2025)

7) Agentic AI governance frameworks

• Model AI Governance Framework for Agentic AI - Singapore Infocomm Media Development Authority (2026)

• AI Agents in Action: Foundations for Evaluation and Governance - World Economic Forum (2025)

• AI Agent Governance: A Field Guide - Jam Kraprayoon, Institute for AI Policy and Strategy (2025)

• Characterising AI Agents for Alignment and Governance - Atoosa Kasirzadeh & Iason Gabriel (2025)

• The State of Agentic AI Security and Governance - OWASP (2025)

• The regulation of delegation: Are AI advisers, agents and companions regulated in the UK? - Julia Smakman, Ada Lovelace Institute (2025)

• Effective legal protections from harms caused by advanced AI assistants - Lucie Audibert & Alex Lawrence-Archer, AWO Agency (2025)

8) How EU law governs agentic AI: EU AI Act and GDPR

• Ahead of the Curve: Governing AI Agents Under the EU AI Act - Amin Oueslati and Robin Staes-Polet (2025)

• Agentic Tool Sovereignty - Lloyd Jones, European Law Blog (2025)

• Engineering GDPR compliance in the age of agentic AI - Keivan Navaie, IAPP (2025)

• Minding Mindful Machines: AI Agents and Data Protection Considerations - Daniel Berrick, Future of Privacy Forum (2025)

  ---

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

Emotion recognition is one of the least understood aspects of the EU AI Act. The prevalence of techniques like sentiment analysis across many industries, combined with the fact that AI-enabled emotion recognition is strictly prohibited in certain EU contexts whilst lawful elsewhere, makes it essential for enterprises to understand the nuances of this prohibition. This article outlines what is prohibited, what is high-risk, and what is not in scope—drawing on the AI Act and the European Commission's guidelines published in February 2025.

This week’s newsletter covers:

✅ Understanding ‘emotion recognition’ under the AI Act
✅ Emotion recognition: what is prohibited?
✅ Is it prohibited? 4-step compliance checklist visual (free download)
✅ Deep-dive on the 4 steps
✅ Emotion recognition: what is high-risk?
✅ Emotion recognition: what is not in scope?

Sign up to secure a 25% discount for my forthcoming book, Fundamentals of AI Governance (2026). It provides a detailed, comprehensive, and visual overview of the EU AI Act, including dozens of flowcharts, checklists, and practical case studies.

Secure a 25% discount

Understanding ‘emotion recognition’ under the AI Act

Article 5 of the EU AI Act outlines 8 distinct prohibited AI practices. One of these prohibited AI practices concerns ‘emotion recognition’. Specifically, Article 5(1)(f) prohibits:

“the placing on the market, the putting into service for this specific purpose, or the use of AI systems to infer emotions of a natural person in the areas of workplace and education institutions, except where the use of the AI system is intended to be put in place or into the market for medical or safety reasons”.

The provisions relating to prohibited AI have been applicable since February 2025. This means that since then, it has been prohibited, under EU law, to use AI to infer emotions in the workplace or educational institutions. Regulatory authorities are empowered to investigate and pursue enforcement action in this area. However, we have not yet seen any major cases or actions.

Breaching the provisions on prohibited AI carries a maximum potential fine for companies of up to 7% of global annual turnover. This is the largest potential fine prescribed by the AI Act and greater than any potential fine under the GDPR.

Also, it is worth noting that the European Commission proposals to amend and simplify the AI Act, published in November 2025 and covered extensively on this newsletter, did not propose any changes directly impacting the prohibited AI practices or emotion recognition. Therefore, we can expect relative stability for this area of AI Act compliance.

Considering this, it is important for enterprises to understand exactly what is and what is not prohibited under the AI Act, to facilitate compliant practices and risk-based decision-making.

The prevalence of techniques like sentiment analysis across many industries, and the fact that AI-enabled emotion recognition is strictly prohibited in the EU whilst lawful in other jurisdictions, reinforces the importance of understanding the nuances of emotion recognition and the AI Act.

The analysis that follows outlines what is prohibited, what is high-risk, and what is not in scope.

Emotion recognition: what is prohibited?

For the use of an AI system for emotion recognition to be prohibited, all four of these conditions must apply simultaneously:

1. The AI system is in scope of the EU AI Act.

2. The AI system processes and evaluates biometric data.

3. The AI system infers emotions.

4. The AI system is used in the workplace or an educational institution, without a valid exception for medical or safety reasons.

If one or more of these conditions does not apply, then it is most likely not a prohibited AI practice. The only caveat is unless that same AI system is somehow used for another prohibited AI practice (e.g., materially distorting behaviour and causing harm via deceptive, subliminal or manipulative techniques).

Each of these four conditions is equally important and warrant further explanation and analysis.

Check out my visual, 4-step check, which is elaborated on in the analysis below

1. The AI system is in scope of the EU AI Act.

For an AI system to be in scope of the AI Act, it must be placed on the market, put into service, or used in the EU. Also, if the AI system outputs (e.g., recommendations, predictions, insights etc.) are used in the EU, then the AI system is in scope of the AI Act.

Crucially, this means that even if an AI system is not placed on the market, put into service, or used in the EU, it would still be in scope of the AI Act if the outputs of the AI system were used in the EU. This could happen, for example, by sharing AI system outputs to a team based in the EU, for processing, analysis, and decision-making.

The prohibited AI practices apply to all actors under the AI Act. This means that a provider could breach the law by placing a prohibited AI system on the market and a deployer could breach the law by using an AI system in a prohibited manner.

Therefore, the notion of scope (in this context) refers to both territorial scope, as well as what is done (if anything) with the AI system and/or its outputs.

Simply put, if an AI system is developed, made available, and used outside of the EU, and the outputs of that AI system are not transferred to or used in the EU, then it is not in scope of the AI Act and thus cannot be prohibited.

2. The AI system processes and evaluates biometric data.

Article 3(39) defines “emotion recognition system” as:

“an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data”

This means that emotion recognition—as defined by the AI Act— necessarily entails the processing and evaluation of biometric data. If an AI system is not processing biometric data and inferring emotions on that basis, then it is not an emotion recognition system.

Interestingly, some confusion has been caused by the fact that the text of the prohibition itself in Article 5(1)(f) (see above) does not refer to ‘emotion recognition system’ or ‘biometric data’. It merely prohibits the use of AI to “infer emotions [...] in the areas of workplace and education institutions”.

However, in its guidelines, the European Commission acknowledged this discrepancy and clarified that, for consistency reasons, the prohibition should be linked to the legal definition and ‘emotion recognition system’ and thereby limited to “inferences based on a person’s biometric data”. Furthermore, Recital 44 of the AI Act refers to “serious concerns” about using AI systems to infer emotions, due to “limited reliability”, specifically noting that:

“AI systems identifying or inferring emotions or intentions of natural persons on the basis of their biometric data may lead to discriminatory outcomes and can be intrusive to the rights and freedoms of the concerned persons”.

Therefore, if the AI system does not process and evaluate biometric data, and infer emotions on that basis, then it is not an emotion recognition system and is therefore not prohibited.

The notion of “biometric data” is defined in Article 3(34) of the AI Act:

“personal data resulting from specific technical processing relating to the physical, physiological or behavioural characteristics of a natural person, such as facial images or dactyloscopic data”.

This could include physiological biometric data such as fingerprints, iris patterns, and facial structure, or behavioural biometric data such as facial expressions, body language, or tone and pace of voice.

An important caveat to this section’s analysis is that the European Commission guidelines are not legally binding and it is theoretically possible that a member state regulator or a court could interpret the AI Act in a different way. However, considering the legal definition of ‘emotion recognition system’, the wording of Recital 44, and the European Commission guidelines, this would be somewhat surprising.


3. The AI system infers emotions or intentions.

Just because an AI system is evaluating biometric data does not necessarily mean it is inferring emotions. At the risk of stating the obvious, the AI system must infer emotions to be prohibited.

The AI Act does not provide a definition of “emotion”. However, Recital 18 provides a non-exhaustive list of emotions and intentions, for illustrative purposes:

• happiness, sadness, anger, surprise, disgust, embarrassment, excitement, shame, contempt, satisfaction, and amusement.

The same recital also clarifies that “physical states, such as pain or fatigue” should not be considered as emotions or intentions. Also, the mere detection of physical expressions (e.g., whether someone is smiling) or presence (e.g., whether someone is in a meeting) does not constitute emotion inference.

The example provided in the AI Act is that using AI to detect the fatigue of professional pilots or drivers, for the purpose of preventing accidents, would not be prohibited.

However, the European Commission guidelines advise that the concept of emotions and intentions should be “understood in a wide sense and not interpreted restrictively”. Therefore, organisations should err on the side of caution when determining whether something is an emotion or not—especially if it is borderline.

Finally, given that AI systems that infer the intentions of individuals are also defined as emotion recognition systems, the European Commission guidelines clarifies that inferring intentions is also in scope of the prohibition.


4. The AI system is used in the workplace or an educational institution, without a valid exception.

Finally, the scope of the prohibition is limited to AI systems used in the workplace or an educational institution.

But not all uses of emotion recognition AI systems in the workplace and education are outright prohibited. The AI Act narrowly exempts AI systems used for inferring emotions for medical and safety purposes from the prohibition.

The European Commission guidelines clarify that the “notion of ‘workplace’ should be interpreted broadly” and includes “any setting where work is performed”. It also covers recruitment and hiring processes and therefore not only employees.

Interestingly, the guidelines provide these two use cases as examples of AI practices that are not prohibited:

• “Using voice recognition systems by a call centre to track their customers emotions, such as anger or impatience, is not prohibited by Article 5(1)(f) AI Act (for example to help the employees cope with certain angry customers)”.

• If only deployed for personal training purposes, emotion recognition systems are allowed if the results are not shared with HR responsible persons and cannot impact the assessment, promotion etc. of the person trained, provided that the prohibition is not circumvented and the use of the emotion recognition system does not have any impact on the work relationship.

Although it feels counterintuitive, the thinking here is probably that such activities and usage patterns are not relevant for the power imbalances and asymmetries in workplace contexts, such as between employers and employees, self-employed workers, and job candidates.

Crucially, there are two exceptions to the prohibition of emotion recognition in the workplace and educational institutions:

• AI systems used for medical reasons.

• AI systems used for safety reasons.

This means that, if the use of the AI system for emotion recognition in a workplace context is for medical or safety reasons, then it may be permissible under the AI Act. However, these exceptions should be interpreted narrowly and there is a high bar to reach them.

With respect to medical reasons, the guidelines clarify that this would include approved medical devices, rather than the use of AI to analyse wellbeing more broadly.

With respect to “safety reasons”, the guidelines clarify that this should be interpreted as only applying to protection of life and health, not protection against other risks, like theft or fraud.

Emotion recognition: what is high-risk?

First, let’s recap on what makes an emotion recognition system prohibited:

1. The AI system is in scope of the EU AI Act.

2. The AI system processes and evaluates biometric data.

3. The AI system infers emotions.

4. The AI system is used in the workplace or an educational institution, without a valid exception for medical or safety reasons.

As explained above, if all four of these conditions apply simultaneously, then the AI system is most likely prohibited under the EU AI Act. However, if only conditions 1, 2, and 3 apply, then the AI system is most likely a high-risk AI system under the AI Act, rather than a prohibited AI practice.

This is because 1) it is in scope of the AI Act, 2) it evaluates biometric data, 3) it infers emotions, but 4) it is not used in a workplace or educational institution, or it is but there is a valid exception in place.

Simply put, if an AI system is used for emotion recognition in a manner that is not prohibited—such as in the workplace and educational institutions for medical or safety reasons, or outside of the workplace and educational institutions altogether—then it would be a high-risk AI system.

Or to put it even more simply, if an AI system used for emotion recognition is not prohibited, it is a high-risk AI system.

For this reason, Annex III(1)(c) of the AI Act includes “AI systems intended to be used for emotion recognition” in the list of high-risk AI systems.

In such scenarios, providers must adhere to the obligations and requirements for the high-risk AI system intended to be used for emotion recognition and deployers must also adhere to the applicable obligations for high-risk AI systems.

Finally, emotion recognition systems are also ‘transparency-requiring’. Article 50(3) of the AI Act stipulates that deployers of emotion recognition systems must inform impacted individuals about the use of the AI system. The only exception to this is for emotion recognition systems used in certain law enforcement contexts.

Emotion recognition: what is not in scope?

Inferring emotion or sentiment merely on the basis of text (i.e., transcript data of what someone said) is not a prohibited AI practice. This was confirmed by the European Commission in its guidelines, which stated:

“An AI system inferring emotions from written text (content/sentiment analyses) to define the style or the tone of a certain article is not based on biometric data and therefore does not fall within the scope of the prohibition.”

Therefore, sentiment analysis, based purely on text transcripts, is not emotion recognition as it is not based on evaluation of biometric data. To further mitigate risk in such scenarios, it is prudent to ensure that text-based transcript data is not enriched with information describing an individual’s behaviour, physical appearance, or physiology.

However, just because text-based sentiment analysis is not emotion recognition and thus not prohibited in this context, does not mean it is always out of scope of the AI Act. It is theoretically possible for sentiment analysis to be part of a high-risk AI system or an alternative prohibited AI practice. For example, if sentiment analysis is embedded within a recruitment screening tool that evaluates candidates’ written responses, the overall AI system would likely be classified as high-risk.

Key takeaways for enterprises

Interpret broadly at this stage. The law is new, enforcement is yet to materialise, and regulatory expectations are still forming. Until we see how regulators and courts apply these provisions in practice, a cautious and expansive interpretation is prudent.
For example, the European Commission guidelines advise that emotions and intentions should be “understood in a wide sense and not interpreted restrictively”. If an AI system evaluates traits closely linked to emotions, treat it as potentially in scope, or at the very least implement additional guardrails.

Determine your global compliance strategy. Will you prohibit AI-enabled emotion recognition activities globally, or only in the EU? A global policy offers simplicity, standardisation, and a higher bar, whereas jurisdiction-specific policies potentially offer more flexibility and agility, albeit with greater compliance spill-over risk. If you do follow a jurisdiction-specific approach, robust controls and guardrails are required to ensure that any AI use cases that would be prohibited in the EU do not inadvertently impact EU-based individuals and that the corresponding AI system outputs are not shared with, or used by, anyone in the EU.

Carefully assess AI applications that evaluate individuals. Any AI system that analyses individuals (e.g., employees or students)—and in particular that evaluates their performance or behaviour in interactions, role-plays, meetings, or interviews—warrants in-depth AI governance review.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

Happy new year! I hope that everyone who celebrates enjoyed a peaceful, relaxing, and pleasant festive season. 2026 is inevitably going to be a busy and intense year for AI governance professionals. But before looking ahead, we are going to look back on 2025, which was a landmark year for our field.

This week’s newsletter provides a comprehensive review of AI law, policy, and governance in 2025. It features:

✅ 8 key themes that defined 2025
✅ Visual timeline: AI Governance in 2025 (free pdf download)
✅ The ultimate AI governance timeline: 70+ key milestones in 2025 (all hyperlinks provided)

Note: due to the length of this article, it is best viewed on your browser or the Substack app.

Sign up to secure a 25% discount for my forthcoming book, Fundamentals of AI Governance (2026). It provides a detailed, comprehensive, and visual overview of Global AI Law and Policy, including U.S., China, and EU comparison charts, as well as practical strategies for cross-jurisdictional compliance.

Pre-order the book

8 key themes which defined 2025

1. EU AI Act compliance becomes a reality. 2025 was the year that EU AI Act compliance became a reality for many organisations worldwide. The first set of provisions, on prohibited AI practices and AI literacy, became applicable on 2 February 2025. And on 2 August 2025, the obligations for providers of general-purpose AI (GPAI) models became applicable. The GPAI Code of Practice was approved just in time for this deadline, and the European Commission also published guidelines on prohibited AI practices, the AI system definition, and GPAI model provider obligations. However, as covered previously in Enterprise AI Governance, there is now uncertainty regarding what the compliance deadline for high-risk AI systems will be, following the Commission’s proposal to delay the 2 August 2026 date, as part of the AI Act “simplification” changes.

2. But the Brussels effect remains limited. The EU’s comprehensive, horizontal AI law remains an outlier. Despite the immense global policy focus on AI in recent years, other jurisdictions have not opted to follow the EU’s approach. Simply put, there is no other AI law globally that is comparable to the EU’s in terms of its comprehensiveness, stringency, and enforcement teeth. China has enacted several AI-specific regulations, but these are more narrowly focused on the way in which AI is used to recommend, amplify, and generate content and information. Japan and South Korea have also enacted more sweeping AI laws, but these are much lighter touch than the EU’s from a compliance perspective. Across major economies like the UK, Canada, Australia, Singapore, and India, no comprehensive AI law is on the horizon. And in the U.S., it’s a non-starter.

3. Divergent approaches at the U.S. federal and state level. One of the first things President Trump did upon assuming office was to revoke President Biden’s landmark Executive Order on AI Safety. Since then, not only has the Trump administration pivoted away from the prior focus on AI governance and safety, it has repeatedly stated that its overriding AI policy objective is to strengthen U.S. leadership and dominance in AI, in order to seize the economic and national security advantages. Regulations that impede or restrict private sector AI activities—and the patchwork of hundreds of state AI laws—have been framed as a blocker to U.S. global AI dominance that could undermine U.S. competitiveness. Because of this, the Trump administration has attempted to deter and block U.S. states from passing and enforcing state level AI laws. However, its 10-year moratorium on state AI law enforcement was rejected by the Senate in July. A recent Executive Order outlines plans to challenge state AI laws via litigation and funding restrictions instead. Despite this, meaningful state AI laws have been enacted across the U.S. in 2025, including in California, New York, and Texas.

4. Foundation models remain the target of AI regulations and standards. The most advanced frontier models continue to receive significant regulatory attention, with various laws and standards worldwide specifically focused on foundation models. For example, California’s Transparency in Frontier AI Act (SB53) requires major AI developers to report certain safety incidents and to publish information about their frontier AI risk management frameworks. Similarly, New York’s RAISE Act requires “frontier AI model” developers to implement mitigations to reduce the risk of “critical harm”. And the EU’s regime for GPAI model providers, and the accompanying GPAI Code of Practice, outlines detailed rules for model and training data transparency, copyright compliance, and systemic risk mitigation. Finally, the safety risks of foundation models with frontier capabilities remains a major focus of policy and research efforts, as evidenced by the inaugural International AI Safety Report, published in January.

5. Increasing focus on how to implement labelling and marking of AI-generated content. Perhaps the most significant regulatory development in China was the Measures for Labelling of AI-Generated Synthetic Content, a regulation which took effect in September. The Measures require “implicit labels” for all AI-generated content (i.e., information embedded within file metadata to enable detectability of AI content), as well as “explicit labels”, such as text or audio notifications, for AI-generated content that might “confuse or mislead the public” (e.g., deepfake videos and AI-generated music). The Measures were accompanied by a mandatory national standard on AI Content Labelling Methods, which provides detailed requirements. Similarly, work is underway to develop the EU Code of Practice on marking and labelling of AI-generated content, with the first draft published in December. According to Merriam-Webster, “slop” was the word of the year, which perhaps highlights the collective fatigue for low-quality, AI-generated content and the inreasing importance of transparency.

6. Soft law, technical standards, and public-private sector collaboration prevail. The lack of comprehensive, EU AI Act-inspired laws does not mean there has been a lack of domestic AI policy activity. Many jurisdictions continue to prioritise soft law measures, such as producing guidelines and non-binding frameworks for organisations to leverage. For example, the UK Government published a Code of Practice on Cyber Security and continues to support the growth of the AI Assurance sector. Similarly, Singapore’s regulators published guidelines on agentic AI security and launched a Global AI Assurance Pilot and Sandbox, to enable industry generative AI testing. Finally, China released version 2.0 of its non-binding AI Safety Governance Framework. At the international level, industry and policy stakeholders worked together to produce various technical standards, such as the IEEE 3119 Standard on AI Procurement and the ISO 42005 Standard on AI System Impact Assessment. These efforts highlight that organisations have plenty of practical resources at their disposal which they can use to inform their AI governance work.

7. Lots of discussion, but less tangible action on the international stage. Topics relating to AI policy and governance continued to receive significant airtime on the international stage. However, it is difficult to pinpoint major developments that go beyond well-intentioned discussion and dialogue. The AI Action Summit, hosted by France in February, was broader in focus than its 2023 and 2024 predecessor events, the AI Safety Summit and the AI Seoul Summit. Also, the UN establishing a Global Dialogue on AI Governance ensures a permanent home for such discussions. And China publishing its Global AI Governance Action Plan, and proposing a World AI Cooperation Organisation, highlights the leading role it seeks to play. Perhaps the most concrete development was the entry into force of the Council of Europe’s AI Treaty, which opened for signature in 2024. However, organisations hoping for greater regulatory convergence globally in 2026 are likely to be disappointed.

8. The U.S.-China “AI race” and the shifting sands of AI export controls. The release of DeepSeek-R1 in January 2025 was a defining moment, as it highlighted the strength and disruptive potential of China’s AI ecosystem, despite the AI chip export controls already in place. DeepSeek, as well as Alibaba’s Qwen model family, demonstrate that China can innovate around hardware constraints through algorithmic efficiency, while reinforcing the Chinese government’s support for open-source AI as a strategic advantage. However, despite China’s strengths in AI talent, research, patents, and open-source AI, the U.S. retains significant leads in private investment, frontier model performance, and global diffusion. The U.S. export control regime continues to evolve. President Biden’s January 2025 Framework for AI Diffusion hardened AI chip export controls and expanded the regime to include model weights. President Trump rescinded it in May, calling it “burdensome”. In December, the Trump Administration permitted Nvidia to export its H200 chips to China, in exchange for a 25% cut of the sales revenue. This pivot from strategic containment to transactional commerce represents a meaningful policy shift.
   
   

   Thanks for reading Enterprise AI Governance! Subscribe for free to receive new posts and support my work.

Global AI Governance in 2025 (free pdf download)

Global AI Governance in 2025 (by Oliver Patel)

399KB ∙ PDF file

Download

Download a high res pdf version of this Cheat Sheet, with hyperlinks to key sources. Can only be used or shared with attribution.

Download

The ultimate AI governance timeline: 70 key milestones in 2025

In scope: AI-related policy, legislation, regulations, official guidance, international agreements and declarations, and technical standards shaping AI governance across major jurisdictions.

Not in scope: Corporate policies, industry standards, AI model releases, technology developments, litigation, enforcement actions, and solely academic or civil society research and advocacy.

January

• 🇬🇧 UK Government publishes AI Opportunities Action Plan.

• 🇺🇸 U.S. Department of Commerce publishes Framework for AI Diffusion, tightening AI export controls.

• 🇰🇷 South Korea formally enacts the Framework Act on the Development of Artificial Intelligence and Establishment of Trust Foundation (AI Basic Act).

• 🇺🇸 President Trump signs Executive Order 14179: Removing Barriers to American Leadership in AI, directing development of a new AI action plan.

• 🇺🇸 President Trump revokes Executive Order 14110 on AI safety, signed by President Biden in October 2023.

• 🌐 The inaugural International AI Safety Report is published, led by Yoshua Bengio and authored by 100+ AI experts.

• 🇬🇧 UK publishes AI Cyber Security Code of Practice.

February

• 🇪🇺 EU AI Act provisions on AI literacy and prohibited AI become applicable from 2 February, marking the first compliance deadline.

• 🇳🇿 New Zealand Government publishes Responsible AI Guidance for the Public Service, covering generative AI use.

• 🇪🇺 European Commission publishes guidelines on prohibited AI practices under the EU AI Act.

• 🇪🇺 European Commission publishes guidelines on AI system definition to clarify scope of the EU AI Act.

• 🌐 Statement on Inclusive and Sustainable AI for People and the Planet signed by world leaders at the AI Action Summit in France.

• 🇸🇬 Singapore launches Global AI Assurance Pilot, focused on technical generative AI testing.

• 🇪🇺 European Commission withdraws the AI Liability Directive proposal.

• 🇬🇧 UK AI Safety Institute becomes the AI Security Institute and shifts focus.

March

• 🌐 ASEAN publishes Responsible AI Roadmap (2025-2030), setting regional AI governance priorities.

• 🇺🇸 NIST publishes updated report on Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations.

• 🇧🇷 Brazil’s AI Bill (2338/2023) forwarded to Chamber of Deputies following Senate approval.

• 🇯🇵 Japan publishes updated AI Guidelines for Business (Version 1.1).

April

• 🇺🇸 OMB issues Memoranda on federal agency use and acquisition of AI.

• 🇪🇺 European Commission publishes AI Continent Action Plan.

• 🇺🇸 President Trump signs Executive Order 14277: Advancing AI Education for American Youth.

May

• 🌐 Singapore Consensus on Global AI Safety Research Priorities published following international summit.

• 🇺🇸 Framework for AI Diffusion rescinded by Trump Administration.

• 🇨🇳 China State Council deprioritises Draft AI Law, removing it from 2025 legislative plan.

• 🌍 African Union officially declares the Africa AI Strategy a top priority for the continent.

• 🇺🇸 TAKE IT DOWN Act signed into U.S. federal law, criminalising non-consensual intimate imagery.

• 🇺🇸 U.S. House of Representatives votes to pass 10-year state AI law moratorium.

• 🌐 ISO/IEC 42005:2025 Standard on AI System Impact Assessment published.

June

• 🇺🇸 Texas enacts Responsible AI Governance Act (HB 149), regulating AI use in the state.

July

• 🇺🇸 U.S. Senate votes to reject 10-year state AI law moratorium.

• 🌐 ISO/IEC 42006:2025 Standard published, outlining requirements for bodies providing audit and certification of AI management systems.

• 🇸🇬 Singapore launches Global AI Assurance Sandbox.

• 🇳🇿 New Zealand Government publishes Responsible AI Guidance for Business.

• 🇪🇺 European Commission publishes guidelines for providers of general-purpose AI models.

• 🇺🇸 White House releases America’s AI Action Plan, outlining 90+ federal policy actions across three pillars.

• 🇺🇸 President Trump signs Executive Order 14320: Promoting the Export of the American AI Technology Stack.

• 🇺🇸 President Trump signs Executive Order 14319: Preventing Woke AI in the Federal Government.

• 🇺🇸 President Trump signs Executive Order 14318: Accelerating Federal Permitting of Data Center Infrastructure.

• 🇪🇺 EU publishes General-Purpose AI Model Training Data Public Summary template.

• 🇨🇳 China proposes World AI Cooperation Organisation at international forum.

• 🇨🇳 China publishes Global AI Governance Action Plan.

August

• 🇪🇺 EU formally approves General-Purpose AI Code of Practice.

• 🇪🇺 Obligations for GPAI model providers become applicable under the EU AI Act from 2 August.

• 🇺🇸 Trump Administration negotiates deal with Nvidia and AMD over China AI chip exports.

• 🇨🇳 China releases national AI Plus Plan to accelerate AI integration across industries.

• 🌐 UN General Assembly adopts Resolution A/RES/79/325, establishing the Global Dialogue on AI Governance and Independent International Scientific Panel on AI.

September

• 🇨🇳 China’s Measures for Labelling of AI-Generated Synthetic Content take effect.

• 🇨🇳 China’s mandatory national standard on Labelling Method for Content Generated by AI (GB 45438-2025) takes effect.

• 🇯🇵 Japan AI Promotion Act takes effect in full.

• 🇬🇧 UK publishes Trusted Third Party AI Assurance Roadmap.

• 🇨🇳 China releases AI Safety Governance Framework 2.0.

• 🇺🇸 California enacts Transparency in Frontier Artificial Intelligence Act (SB53).

• 🇺🇸 CAISI publishes Evaluation of DeepSeek AI Models.

October

• 🇸🇬 Singapore publishes Draft Addendum on Securing Agentic AI.

November

• 🌐 Council of Europe Framework Convention on AI enters into force.

• 🇨🇳 China updates National Cybersecurity Law with AI-specific provisions.

• 🇮🇳 India publishes AI Governance Guidelines.

• 🇪🇺 European Commission announces proposed changes to EU AI Act as part of Digital Package simplification proposal.

• 🇪🇺 EU AI Act whistleblower tool launched by European Commission.

• 🇦🇺 Australia establishes AI Safety Institute.

December

• 🇦🇺 Australia publishes National AI Plan.

• 🇺🇸 Trump Administration allows NVIDIA to export H200 AI chips to China.

• 🇦🇺 Australia passes social media age restrictions law, with AI verification implications.

• 🇺🇸 President Trump signs Executive Order on Ensuring a National Policy Framework for AI, establishing AI Litigation Task Force to challenge state AI laws.

• 🇺🇸 OMB publishes Memorandum M-26-04: Increasing Public Trust in AI Through Unbiased AI Principles.

• 🇬🇧 UK publishes progress report on Copyright and AI consultation.

• 🇪🇺 European Commission publishes first draft of Code of Practice on marking and labelling of AI-generated content.

• 🇬🇧 UK AI Security Institute publishes Frontier AI Trends Report.

• 🇺🇸 New York enacts RAISE Act (S6953B/A6453B) mandating transparency for frontier AI models.

• 🇨🇳 China releases draft regulations for AI-powered anthropomorphic interaction services.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

As many of us wind down for the holiday season, I wanted to take a moment to thank you all for reading, supporting, and sharing my newsletter throughout 2025.

I also want to share some reflections from the year and the journey ahead, including the Top 10 Most Popular Posts of 2025 and what you can expect from me in 2026.

I greatly appreciate every single time this newsletter is opened, read, liked, and shared, as well as all of the insightful comments and responses I receive. We live in an age of information overload, exceptionally busy schedules, and unprecedented demands in our work and personal lives. Therefore, I do not take for granted the valuable time you spend engaging with my work. And I commit to doing all I can to ensure my content and resources remain truly valuable for your work next year.

It’s been just over 1 year since I launched Enterprise AI Governance, on 20 December 2024. I can confidently say that this turned out to be one of the best decisions I’ve made.

I am proud that we now have a community of 6,810 Enterprise AI Governance subscribers. This includes people from 135 countries and almost every continent—please forward this post to your friends and colleagues in Antarctica! The top five most represented countries are the U.S., UK, India, Germany, and Canada.

When I checked the stats today, I was also thrilled to see that the posts on Enterprise AI Governance have been viewed over 222,000 times in the past year.

To me, this highlights the importance of continuing and deepening this work in 2026. The ways in which AI is rapidly advancing, the increasingly democratised nature of this technology, its development and deployment in sensitive, real-world contexts—from healthcare to defence—and the practical ways in which individuals, organisations, governments, and societies can effectively manage risks and promote safe and ethical outcomes, are among the defining issues of our time.

Although it may not always feel like it, we remain at the very early stages of AI governance. The choices we make today will have important ripple effects on humanity for many years to come.

I started this newsletter for two simple reasons: to (re)cultivate my love for writing and to fill a gap in the market with practical guidance on AI governance implementation.

Before joining AstraZeneca, I’d spent my entire career writing. And I mean pre-ChatGPT writing, where you start with nothing but your mind, a blank page, and random Google searches!

After spending several years crafting papers in academia and government, I found myself in corporate leadership. In this environment, as many of you can relate, reading and writing longer-form content was suddenly no longer a core part of my work. This was something I missed and I didn’t want to lose a skill that I valued. Also, given that my shorter-form LinkedIn posts on AI governance were resonating, I thought it made sense to go deeper.

Doing so has provided me with several powerful benefits. The way in which producing longer-form content requires you to grapple with complexity, engage directly with primary sources, and formulate cogent arguments sharpens your mind in ways not many other activities can. This has enabled me to understand the challenges and dynamics of Enterprise AI Governance in a meaningful way, which has also connected me with many others at the forefront and on the frontline of this emerging field.

So far, I am proud to have published 36 editions of Enterprise AI Governance. Not quite once per week (sorry!), but not too far off either. As you will see from the list below, the most popular content is on the EU AI Act, agentic AI governance, and the practicalities of AI governance implementation, including AI literacy, AI procurement, and AI usage policies.

Next year, I will double down on these topics, by continuing to provide in-depth analysis on the EU AI Act (including how it may be amended) and global AI regulations, as well as the practicalities of how to implement AI governance in an organisational context, covering open-source AI, AI risk assessments, and accountability structures, and more.

I will also cover more technical topics, such as multi-agentic systems, explainability, hallucinations, model sycophancy, retrieval-augmented generation (RAG), knowledge graphs, and context engineering. Finally, I will provide an overview of the Top 10 AI Governance and Safety Papers in 2025.

This newsletter will remain unashamedly practical, with an emphasis on longer-form content and tangible resources that empower you to understand, implement, and master AI governance.

Of course, the thing I am most excited for in 2026 is the launch of my book, Fundamentals of AI Governance. It will cover all of the topics covered in this newsletter and so much more, in a uniquely practical, visual, and comprehensive way. Given that Enterprise AI Governance is a free newsletter, signing up for my book is the best thing you can do to support my work. To secure a 25% discount during the pre-launch period, sign up at the link below.

Secure your 25% discount

Top 10 Most Popular Posts of 2025

Here are the Top 10 Enterprise AI Governance Posts of 2025, ranked by total number of views:

1. The Unofficial AIGP Resource Guide* (February 2025)

2. The Ultimate EU AI Act Resource Guide* (May 2025)

3. Initial Reflections on Agentic AI Governance (March 2025)

4. Top 12 Papers on Agentic AI Governance (Part 1) (June 2025)

5. The Ultimate Guide to AI Literacy (March 2025)

6. What’s next for EU AI Act “simplification”? (November 2025)

7. AI Usage Policy Playbook (March 2025)

8. General-Purpose AI Model Compliance Guide (Part 1) (August 2025)

9. EU Publishes General-Purpose AI Code of Practice (July 2025)

10. The AI Procurement and Contracts Toolkit: 10 Practical Resources (October 2025)

Given the popularity of these two resource guides, and the fact that they are now slightly out of date, I will publish updated versions in early 2026.

Top 10 Posts of 2025

3.92MB ∙ PDF file

Download

Download the pdf, hyperlinked version here!

Download

Meme of the Year

To end on a light-hearted note, Homer Simpson was my most popular meme of 2025—and also my second most popular LinkedIn post (with over 166k views).

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

On Thursday 11 December 2025, President Trump issued an Executive Order on Ensuring a National Policy Framework for AI. This represents the administration’s latest attempt at blocking and constraining how U.S. states regulate AI. This article summarises the new Executive Order, situates it in the wider context of Trump’s AI policy, assesses the challenges the administration faces in preempting state AI laws, and reflects on what companies should do next.

For a detailed, up-to-date, and visual guide to U.S. AI law and policy (covering the federal and state levels), as well as U.S, China, and EU comparison charts, sign up to secure a 25% discount for my forthcoming book, Fundamentals of AI Governance (2026).

Get a 25% discount

What just happened?

On 11 December 2025, President Trump issued a new Executive Order that attempts to block states from enforcing existing AI regulations and deter states from enacting new AI laws.

Just five months after the Senate rejected the proposed 10-year moratorium on state AI laws by 99 votes to 1, the administration is having another bite of the cherry—this time through litigation, funding restrictions, and federal preemption, potentially via new legislation.

However, it is important to note that the latest development is an Executive Order (i.e., a presidential directive), not legislation. Meaningfully preempting state laws in this way would require legislation, which requires approval from both the House of Representatives and the Senate. It does not appear that substantive political changes have occurred in the past few months to render this more likely than it was back in the summer.

Nonetheless, this latest Executive Order is a powerful signal of intent that highlights the administration is digging in on this particular issue, despite the large-scale opposition to the previous 10-year moratorium proposal.

Indeed, the administration has repeatedly stated that its AI policy objective is to “sustain and enhance America’s global AI dominance”. State AI laws have been in the firing line, as part of the wider focus on pursuing deregulation as a means to propel the U.S. as the world’s dominant AI power.

Preemption is a U.S. constitutional principle whereby federal law (i.e., law passed by Congress) takes precedence over state law when there are conflicts between the two. When a federal law “preempts” a state law, the state law is effectively nullified. However, the controversy regarding state AI law preemption is partly due to the fact that there is no comprehensive federal AI law. And the federal AI laws that are on the books cover narrower domains such as strengthening the U.S. AI industry, support and funding for AI research and infrastructure, and export controls. Where state AI laws focus on responsible AI topics like transparency or bias, there is arguably a lack of federal law to preempt these laws.

What is the new Executive Order on AI?

President Trump signed the Executive Order: Ensuring a National Policy Framework for Artificial Intelligence on Thursday 11 December 2025,

An Executive Order is a directive issued by the President to federal agencies and executive branch officials. It is issued by the President unilaterally, does not require Congressional approval and cannot, by itself, override state laws.

The core argument Trump presents in this Executive Order is that the patchwork of many state AI laws creates compliance burdens and administrative complexity that could undermine U.S. competitiveness. Having 50 different AI regulatory regimes, the Executive Order argues, “makes compliance challenging, particularly for startups“.

Colorado’s AI law—Consumer Protections for AI (SB24-205), effective date 30 June 2026—is singled out for criticism, with the Executive Order claiming that such laws tackling “algorithmic discrimination” may force AI models to produce false results in order to avoid differential treatment of protected groups.

The declared policy of the administration is to establish a “minimally burdensome national standard” for AI, as opposed to “50 discordant State ones”. To achieve this, the Executive Order directs a multi-pronged set of actions and broader strategy for the U.S. federal government to pursue:

• First, the Attorney General must establish an AI Litigation Task Force within 30 days (of the Executive Order), with the goal of challenging state AI laws in court. The focus will be on identifying state laws that are inconsistent with the U.S. AI policy of “sustaining and advancing U.S. global dominance in AI”. The grounds for legal challenge could include “unconstitutional regulation” of interstate commerce or preemption by existing federal regulations.

• Second, the Secretary of Commerce, in consultation with other government leaders, must publish an evaluation of existing state AI laws within 90 days (of the Executive Order), identifying “onerous” laws that conflict with the U.S. AI policy objective mentioned above. This evaluation must focus on laws that “require AI models to alter truthful outputs” or that compel organisations to “disclose information in a manner that would violate the First Amendment”. This is significant, as it explicitly targets laws relating to AI transparency and bias mitigation, two core threads throughout many existing state AI laws.

• Third, the Order imposes funding restrictions on states that are deemed to have “onerous AI laws”. States identified through the evaluation referenced above will be ineligible for certain categories of federal funding under the Broadband Equity Access and Deployment (BEAD) Program. U.S. government executive departments and agencies are also directed to assess whether they can condition discretionary grants on states agreeing not to enforce their existing AI laws and/or not enacting new AI laws. This could result in government agencies withholding certain types of grant funding from states.

• Fourth, the Federal Communications Commission (FCC), the agency that regulates U.S. communications (e.g., broadcasting, internet, and telecommunications), must initiate work to determine whether to adopt a federal reporting and disclosure standard for AI models that would preempt conflicting state laws. It is important to note that the Executive Order requires the FCC to “initiate a proceeding”; it does not require the FCC to adopt such a standard for AI transparency. However, the implied thinking is that if there is a national standard, this could be argued to supersede state AI laws covering AI reporting and disclosure.

• Fifth, the FTC must issue a policy statement explaining when state laws requiring the alteration of the “truthful outputs” are preempted by federal prohibitions on deceptive practices, which largely stem from the FTC Act that prohibits “unfair and deceptive practices”. Again, the implication is that such guidance would strengthen the case for existing federal laws superseding state AI laws in this domain too.

• Finally, the administration will prepare a legislative proposal for a uniform federal AI framework that preempts state laws. However, the proposal would be narrow and would not seek to preempt state laws covering any of the following policy areas:

  • child safety;

  • AI compute and data centre infrastructure;

  • state government procurement and use of AI; and

  • other topics to be determined.

This is significant as it highlights the policy areas which the administration deem to be legitimate domains of autonomous state AI lawmaking, even if the result is regulatory divergence within the U.S.

How did Trump attempt to block state AI laws previously?

This latest Executive Order is not the administration’s first attempt to constrain state AI laws. Back in May 2025, the administration proposed adding a 10-year moratorium on state AI laws enforcement to Trump’s flagship domestic policy and taxation bill, the ‘One Big Beautiful Bill Act’ (H.R.1).

I covered the 10-year moratorium in detail in a previous edition of Enterprise AI Governance: Unpacking the 10-year Moratorium on U.S. State AI Laws. Here is a summary of that article.

The proposed moratorium was designed to prevent U.S. states from being able to enforce “any law or regulation limiting, restricting, or otherwise regulating AI models, AI systems, or automated decision systems” for a period of 10 years. Although this would not technically have prevented states from introducing and passing new AI laws, the broad restriction on enforcement would have rendered doing so pointless.

This was an attempt by the federal government to preempt state AI laws by blocking states from enforcing laws they had already passed, as well as any new laws they might pass in the future. The stated goal was to halt the “proliferation of a complex and fragmented patchwork of state AI laws”, in support of AI innovation across the country.

The House of Representatives voted to pass the state AI law moratorium by a narrow margin, largely along party lines, with 215 in favour and 214 against. However, following this, the moratorium was decisively rejected. In July 2025, the Senate voted 99 to 1 to remove it from the bill. This followed a significant bipartisan campaign against the provision. A June 2025 letter, signed by 260 state lawmakers, stated that “states are laboratories of democracy accountable to their citizens and must maintain the flexibility to respond to new digital concerns”. Several Republican state governors also campaigned against the proposal.

The main reason the moratorium proved so controversial was not really about AI. The case (against it) centred on the philosophical objection to the federal government constraining states in this way, particularly given that there is no comprehensive federal AI law to take precedence. The argument was that the states’ hands were being tied, without an alternative federal framework on the table. The fundamentals of the situation do not appear to have meaningfully changed in the months that have passed.

However, the December 2025 Executive Order represents a pivot in strategy. Rather than seeking blanket preemption through legislation—which requires Congressional approval—the administration is now pursuing litigation, agency action, and funding leverage. These approaches can be initiated by the executive branch alone. However, as noted above, the President’s legal authority to actually preempt state laws remains limited without legislation. The administration may also pursue new legislation, but this will likely face similar challenges in Congress.

What AI laws do states have and will Trump succeed in blocking them?

In lieu of comprehensive federal AI law, dozens of U.S. states have enacted AI-related laws. 131 such laws were passed between 2016 and 2024, and over 700 AI-related bills were proposed in 2024 alone. The states with the most AI-related laws are California, Colorado, Maryland, Utah, and Virginia. California has been particularly active, enacting dozens of laws that regulate AI in different ways. These laws cover themes including fairness and accountability, transparency, data privacy, deepfakes, and government use of AI.

However, the administration faces an uphill task in its efforts to block the enforcement and enactment of these laws. Now that the moratorium has been rejected by the Senate due to concerns regarding preemption and the ways in which the federal government can constrain the states, it is fair to argue that subsequent federal legislative proposals will be met with similar levels of scrutiny and controversy.

What other AI policy actions has this administration taken?

The new Executive Order sits within a broader pro-business and pro-innovation AI policy agenda.

Trump has pivoted away from the Biden administration’s AI governance and safety agenda. Promoting U.S. AI leadership—which has always been a core federal AI policy objective—now takes centre stage. One of the first actions taken by Trump at the start of this second term, in January 2025, was to revoke various Biden-era executive orders, including the flagship Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.

In July 2025, the administration published Winning the Race: America’s AI Action Plan, which outlines how the U.S. can achieve and maintain “unquestioned and unchallenged global technological dominance” in AI.

I also covered the administration’s AI Action Plan in a previous edition of Enterprise AI Governance: What America’s AI Action Plan Means for AI Governance. A summary of that explainer piece is provided below.

The AI Action Plan contains dozens of policy recommendations across three pillars:

• Pillar 1. Accelerate AI Innovation

• Pillar 2. Build American AI Infrastructure

• Pillar 3. Lead in International AI Diplomacy and Security

Notably, under Pillar 1, the AI Action Plan recommended withholding funding for AI-related initiatives from states with “burdensome AI regulations”. This core idea has now been operationalised through the new Executive Order and its funding restrictions. Moreover, the rejection of the 10-year moratorium by the Senate did not extinguish the underlying policy objective; rather, it has been repackaged and is now pursued through alternative means.

The AI Action Plan is significant because it directly links AI regulations with the ability (or lack thereof) of companies to innovate at speed. Put simply, the Trump administration believes that AI should not be constrained by regulations and that doing so would impede the U.S. economic and security prospects in a damaging way.

Other recommended policy actions in the AI Action Plan included:

• Taking action to review and remove any existing Federal regulations that impede AI innovation.

• Ensure the federal government only procures “unbiased” and “ideologically neutral” large language models.

• Fast-track and streamline processes for data centre construction review, approval, and licensing.

• Advocate for “pro-innovation” approaches to international AI governance, that reflect “American values” and shift away from “burdensome regulations”.

Other notable AI policy measures pursued by this administration include Executive Orders on advancing AI education for American youth (April 2025), accelerating federal permitting for data centre infrastructure (July 2025), preventing “woke AI” in federal government procurement (July 2025), and promoting the export of the American AI technology stack (July 2025). The TAKE IT DOWN Act, which criminalises the publication of non-consensual intimate deepfakes, was signed into law in May 2025.

What should companies operating in the U.S. do now?

If you think AI governance is not relevant for your organisation because the current administration is pro-AI and against restrictive AI regulation, you could be in for a rude awakening if things go wrong.

Deregulation does not mean that AI risks no longer apply to you or that you are not exposed. Indeed, the AI Action Plan itself highlights various AI-related risks that could slow innovation, including interpretability, robustness, and misalignment. Furthermore, all enterprises using generative AI at scale are exposed to a litany of (relatively novel) data-related risks. I outlined these in my article on the PROTECT Framework: Managing Data Risks in the AI Era.

The PROTECT Framework empowers you to understand, map, and mitigate the most pertinent data risks that are fuelled by widespread adoption of generative AI, covering themes such as public AI tool usage, rogue internal AI projects, opportunistic vendors, and compliance and copyright breaches. These risks cannot be mitigated without a robust approach to AI governance, which serves as a reminder that AI-specific regulatory compliance is not the sole driver for enterprise AI governance.

Moreover, even the White House’s Office of Management and Budget (OMB) describes effective AI governance as “key to accelerated innovation”. Indeed, the AI governance framework that OMB directs federal agencies to implement—covering AI development, deployment, procurement, and use—is robust. This suggests that although the U.S. government does not want there to be any regulatory measures getting in the way of U.S. companies’ AI activities, it nonetheless recognises that a well-designed and proportionate AI governance framework is important for both risk mitigation and value generation, especially in sensitive domains.

For U.S. companies, the reality today is the same as it was yesterday. There still exists a complex patchwork of many state AI laws to contend with, as well as federal and state laws that meaningfully regulate or implicate AI in specific ways, such as privacy, copyright, employment, and consumer protection laws. Given the complexity of developing different internal AI governance frameworks for different jurisdictions, I always recommend having a company-wide AI governance framework that promotes and facilitates compliance, risk management, and AI-enablement across all the important jurisdictions you operate in.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

Your enterprise increasingly relies on AI products and foundation models developed by other companies. This article tackles two critical challenges for AI governance leaders: i) how to move from ‘paper-based’ vendor due diligence to dynamic and continuous oversight, and ii) how to manage the novel risks of building AI systems with externally provided foundation models.

If you enjoy my work and want to read a comprehensive, step-by-step guide to enterprise AI governance, sign up to secure a 25% discount for my forthcoming book, Fundamentals of AI Governance (2026).

Secure your 25% discount

Over the past few weeks, this newsletter has analysed some of the most pertinent challenges facing AI governance leaders in 2025.

Part 1 explored challenges 1-3 (see below). It argued that the democratisation and widespread accessibility of AI is driving intense volume and velocity of AI use cases. This is contributing to an AI risk ‘vibe shift’, overwhelming AI governance functions, and necessitates updating the risk-based approach to AI governance.

We then took a mini detour for a deep dive on challenge 4—protecting confidential business data—in which I outlined the PROTECT Framework for Managing Data Risks in the AI Era. The purpose of the PROTECT Framework is to enable organisations to understand, map, and mitigate the most pertinent data risks that are fuelled by widespread adoption of generative AI.

Today’s article continues the series by analysing challenges 5 and 6, both of which are focused on the way in which enterprises leverage, and increasingly rely on, AI models, products, and services developed and provided by external organisations. It provides practical advice and guidance on how to conduct effective vendor due diligence and oversight, what AI procurement processes should entail, and notable challenges and risks to mitigate when using pre-trained foundation models to develop and deploy customised AI applications.

As a reminder, here are the top 10 challenges this series covers.

Top 10 Challenges for AI Governance Leaders in 2025

1. The ‘democratisation dilemma’. How to maintain robust oversight and promote compliance when the ability to develop, deploy, and use AI is democratised and widely accessible?

2. Volume and velocity. How to keep up with the sheer volume and rapid pace of enterprise AI initiatives, whilst cutting through the noise and deploying finite resources and expertise on the highest value work?

3. Refining the risk-based approach. How to respond to the AI risk ‘vibe shift’ and effectively target governance on the relatively small proportion of AI systems and use cases that could pose significant risks?

4. Protecting confidential business data. How to protect confidential business data when there is immense hunger to experiment with and use the latest AI applications that are released on the market?

5. Ongoing vendor due diligence and oversight. How to move beyond ‘paper-based’ vendor due diligence and apply continuous oversight on the performance, trustworthiness, and safety of externally provided AI applications?

6. AI engineering: building with foundation models. How to determine your rights, responsibilities, and liabilities—as well as the novel risks and tangible mitigations—when building AI systems with foundation models provided by external organisations?

7. Open-source AI model oversight. How to effectively govern the widespread access and use of open-source AI models, to safeguard your organisation from legal, compliance, and cyber security risks, whilst promoting innovation?

8. Embedding compliance by design. How to build AI systems that promote compliance by design and default, to make it seamless for your workforce to do the right thing?

9. Agentic AI governance: taking the human out of the loop. How to promote responsible, meaningful, and empowered human oversight of AI, when the fundamental goal of agentic AI is to take the human out of the loop?

10. Digital governance silos and inefficiencies. How to effectively streamline and integrate your disparate digital governance and risk management processes and capabilities, to improve the user experience and accelerate AI innovation, whilst also strengthening your compliance posture?
    

    Thanks for reading Enterprise AI Governance! Subscribe for free to receive new posts and support my work.

5. Ongoing Vendor Due Diligence and Oversight

How to move beyond ‘paper-based’ vendor due diligence and apply continuous oversight on the performance, trustworthiness, and safety of externally provided AI applications?

Modern enterprises are becoming heavily reliant on AI products, applications, and services developed and provided by external organisations. Core enterprise workflows and domains, including recruitment, learning and development, IT support, and financial analytics are powered by bespoke vendor AI solutions. Furthermore, virtually all legacy enterprise software is being enhanced and updated with novel AI features and capabilities.

For most organisations, it makes more sense to buy rather than build. However, this amplifies various AI risks, such as unauthorised use and sharing of confidential data, liability exposure for AI performance issues and incidents, and copyright breaches.

For these reasons, AI vendor due diligence and oversight is an integral part of enterprise AI governance—but it is one of the hardest pillars to get right. This is because, fundamentally, organisations struggle to effectively monitor and control how well these externally provided AI applications work.

There are three core elements to the ‘paper-based’ governance referred to above:

1. Vendor due diligence assessment process. This typically consists of a set of assessments and questions that must be completed by the vendor before they are onboarded, as well as documentation and assurances they must provide.

2. Contracts, clauses and templates. This refers to standardised contractual clauses that can be included in agreements with AI vendors, covering topics like service delivery, data use, regulatory compliance, indemnity protection, and allocation of responsibilities and liabilities.

3. AI policies, covering procurement and partnerships. Finally, enterprise AI policy foundations should outline the principles, requirements, and processes for AI procurement and partnerships.

The challenge is that these core elements are necessary, but not sufficient, for effective AI vendor oversight and risk management. In isolation, they can represent relatively blunt instruments that risk creating the impression of robust governance and risk management without significantly impacting outcomes or mitigating the most serious risks.

For example, pre-vendor onboarding due diligence covers a snapshot in time and does not enable dynamic response to future issues or concerns that may arise. This is especially problematic given the pace at which vendors are updating and evolving their applications, and the importance of monitoring performance, reliability, impacts, and incidents across a large organisation.

Practical solutions

• Pre-deployment testing and PoCs: before onboarding new solutions and signing hefty contracts, enterprises should perform testing and evaluation of vendor AI solutions. This testing can focus on both performance (i.e., how well it works for various use cases) as well as safety (i.e., to what extent are guardrails effective). This can include undertaking proof-of-concept (PoC) engagements and leveraging sandbox-type environments. This provides an additional layer of confidence over solely paper-based assessment exercises. Simply put, the best way to determine how well an AI solution will work for your organisation is to rigorously test and evaluate it.

• Ongoing monitoring and evaluation: once externally provided AI solutions are deployed in production, they should be monitored and evaluated on an ongoing and dynamic basis. Enterprises should not merely rely on vendors’ internal monitoring processes and the written assurances they provide pre-onboarding. This should be augmented with independent testing and monitoring—perhaps even involving third parties—optimised for their use cases. This approach is especially important for mission critical applications. This can be both quantitative (e.g., automated tracking of output accuracy) and qualitative (e.g., user feedback surveys and incident reports). Ideally, quantifiable performance metrics and thresholds (or baselines) should be contractually guaranteed, with robust response procedures in case of deviations or deficiencies.

• New AI features and capabilities: the pre-vendor onboarding due diligence, and the resulting contract that is signed, often fails to account for the myriad ways in which the product could be updated with new AI features or capabilities. Oftentimes, such new AI features or capabilities integrated into an existing product can materially change the risk level, compliance scope, or potential impact of the overall product. Therefore, mechanisms must be in place for formal assessment and evaluation of new AI features and capabilities that trigger certain risk-based criteria—even when the vendor and the product they provide is already approved and deployed in production.

• Shaping product implementation and roadmaps: the best way to be aware of product enhancements, including the integration of new AI features and capabilities that can amplify risk, is to maintain active and focused engagement with the vendor. Part of this should include working with the vendor to implement the product in a responsible way, taking advantage of all available guardrails, monitoring, and safety features—as well as the expertise they should have from countless prior deployments. More broadly, this is an underappreciated facet of enterprise AI governance. The more influence you have on the vendor and their products, the more effective you will be at managing and mitigating risks.

6. AI Engineering: Building with Foundation Models

How to determine your rights, responsibilities, and liabilities—as well as the novel risks and tangible mitigations—when building AI systems with foundation models provided by external organisations?

The generative AI boom of the past few years has fundamentally reshaped the nature of enterprise AI. Almost all enterprises are now building AI applications with pre-trained foundation models. And it is becoming less common for organisations to independently train and develop AI models for deployment in production.

Chip Huyen characterises “AI engineering” as a new discipline, distinct from traditional machine learning engineering, that centres around building production applications (i.e., AI systems) using readily available foundation models developed and provided by external organisations, rather than training AI models from scratch.
Although AI and data science teams have always used external models, components, packages, toolkits, and libraries, AI engineering has quickly emerged as the dominant modality for enterprise AI development and deployment today.

This is true for both major enterprises and the vendors that serve them. For the latter, many of the vendor-provided AI products discussed above are ultimately software wrappers around advanced foundation models developed by AI labs. For the former, many of their internal AI use cases and AI systems leverage those same models, through services enabling foundation model consumption and use via APIs.

This challenges traditional AI governance frameworks and regulations, many of which were developed primarily during the traditional machine learning era. For example, the AI engineering modality disrupts the binary and outdated distinction between internal AI development of AI systems and procurement and use of externally provided AI solutions. Today, AI development is hybrid, as internal engineering teams build with external models. AI engineering also challenges the notion of AI systems designed and intended for a specific use case (or set of use cases), given the advanced, general-purpose capabilities of these frontier models.

Given that enterprises cannot rely solely on mainstream AI governance frameworks, standards, and regulations to surface and mitigate the myriad risks that building with foundation models entails, AI governance leaders need to do the heavy lifting themselves. Although there exists a broad spectrum of approaches and techniques for AI engineering—including prompt engineering, context management, retrieval augmented generation (RAG), and model fine-tuning—there are some shared risks and challenges that apply across the board.

Although this is not an exhaustive list, here are five of the most notable AI engineering challenges enterprises are currently grappling with:

• EU AI Act obligations for general-purpose AI (GPAI) model providers: it is important for enterprises to verify that the GPAI model providers they work with are complying with their applicable obligations under the AI Act and adhering to the GPAI Code of Practice (if they are a signatory). The most important aspect of this is obtaining (and adhering to) the technical documentation (e.g., instructions for use, usage policy, training data summary etc.) provided by the GPAI model provider, prior to building an AI system which integrates that GPAI model. Furthermore, enterprises must also track scenarios where they are using extremely large amounts of compute for GPAI model modification (i.e., fine-tuning), as this can trigger GPAI model provider status if certain thresholds are breached. See this previous newsletter article for a deep dive on the topic.

• Jurisdictional access restrictions: there are various jurisdictions that restrict access to, and use of, certain types of AI models and services. This includes use case type restrictions that apply in specific scenarios, as well as broad restrictions that always apply. For example, China’s Generative AI Services law restricts the types of AI models that can be integrated into “public-facing” AI applications, in order to prevent unauthorised information from being shared to the public. Also, various sanctions and export control regimes, such as the EU’s sanctions against Russia, restrict which AI models, services, and technologies can be provided or made available from one jurisdiction to another.

• Fragmentation of vendor terms and acceptable use policies: enterprises typically pay for access to various foundation model platforms, to enable on-demand consumption and use of a buffet of pre-trained foundation models. A key challenge is that different AI model providers have different terms of service and acceptable use policies, at both the platform level and the model level. For example, the overarching terms of service governing an organisation’s use of foundation model platforms provided by organisations like Google, Microsoft, and IBM will always differ, perhaps in important ways. And specific model providers, which enable access to their models for consumption via these platforms, have differing acceptable use policies and “pass through” terms that apply to the use of their foundation models. This becomes extremely complex to keep track of, especially for the average engineering team that simply wants to get on with their job.

• Black box opacity: it is widely accepted that explainability is difficult to achieve in the context of generative AI, in part due to the complexity and vastness of model architectures and the difficulties even the technical experts who developed those models have in understanding, interpreting, and explaining their outputs. This problem is amplified when using pre-trained, proprietary generative AI models, as you usually do not have access to the model weights and most likely have limited information about its training data sources. Therefore, it is virtually impossible to operationalise explainability in a meaningful way, which is a legal requirement or ethical imperative in certain scenarios.

• Compliance and reputational contagion: there is intense media and political scrutiny on every move made by the major AI companies. The more your organisation builds products and applications with the models they provide, the more exposed you may be in case something dramatic goes wrong. The reputational risks of working with certain vendors, as well as their AI and data compliance posture, are key considerations when selecting which foundation models to use.

Practical solutions

It is difficult to outline comprehensive solutions to these novel challenges without proposing a full enterprise AI governance framework, which is clearly beyond the scope of this article. However, that is what you can expect from my forthcoming book.

However, what can be said is that the common thread across these challenges is dependency and complexity. The imperative for AI governance leaders is to:

• manage the dependency proactively rather than reactively; and

• simplify the complexity of this domain into practical and actionable guidance for the wider organisation.

Foundation model providers—like AI vendors providing domain-specific AI products and services—require continuous and dynamic oversight and monitoring, as opposed to one-time procurement due diligence and contracts that are left to gather dust. A key part of this is maintaining centralised visibility regarding which models are approved and available for use, which jurisdictions these models can be used in, what the applicable terms of use are, which use cases and AI systems use which foundation models, and how much compute these engineering teams are using. Doing this requires modern tooling, such as an enterprise AI catalogue, which is linked to dynamic AI governance assessment and monitoring processes, as well as role-based training for AI engineering and data science teams, so that they are aware of their responsibilities.

Finally, do not rely solely on vendor assurances about compliance, safety, or performance. Independently test and evaluate foundation model performance for your specific use cases and with your data, as this matters much more than external benchmarks. And implement and document compensating controls if inherent limitations like explainability cannot be overcome.

Good luck!

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

On Wednesday 19 November 2025, the European Commission unveiled its Digital Omnibus Package, proposing targeted yet impactful amendments to the EU AI Act. This article distils what could change, why it matters for enterprise AI governance practitioners, and what to watch as trilogue negotiations begin.

If you value my work and want to learn more about the EU AI Act and AI governance implementation, sign up to secure a 25% discount for my forthcoming book, Fundamentals of AI Governance (2026).

Secure a 25% discount

What are the most important proposed EU AI Act changes?

On Wednesday 19 November 2025, the European Commission (henceforth the Commission) announced proposed changes to the AI Act. These changes are presented as “innovation-friendly AI rules” that will “reduce compliance costs for businesses”. It did so by publishing a proposal for a new regulation. The purpose of this proposed regulation is to “simplify” the AI Act with targeted yet meaningful amendments.

This is part of the Commission’s broader “Digital Package”, which is a major programme of work aiming to “simplify EU digital rules and boost innovation”. The Digital Package—which encompasses the “Digital Omnibus Regulation”—includes proposals to amend flagship digital laws like the AI Act, the GDPR, the ePrivacy Directive, the Data Act, and the NIS 2 Directive. Specifically, the Commission simultaneously published proposals for two regulations (so it’s not really an “omnibus” anymore):

• Proposal for Regulation on simplification of AI rules (which covers the AI Act amendments); and

• Proposal for Regulation on simplification of the digital legislation (which covers the amendments to the other EU digital laws mentioned above).

This article explains and analyses the six most important AI Act amendments that enterprise AI governance professionals need to understand. These are:

1. Timeline changes for high-risk AI system compliance.

2. Timeline changes for transparency-requiring AI system compliance.

3. Limiting registration in the public EU database for high-risk AI systems.

4. Softening of the AI literacy obligation.

5. Expanding the scope of the European AI Office’s regulatory powers.

6. Proportionality for small mid-cap (SMC) enterprises.

For each of these six proposed amendments, I explain what is in the law today, what changes are being proposed, and what the impact of these changes would be.

Earlier this week, I published an article on Enterprise AI Governance that explains how we got to this point and why the EU is now doing this. It provides a detailed account of the background context to AI Act simplification, highlighting how the ‘Draghi report’—which argued that digital regulatory burdens are impeding European growth and competitiveness—has influenced the Commission’s proposals.

It also outlines three important caveats on the EU’s legislative process that are worth repeating:

• This merely represents the proposal of one EU institution (the Commission). Such amendments of EU law require formal approval from both the European Parliament and the EU member states via the Council of the EU (the Council).

• Therefore, this proposal will now be followed by lengthy and potentially fraught trilogue negotiations between the Commission, European Parliament, and Council.

• Finally, it is impossible to predict what the final legislative text will consist of, how long the negotiation and approval process will take, and whether approval to amend the AI Act will ultimately be agreed on and enacted.

Scope of this article: this is not an exhaustive analysis of the entire AI Act simplification proposal and it does not cover every proposed amendment in the Commission’s 65-page document. Rather, it focuses on the six proposed changes that would be most consequential (if passed) for enterprises implementing AI governance. Also, it intentionally does not cover the proposed changes to the GDPR, nor the AI Act amendments that are directly related to the processing of personal data (e.g., use of sensitive personal data for bias mitigation), as this topic will be addressed in a future article on Enterprise AI Governance.

Disclaimer: this article is not intended to be legal advice and must not be relied upon or used in that way. Always consult a qualified legal professional.

1. Timeline changes for high-risk AI system compliance

What is in law today?

The key provisions relating to high-risk AI systems apply from 2 August 2026. This means that from 2 August 2026, unless there is a change in the law, providers and deployers must adhere to the obligations and requirements for high-risk AI systems and can be subject to investigations and penalties for non-compliance. However, this applicable date only applies to high-risk AI systems listed in Annex III (e.g., education, employment, administration of justice etc.) that are placed on the market or put into service from 2 August 2026 onwards.

For such Annex III high-risk AI systems that were placed on the market or put into service before 2 August 2026, providers and deployers are only subject to AI Act obligations and requirements if, from that date onwards, there is a significant change in design or intended purpose of the AI system. Furthermore, for high-risk AI systems that are products, or safety components of products, regulated by specific EU product safety laws listed in Annex I, the applicable date is 2 August 2027.

What changes are being proposed?

If you are asked “when do the compliance obligations for high-risk AI systems apply?”, your answer now has to be “it depends”.

Given the nature of the proposed amendments, there are various potential scenarios. The Commission is seeking to link the applicability of high-risk AI system provisions with the availability of technical standards and associated support tools. However, if these artefacts are not approved and available within a certain timeframe, there is a backstop date, which represents the latest applicable date.

Under this proposal there are, broadly speaking, three potential scenarios for when most of the provisions relating to high-risk AI systems may apply (covering high-risk AI system classification, development requirements, and obligations of providers, deployers, and other parties):

Scenario 1. If technical standards and associated support tools for high-risk AI system compliance are finalised and approved by the Commission, then the applicable compliance date will be six months after this approval (for high-risk AI systems listed in Annex III) and 12 months after this approval (for high-risk AI systems that are products, or safety components of products, regulated by an EU law listed in Annex I).

Scenario 2. However, if technical standards and associated support tools for high-risk AI system compliance are not finalised or approved by the Commission in time (i.e., before the dates below), then the applicable compliance dates will be 2 December 2027 (for high-risk AI systems listed in Annex III) and 2 August 2028 (for high-risk AI systems that are products, or safety components of products, regulated by an EU law listed in Annex I).

Scenario 3. Given that the applicable date in law today is 2 August 2026, if these amendments are not approved and enacted before this date, then the provisions relating to high-risk AI systems will, technically speaking, apply from then. This creates timeline pressure to get these changes approved within the next few months.

What impact would these changes have?

To clarify, 2 December 2027 and 2 August 2028 are the backstop dates for high-risk AI system compliance. To reinforce this point, the Commission has explained that the grace period will be up to sixteen months (referring to the time between 2 August 2026 and 2 December 2027). This means that if technical standards come too late (i.e., after 2 June 2027, which is six months before 2 December 2027) these backstop dates will apply.

These amendments shine the spotlight on the ongoing work being led by CEN/CENELEC to agree and publish technical standards. They also highlight the importance that the Commission places on these artefacts to support organisations and facilitate compliance.

However, even if the applicable date may be delayed—giving providers and deployers more time to prepare—this extra time has been achieved at the expense of certainty, with organisations now not knowing what the applicable date will be.

2. Timeline changes for transparency-requiring AI system compliance

What is in law today?

Article 50 of the AI Act outlines transparency obligations for providers and deployers of certain AI systems. Article 50 covers obligations relating to disclosure, informing end users about the use of AI, labelling certain deep fake content, and detectability of AI system outputs. Specifically, Article 50(2) stipulates that:

“providers of AI systems, including general-purpose AI systems, that generate synthetic audio, image, video, or text content shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated”.

Currently, this specific obligation applies from 2 August 2026. This compliance date applies to all AI systems, irrespective of whether they are placed on the market or put into service before or after 2 August 2026.

What changes are being proposed?

The Commission proposes to push back the applicable date for this specific transparency obligation to 2 February 2027 for providers of AI systems that have been placed on the market before 2 August 2026. This proposed six month delay to the applicable date only applies to obligation stipulated in Article 50(2) (on AI system output machine readability and detectability) and not the other transparency obligations outlined in Article 50.

What impact would these changes have?

This change would give providers of AI systems that have already been placed on the market or put into service, or that will be before 2 August 2026, and that generate synthetic audio, image, video, or text content (i.e., most generative AI systems), an additional six months to ensure that these AI systems are developed in such a way that ensures the outputs they generate are detectable as AI-generated.

Although this is a relatively short delay, it is nonetheless an acknowledgement by the Commission of the technical and engineering challenges providers face in developing or modifying their AI systems to adhere to this obligation. However, any AI systems placed on the market on or after 2 August 2026 will have to comply with this obligation from their release date.

3. Limiting registration in the EU public database for high-risk AI systems

What is in law today?

Annex III of the AI Act lists eight categories of high-risk AI system, including law enforcement (#6), education and vocational training (#3), and employment, workers’ management and access to self-employment (#5). However, there are classification rules which mean that just because an AI system is intended for use or used in one of these domains does not necessarily mean it is a high-risk AI system.

AI systems listed in Annex III are not considered high-risk if it is demonstrated that they do not pose significant risk of harm to health, safety, or fundamental rights. For example, if the AI system does not materially influence decisions or is only used for a narrow procedural task, the provider is entitled to demonstrate, based on a documented assessment, that it is not a high-risk AI system. This derogation, including the conditions to fulfil it, is outlined in Article 6(3) and only applies to AI systems listed in Annex III.

Providers must register high-risk AI systems listed in Annex III in the EU public database for high-risk AI systems, before those AI systems are placed on the market or put into service. Interestingly, this registration obligation also includes AI systems that the provider has concluded are not high-risk via the derogation procedure outlined in Article 6(3).

What changes are being proposed?

The Commission proposes to limit the scope of this registration obligation so that it no longer applies to AI systems that providers have concluded are not high-risk via the Article 6(3) derogation procedure. Simply put, where a provider has assessed and documented that an AI system used in an Annex III domain is not high-risk, the provider will not have to register that AI system in the EU public database for high-risk AI systems.

However, although providers can make this assessment independently and do not require any external approval (e.g., from the AI Office or market surveillance authority), providers will still be obliged to share the documentation of the assessment, containing the justification and supporting evidence, upon request from a regulator.

What impact would this have?

This may seem like a subtle change at first glance, but it would be consequential for organisations using AI at scale.

Most enterprise AI governance practitioners likely raised their eyebrows when they realised that every AI system used in a high-risk domain, including AI systems that are not high-risk due to their use for mere assistive, procedural, or preparatory tasks, would have to be registered. This will be difficult (or perhaps near impossible) to keep track of and implement, due to the increasingly ubiquitous use of AI to support and augment workflows across virtually all domains of enterprise activity. Indeed, the Commission describes this current registration obligation as a “disproportionate compliance burden”.

Therefore, the most obvious impacts of this change would likely be far fewer AI systems registered in the EU public database for high-risk AI systems and reduced administrative overheads for organisations developing and deploying AI systems.

However, it would also reduce public transparency regarding which AI systems providers deem not to be high-risk and how they have made such determinations. This could incentivise some providers to take a more expansive approach to interpreting Article 6(3) and determining what is not a high-risk AI system, as they may reasonably judge that the risk of doing so (and being penalised for getting it wrong) is lower with significantly less public scrutiny.

4. Softening of the AI literacy obligation

What is in law today?

Under Article 4 of the AI Act providers and deployers of AI systems are obliged to implement “AI literacy”. This is one of the most important aspects of the law, because it has contributed to many organisations in the EU and further afield rolling out AI training and upskilling initiatives for their workforce. Specifically, Article 4 requires organisations to ensure that “staff and other persons dealing with the operation and use of AI systems” have a “sufficient level of AI literacy”.

In practice, given that all staff in modern organisations can use AI systems (e.g., ChatGPT or Gemini), a reasonable interpretation of Article 4 is that all of these staff should receive some form of AI-focused training. The AI literacy obligation has been applicable since February 2025. However, there are no enforcement penalties for non-compliance with it. Although non-compliance could be taken into account during enforcement investigations or proceedings relating to other aspects of non-compliance.

What changes are being proposed?

The Commission proposes to remove the obligation for providers and deployers to implement AI literacy. Rather than providers and deployers being legally required to ensure their staff operating and using AI systems have sufficient levels of AI literacy, the Commission and Member States will be required to foster AI literacy and “encourage providers and deployers of AI systems to take measures to ensure a sufficient level of AI literacy”. The Commission has alluded to the fact that the ambiguity of the current “unspecified obligation” has caused issues for businesses—especially smaller firms.

What impact would this have?

This change would be significant because it would remove the broad and expansive legal obligation for companies to implement AI literacy. However, human oversight of high-risk AI systems must still be assigned to staff with sufficient training and competence. Therefore, ensuring AI literacy is still required in that context.

Moreover, it will be practically impossible for any organisation to comply with the AI Act—or to manage AI risks, implement AI at scale, and maximise the value of AI—without educational and training initiatives focused on AI. Therefore, forward-thinking enterprises are unlikely to abandon their AI literacy programmes because it is no longer a legal requirement. However, certain initiatives may be scaled back, deprioritised, or change in focus or scope.

5. Expanding the scope of the European AI Office’s regulatory powers

What is in law today?

Here is a simplified summary of the (rather complex) AI Act governance regime:

• There are governance and regulatory bodies at both the EU and member state level.

• At the EU level, the most important bodies are the European AI Office (which is part of the Commission) and the European AI Board.

• At the member state level, the most important bodies are the market surveillance authorities. They are responsible for monitoring, investigations, and enforcement of the AI Act. There will potentially be several in each EU member state.

• The AI Office is responsible for overseeing and enforcing the provisions on general-purpose AI models, whereas the market surveillance authorities are responsible for overseeing and enforcing the provisions on AI systems (e.g., high-risk and transparency-requiring AI systems), as well as most other AI Act provisions.

What changes are being proposed?

The Commission proposes to “centralise oversight over a large number of AI systems built on general-purpose AI models” when the same provider develops both the general-purpose AI model and the AI system.

The proposed amendments to Article 75 would render the AI Office as the body responsible for monitoring and supervising compliance of AI systems that leverage general-purpose AI models. However, this would only apply when the general-purpose AI model and the AI system are developed and placed on the market or put into service by the same provider. In such scenarios, the AI Office would be “exclusively competent”, which means that the market surveillance authorities in the respective EU member states would no longer have a supervisory role. The AI Office would also have “all the powers of a market surveillance authority”.

This expansion in scope of the AI Office’s responsibilities does not apply to high-risk AI systems covered by an Annex I EU product safety law. Therefore, this change primarily impacts high-risk AI systems listed in Annex III and transparency-requiring AI systems regulated by Article 50 (where such AI systems leverage general-purpose AI models).

Finally, under this proposal, the AI Office would also have exclusive competence as the regulator of “AI systems that constitute or that are integrated into a designated very large online platform or very large online search engine” (as defined and regulated by the Digital Services Act).

What impact would this have?

The implied rationale behind this change is that the Commission does not think it makes sense for the AI Office to be responsible for overseeing providers of general-purpose AI models but not the AI systems developed, made available, and put into service by those same providers.

These changes would make the AI Office the supervisory authority for many of the most widely used AI systems worldwide. This is because most mainstream generative AI platforms, such as ChatGPT, Gemini, Claude, Grok, and Microsoft Copilot, are AI systems built on general-purpose AI models, with the same organisation being the provider of both the AI model and the AI system. Therefore, this would represent a meaningful increase in the relevance and prominence of the AI Office for AI Act oversight and enforcement, and it would also enable the AI Office to pursue investigations and enforcement action relevant for both general-purpose AI model and AI system compliance in a coordinated manner.

This would also mean that certain AI system providers would not be subject to regulatory investigations and enforcement action across multiple EU member states, which is possible if the law isn’t amended.

6. Proportionality for small mid-cap (SMC) enterprises

The AI Act provides an element of flexibility and proportionality for micro, small, and medium-size enterprises (SMEs), including start-ups. For example, the compliance penalties which SMEs can face are capped in the following way:

• 35 million EUR or 7% of total worldwide annual turnover (whichever is lower).

• 15 million EUR or 3% of total worldwide annual turnover (whichever is lower).

• 7.5 million EUR or 1% of total worldwide annual turnover (whichever is lower).

This contrasts with the “whichever is higher” penalty logic that applies for all other businesses (i.e., those which are not SMEs). In practice, this means that many non-SME businesses could face potential penalties into the billions of euros, whereas penalties for SMEs will always be capped as per the above.

Other ways in which the AI Act seeks to ease compliance burdens for SMEs include allowing SME providers of high-risk AI systems to provide the required technical documentation in a simplified way and providing SMEs with free access to AI regulatory sandboxes.

What changes are being proposed?

The first proposed change is to add legal definitions of SME and small mid-cap enterprise (SMC) to the AI Act. These are:

• SME: an enterprise which employs fewer than 250 people and which has an annual turnover not exceeding 50 million EUR, and/or an annual balance sheet total not exceeding 43 million EUR.

• SMC: an enterprise which employs fewer than 750 people and which has an annual turnover not exceeding 150m EUR or an annual balance sheet total not exceeding 129m EUR.

The second and more significant proposed change is to extend the flexibility and proportionality penalties afforded to SMEs to SMCs also. This means that SMCs would benefit from the same capped enforcement penalty regime as SMEs, significantly reducing their total potential penalty exposure in certain circumstances. SMCs that are providers of high-risk AI systems would also be able to provide the required technical documentation in a simplified manner.

What impact would this have?

Core threads from the Draghi report are woven throughout this proposal. The Commission will be hoping that easing regulatory compliance burdens and softening the enforcement environment for a larger pool of companies will make it easier for EU digital start-ups and scale-ups to grow, innovate, and compete internationally. Indeed, the Draghi report argued that “regulatory burdens” are particularly damaging for digital sector SMEs trying to rapidly scale up.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance and author of the forthcoming book, Fundamentals of AI Governance (2026).

On Wednesday 19 November 2025, the European Commission will publish its Digital Omnibus Package—a sweeping proposal to reform the EU’s digital legislative framework. The EU AI Act is in sharp focus, alongside the GDPR and a host of other flagship EU laws. This article explains how we got here, what’s at stake, and the potential changes being discussed in Brussels. It will be followed by in-depth analysis of the proposed changes, once they are published.

How did we get here?

The EU has consistently sought to shape global regulatory standards on digital technology, data, and AI. The EU has been open about its policy objectives of protecting citizens and promoting global regulatory convergence towards its high standards, in domains like data protection and privacy. And in some ways, it has succeeded.

With GDPR, for example, there has been a demonstrable “Brussels effect”. Many countries worldwide have enacted data protection laws of a similar flavour and corporations have prioritised aligning their internal frameworks accordingly. Despite the lack of U.S. federal privacy law (which renders the U.S. a global outlier), almost all major American enterprises are significantly impacted by the GDPR and have implemented dedicated privacy compliance programmes.

However, as EU laws covering digital governance have proliferated, the picture has become increasingly complex and convoluted. Keeping track of the EU’s regulations and directives covering the digital sphere is no mean feat.

CEPS and Kai Zenner’s dataset of EU digital sector legislation, updated in July 2025, lists 101 different laws, including the EU AI Act and the GDPR. The authors remark that there has been an “absolute explosion” of EU digital laws and that “even the best experts struggle to keep up with this torrent of legal and policy instruments”. To complement the 101 laws, there exists a far greater number of regulatory bodies covering digital issues.

Enterprises require large teams of people to make sense of these developments and determine how to be compliant, not least because obtaining deep expertise in just one legislative area takes years. In response, enterprises have also established a range of somewhat separate yet overlapping digital governance capabilities to address key legal obligations and requirements. This includes implementing enterprise AI governance, which has become a core priority in recent years due to both the EU AI Act and the surge in AI adoption. But you didn’t need me to tell you that…

For AI governance professionals, this regulatory complexity is especially acute, as AI systems frequently trigger multiple regulatory requirements simultaneously—from GDPR’s rules on automated decision-making and use of sensitive personal data, to the AI Act’s requirements for high-risk and transparency-requiring AI systems, cybersecurity mandates, copyright and intellectual property, and existing sectoral laws.

In recent months, discussion regarding simplification of the EU’s digital rulebook has intensified—both within the EU and externally. It is widely reported that EU officials and member states have faced sustained lobbying from U.S. industry and the Trump administration. There are even reports of President Trump considering imposing tariffs and sanctions on the EU and its officials, due to the perceived impact of EU digital laws on U.S. companies and citizens.

This pressure has been accompanied by increasingly vocal calls from segments of European industry that EU digital regulations need to be reformed, streamlined, simplified, to reduce compliance burdens. For example, in July 2025, dozens of companies—including industry heavyweights like Mercedes Benz, Deutsche Bank, and L’Oreal—signed an open letter to the EU urging for a two-year delay to the EU AI Act’s key provisions on general-purpose AI (GPAI) models and high-risk AI systems.

This is coming to a head on Wednesday 19 November, as the European Commission is set to publish its “Digital Omnibus Package” proposal. This will outline, for the first time, the comprehensive set of legislative amendments the Commission proposes, to simplify the EU’s digital rulebook.

What was the Draghi report?

Although this situation has evolved over several years, the September 2024 publication of the “Draghi report” on The Future of European Competitiveness was an important milestone in this story. Mario Draghi is the former president of the European Central Bank and former prime minister of Italy. This independent report, commissioned by the EU, is shaping aspects of the European Commission’s policy agenda.

The report presents the EU’s economic challenges, focusing on diminished competitiveness, weakening productivity, and slowing growth. Draghi frames the economic outlook as an “existential challenge” for the EU, arguing that it will be unable to finance its social model, achieve its environmental ambitions, and deliver the prosperous and fair society that represents its raison d’être if it fails to make radical changes.

The Draghi report outlines three action areas to “reignite growth”. These are:

1. Closing the innovation gap with the U.S. and China, especially in advanced technologies.

2. Forging a joint plan for decarbonisation and competitiveness.

3. Increasing security and reducing dependencies.

It is the first action area which is most relevant for our focus—EU AI Act simplification. The Draghi report argues that the EU’s complex digital regulatory environment impedes innovation and growth. It elevates “reducing the regulatory burden” as a core priority for transforming the EU’s economic prospects.

Examining the “innovation gap” between the U.S. and China on the one hand and the EU on the other, the report claims that innovative European companies attempting to scale up are “hindered at every stage by inconsistent and restrictive regulations”. This is why, the report argues, European tech entrepreneurs routinely seek to grow their businesses in the U.S. The report also cites that 55% of SMEs flag “regulatory obstacles and administrative burden” as their greatest challenge, arguing that these “regulatory burdens” are particularly damaging for digital sector SMEs.

The report criticises the “precautionary approach” taken by EU digital laws, including the EU AI Act. It specifically calls out the compute threshold for determining whether a general-purpose AI (GPAI) model poses “systemic risk”, noting that various frontier AI models already exceed the threshold, despite the EU AI Act’s nascency. It also highlights the increasing difficulty that companies face in navigating the various overlapping laws relevant for AI and the hundreds of regulatory bodies across the EU responsible for digital governance.

It is important to note that a vast array of other impediments and challenges are highlighted—from inadequate research talent pipelines to low investment in innovation commercialisation—as contributing factors to the EU’s competitiveness challenge. Therefore, my intention is not to claim that the Draghi report is all about digital “regulatory burdens”, as that would be misleading. It would be equally misleading to claim the Digital Omnibus Package results solely from the Draghi report. However, the complexity of the EU’s digital legislative framework has undeniably become a totemic issue, which the Draghi report shone a very bright light on.

Next, we turn our attention to the European Commission’s Digital Omnibus Package. Will it deliver the changes that Draghi seeks? And what could this mean for the EU AI Act?

What is the EU’s Digital Omnibus Package?

The Digital Omnibus Package is the European Commission’s much anticipated proposal to reform the EU’s digital legislative framework. It is due to be announced and published on Wednesday 19 November. This follows public consultations earlier in the year.

The proposal will feature meaningful amendments to some of the EU’s flagship laws and perhaps even the repeal of specific instruments. The Digital Omnibus Package is expected to focus on the EU AI Act, the GDPR, the ePrivacy Directive, the Data Act, and the NIS2 Directive—horizontally cutting across the core digital governance domains of AI, data protection and privacy, and cyber security.

The purpose of the Digital Omnibus Package is to simplify and streamline the EU’s digital legislative framework, to ease compliance burdens and cut costs for organisations (particularly startups and SMEs), promote the growth and competitiveness of European companies, and boost innovation. It is inevitable that the core threads from the Draghi report will be woven into the European Commission’s proposal.

Before engaging in any further analysis of the imminent proposal, several caveats are required.

First, the current discussion, including this article, is based on leaked documents, media reporting, speculation, and rumours. Brussels is a famously leaky city, so this is nothing new. However, until the European Commission officially publishes its proposal, we do not know exactly what the proposal consists of. At the time of writing, nothing official has been published.

Second, the Digital Omnibus Package that will be published later this week merely represents the European Commission’s initial proposal on this controversial set of issues. Therefore, even when we have the proposal, all we will have is the official starting position of one of the EU’s institutions. To amend EU laws in this way will require extensive trilogue negotiations and approval from the European Parliament and EU member states via the Council of the EU (the Council).

Third, the aforementioned trilogue negotiations—and the process of amending and repealing a suite of EU laws in this way—are bound to be lengthy, complex, and fraught with drama. Although the precise legislative instrument to operationalise the Digital Omnibus Package is yet to be confirmed, it is most likely to be an EU regulation (which is the same legal instrument as laws like the GDPR and the EU AI Act).

To amend existing EU laws via a new regulation, the EU’s “ordinary legislative procedure” must be followed. This necessitates dual approval from both the European Parliament and the Council, following trilogue negotiations where both institutions have several opportunities to amend and update the legislative proposal. On average, it takes the EU 19 months to agree new laws, from the initial Commission proposal to formal adoption.

Plainly speaking, what all this means is that:

• We don’t yet have an official proposal from the European Commission, merely leaks and media speculation.

• When we do, it will be subject to lengthy and fraught trilogue negotiations, the outcome of which is impossible to predict.

  • However, what can be predicted with a degree of certainty is that there will be a substantial difference between the European Commission’s initial legislative proposal and the final text that is voted on.

• Following this, formal approval will be required from both the European Parliament and the Council to pass any regulation that meaningfully amends existing EU laws.

• It is impossible to predict what the final legislative text will consist of and how long the negotiation and approval process will take.

• Finally, there is no guarantee that any legislative changes will be approved—although this does seem unlikely given the various points made above.

How could the EU AI Act change?

Caveats aside, the final part of this article will highlight some of the potential changes that are reported to be on the cards for the EU AI Act.

Although based primarily on leaks and tip-offs, recent media reporting nonetheless shines a light on what is being discussed in the EU’s corridors of power. Below is a list of the various potential EU AI Act changes that have been reported. A special shout out for the reporting from MLex’s Luca Bertuzzi (who is a must follow for AI governance practitioners), which this list is largely based on:

• Delaying the applicable date for the obligations for providers and deployers of transparency-requiring AI systems.

• Delaying the applicable date for the obligations for providers and deployers of high-risk AI systems.

  • These obligations apply from 2 August 2026, but this enforcement could reportedly be delayed for one year. This is partly due to delays in finalising technical standards that organisations can use to comply with these provisions.

• Scrapping the AI literacy obligation for organisations and shifting it to governments, regulators, and EU institutions.

• Centralising enforcement powers with the European Commission’s AI Office.

  • This could be done by designating the AI Office as the regulatory authority responsible for supervising AI systems based on GPAI models. This is partly driven by concerns regarding readiness and capacity of EU member state regulators (some of which have not yet been designated), as well as the complexity firms could face in engaging with many different regulatory bodies.

• Introducing smaller and more proportionate compliance penalties for ‘small mid-caps’ (defined as companies that employ up to 750 people and have an annual turnover of under €150 million).

  • This would complement and expand the scope of the existing proportionality for compliance penalties for SMEs.

• Removing the obligation for providers to register, in the EU’s public database, AI systems that are used in a high-risk domain which they have deemed and demonstrated are not high-risk due to the nature of the use.

It will be interesting to see which (if any) of these potential changes survive in the Digital Omnibus Package that the European Commission publishes on Wednesday. Given the speculative nature of this, I will not provide any further analysis on these potential changes in this article.

One final point to note is that the European Commission does have powers to amend or change aspects of the EU AI Act, via instruments called delegated acts and implementing acts. For example, the European Commission can modify the compute threshold for GPAI models with systemic risk via a delegated act.

Where these powers exist, the European Commission is able to drive changes without the need for the ordinary legislative procedure and formal approval from the European Parliament and Council. Although, there is always a degree of oversight from these institutions, who retain veto powers. However, the EU AI Act “simplification” changes discussed above extend far beyond the scope of the Commission’s powers to drive changes via delegated and implementing acts.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance.

This week’s newsletter presents, for the first time, my PROTECT Framework for Managing Data Risks in the AI Era © 2025.

Last week I published Part 1 of my 2-part series on the Top 10 Challenges for AI Governance Leaders in 2025. It focused on how the democratisation of AI, coupled with the sheer volume and rapid velocity of AI initiatives, is putting serious strain on the enterprise AI governance function. It outlined how, in response to these challenges, AI governance leaders must refine and update their risk-based approach and narrow the focus of their teams’ work, to avoid being overwhelmed, distracted, or neglecting the most serious risks.

Although I promised Part 2 this week, I ended up writing a full article on the fourth challenge: protecting confidential business data. This is a hugely important topic and there was simply too much to say. The beauty of having your own Substack is that you can follow whichever creative or intellectual direction most appeals to you, rather than rigidly sticking to prior plans. I hope that you will indulge my partial detour and that you find value in this article for your work. The “real” Part 2, covering challenges 5-10, will have to wait another week.

Challenge 4. Protecting Confidential Business Data

How can enterprises protect confidential business data when there is immense hunger to experiment with and use the latest AI applications that are released on the market?

The generative AI boom has amplified and exacerbated a plethora of data risks that all enterprises are exposed to. It has never been more important to ensure that your AI governance, cybersecurity, and information security frameworks are designed to, and capable of, mitigating these risks.

However, designing, implementing, and scaling robust controls that actually protect your organisation’s data and intellectual property requires precise understanding of what these risks are and how they are impacted by AI. That’s where my PROTECT Framework comes in.

The PROTECT Framework empowers you to understand, map, and mitigate the most pertinent data risks that are fuelled by widespread adoption of generative AI. Below is a high-level summary of the framework, followed by a detailed breakdown of each of the 7 themes.

PROTECT: Managing Data Risks in the AI Era

The PROTECT Framework focuses primarily on protecting (no surprises there) confidential business data from exposure, disclosure, and misuse—as well as associated data privacy and security risks fuelled by AI. It also outlines how organisations can use data in a compliant way, in the context of AI development, deployment, and use.

P - Public AI Tool Usage

R - Rogue Internal AI Projects

O - Opportunistic Vendors

T - Technical Attacks and Vulnerabilities

E - Embedded Assistants and Agents

C - Compliance, Copyright, and Contractual Breaches

T - Transfer Violations

The rest of the article breaks down each of the seven themes, highlighting both the core risks that enterprises are exposed to, as well as practical mitigations that can be implemented to manage and control these risks.

My forthcoming book, Fundamentals of AI Governance, provides a comprehensive visual overview of the PROTECT Framework, as well as a deep-dive on the Top 10 AI Risks impacting the modern enterprise. To secure a 25% discount during the pre-launch period, sign up at the link below.

Secure your 25% discount

P - Public AI Tool Usage

Enterprise risks: Use of publicly available AI tools is arguably the most severe data risk, because of how easy it is to do and how difficult it is to prevent. Simply put, there are thousands of publicly available AI tools that anyone can access via the internet, most of which are free or cheap. As well as mainstream generative AI chatbots like Claude, Gemini and ChatGPT, there are countless AI tools for creating presentations, managing email inboxes, transcribing meetings, and generating videos.

No matter how mature your enterprise AI capabilities are—and even if you are a frontier AI company— it is not going to be feasible to keep up with the latest and greatest AI tools and AI models that are released on the market each day, whilst also performing robust due diligence on AI vendors. Even with enterprise-grade licenses to mainstream generative AI services, you will not always get immediate access to the latest features included in the consumer version. This fuels immense hunger for employees to experiment with and use the most cutting-edge AI tools, irrespective of whether they are “internal” and approved or “publicly available” and unapproved.

This inability to keep pace with the market, coupled with a lack of awareness regarding the risks of using publicly available AI tools and the pressure that employees and teams are under to become “AI-first”, exacerbates the risks. Many employees may not understand the difference, from a data risk perspective, between using internal AI tools and public AI tools. But the risk is real. For example, when enterprise data is shared with publicly available AI tools, the organisation no longer has any control over what happens to it. This confidential business data could be used to train the AI models that power publicly available AI tools and in turn be disclosed, via future AI outputs, to competitors or malicious actors. It may even end up on the public internet—as we saw when various shared AI chat logs were indexed and publicly accessible online—or be retained indefinitely, as a result of court orders like the one OpenAI faced from the New York Court. Simply put, if you want to be in control of how your data and intellectual property is used, who has access to it, and for how long it is retained, your employees should avoid using publicly available AI tools.

Practical mitigations: Although shadow AI use is ubiquitous, there are various controls you can implement to mitigate this risk. My 3 Gs for Governing AI Democratisation offers a useful starting point:

Guidance: educate and train the workforce on the risks of using publicly available AI tools and the importance of protecting confidential business data.

Greenlight: provide access to secure, best-in-class AI tools, platforms, and capabilities that are approved for internal use and can process confidential business data.

Guardrails: implement guardrails—including technical and legal mitigations—to mitigate outstanding data risks that the use of internally approved AI tools entails. This can include scanning and blocking certain types of data from being uploaded as an input or generated as an output.

R - Rogue Internal AI Projects

Enterprise risks: Bypassing governance, especially when it happens at scale, creates compliance blind spots. Various risks emerge, and are difficult to manage, when teams develop and deploy AI systems, or procure AI solutions from vendors, without adhering to the mandatory AI governance, privacy, and cyber security processes.

In such scenarios, there is unlikely to have been any legal or compliance review, privacy assessment, or security evaluation. In turn, this means that genuine risks are unlikely to be understood, required documentation and artefacts may not have been produced, and robust mitigations will not be in place to address any important risks.

This increases organisational technical debt, which can lead to costly and burdensome efforts to retrospectively re-engineer non-compliant AI systems that are already deployed in production. Finally, it increases the likelihood that data is used without authorisation, and in a manner that constitutes a contractual breach or potential compliance violation.

In most cases, this does not happen because of malicious internal actors or intentional rule-breaking. Rather, it is more likely due to enthusiasm, competitive internal pressures and competing priorities, or a lack of awareness of internal governance processes and how to navigate them.

Practical mitigations: To mitigate the risk of rogue internal AI projects bypassing compliance checks, a proportionate degree of oversight must be applied to all AI projects. The level of governance scrutiny and oversight should flex in relation to the type of data being used. The use of sensitive personal data, confidential business data, or copyright protected material should entail more rigorous oversight, both at the project outset and throughout the AI lifecycle. Oversight should also be more rigorous for scenarios that involve sharing data with external vendors and the applications they provide.

The most important thing you can do is to make your AI and digital governance processes as easy to navigate as possible, by integrating different processes where possible, providing accessible guidance and support, and using automation to streamline processes and improve the user experience.

O - Opportunistic Vendors

Enterprise risks: Almost all enterprises must work with, and procure from, external organisations to progress with their AI ambitions. In the generative AI era, the trend is from build to buy. Whether it is leveraging pre-trained foundation models and generative AI chatbots, or working with vendors that provide bespoke AI products, exposure to third-party AI risk is unavoidable. In some cases, AI vendors and service providers may seek to use your confidential business data to train, develop, and improve their AI models and services—potentially without your explicit knowledge or consent.

The terms of service for many AI platforms and products are ambiguous or difficult to understand, and grant vendors broad rights over customer data. The key risk is whether your organisation’s data is used to train AI models that other customers can access and use. If so, competitors (or any other organisation) using that same vendor’s products and services may benefit from insights derived from your data. The risk is lower—or potentially fully mitigated—if your data is only used to train AI models and services that only your organisation has access to. This can enable you to benefit from feature improvements and customisation whilst mitigating data exposure and leakage risks.

Practical mitigations: Consider contractually prohibiting AI vendors from using your data (e.g., prompt, input, log, and output data), to train AI models and improve services that are accessible to other customers. This requires robust vendor due diligence, and a specific AI governance process pathway for AI procurement. It also requires clear guidance and training on third-party AI risks, acceptable data use terms, as well as template contracts and addendums that can be used across the business. Although the demand for AI vendors (and the products they provide) is high, the presence of opportunistic or shady operators in the market—and the immense value of the data they can obtain from enterprise customers—makes rigorous due diligence and contractual safeguards essential.

T - Technical Attacks and Vulnerabilities

Enterprise risks: AI systems introduce novel attack vectors that are linked to the distinct vulnerabilities of these systems. In particular, generative AI and agentic AI systems can be compromised and exploited, leading to confidential data—that was part of the AI model’s training, input, or output data—being extracted and stolen. Such data exfiltration is a known and widely documented AI vulnerability and can be caused by various attack methods. Prompt injection attacks, for example, are when an AI model is provided with malicious inputs (during inference) that are designed to manipulate and steer the outputs it generates, by jailbreaking model guardrails.

Simply put, the goal of prompt injection is to make the AI model do something that it is not supposed to. This includes, but is not limited to, data exfiltration, as well as reconstruction of training data. This risk of prompt injection is amplified with agentic AI, given the ability of AI agents to use tools and execute actions that can have a material impact (rather than “just” generate outputs for consumption).

Practical Mitigations: Although there are no foolproof mitigations against prompt injection attacks, there are nonetheless important steps you can take. Consider implementing AI system-level security controls and guardrails including input validation, prompt sanitisation, output filtering, and incident monitoring and detection. Also, for high risk applications, conduct red-teaming and adversarial testing. Cybersecurity best practice emphasises the importance of multiple overlapping security layers. However, no technical controls can fully prevent prompt injection or data exfiltration. Therefore, carefully control who has access to sensitive AI systems, what data they can access, and what actions agentic AI can execute.

E - Embedded Assistants and Agents

Enterprise risks: The AI assistants and agents that are increasingly embedded in the core workplace software we all use pose novel data risks. In particular, these embedded AI tools can inappropriately disclose and disseminate data to people and groups that were not supposed to have access to it. For example, a personalised AI assistant that summarises your emails and daily tasks can analyse wider organisational data that you have access to, such as document libraries and shared calendars. If that data has not been protected with appropriate file sharing permissions—and moreover has erroneously been made available to the entire organisation—then elements of it may be surfaced to you via your handy AI assistant.

Although the root cause of this is often inappropriate or inadequate file sharing permissions, AI significantly increases the likelihood of such data being shared with the wrong person. It also provides malicious internal actors with a powerful tool for mischief.

AI meeting assistants and note-taking applications are of particular concern. It is commonplace to join calls with external organisations, only to find that a random AI bot is also on the call, recording and transcribing everything that is said. From an enterprise perspective, this is akin to uploading all of this information into a publicly available AI tool (which poses similar risks to those outlined in ‘Public AI Tool Usage ’), unless you have assurances from the external organisation regarding the technology they are using and how it processes your data. Furthermore, be wary of individuals having access to AI meeting recordings and transcripts of parts of the discussion they were not part of.

Increasingly autonomous agentic AI systems exacerbate these risks. In order to effectively determine the best course of action and use tools to execute tasks, LLM-based AI agents will need to mine, retrieve from, and synthesise myriad enterprise data sources. Establishing appropriate access controls, and maintaining AI agent audit trails, will become increasingly complex yet important.

Practical mitigations: Robust data governance and data risk management is critical to ensuring your increasingly autonomous AI tools do not cause havoc. Ensuring appropriate file sharing permissions for sensitive and critical data sources is paramount, given the ways in which AI agents can mine through your document libraries and other repositories, as well as the obvious value this capability provides. Also, when deploying agentic AI, start with lower risk use cases, applications, and data sources. Furthermore, apply the principle of least privilege, to ensure that AI agents only have access to data that is necessary for their tasks.

C - Compliance, Copyright, and Contractual Breaches

Enterprise risks: Data science, AI, and business teams are under significant pressure to leverage AI to deliver value for the business. To do so, they require seamless access to vast amounts of high-quality, business-critical data. However, the increasingly stringent data and AI regulatory landscape—particularly in the EU—creates numerous compliance risks when using data for AI activities. It is therefore crucial to have robust controls in place to prevent unauthorised or non-compliant use of data. The most important regulatory and legal domains to consider are:

Privacy and data protection: Privacy and data protection laws restrict the way in which personal data—in particular sensitive personal data—can be used. For example, under the EU’s GDPR, you must have a lawful basis to process personal data. Personal data processing is therefore only lawful if at least one of the following lawful bases apply: i) consent, ii) performance of a contract, iii) legal compliance, iv) protection of vital interests, v) performance of a task in the public interest, or vi) legitimate interests. Therefore, just because you have access to personal data, does not mean you are permitted by default to use it to develop or deploy AI.

Copyright and intellectual property: Organisations must be cautious when using external data for AI development and deployment, as it is often copyright protected. Different data sources come with different licenses, terms, and conditions. This cautiousness must extend to “everyday” employee use of generative AI tools—in particular their prompts and document uploads.

Contracts: Your organisation may have access to data that another organisation provided in the course of an engagement, such as the use of a product or service you provide, that is governed by a bespoke legal agreement. Therefore, you must protect and handle that data in accordance with the applicable legal agreement.

AI-specific laws: Finally, AI regulations like the EU AI Act typically include provisions that stipulate how data should be used in the context of AI activities. For example, the AI Act requires providers of high-risk AI systems to use high quality, accurate, and “representative” training, validation, and testing data sets, in order to mitigate bias risks and promote reliable AI performance.

Practical mitigations: The legal and regulatory domains outlined above are vast; comprehensive risk mitigation across all of them is beyond the scope of the PROTECT Framework. However, the overarching principle is that you must embed proportionate governance and oversight throughout the AI lifecycle, to prevent non-compliant or unauthorised data use. This means legal and compliance review must occur at critical stages, including before data is sourced, before AI models are trained, and before AI systems are deployed in production or released on the market. As ever, this governance must be complemented and reinforced by company-wide and role-specific training. When these governance checkpoints are bypassed—as discussed in the ‘Rogue Internal AI Projects’ theme—the aforementioned data-related compliance and legal risks materialise.

T - Transfer Violations

Enterprise risks: Leveraging cloud-based AI services, such as platforms for accessing and using foundation models, almost always involves international data transfers. Given that AI processing is rarely confined to one jurisdiction, navigating international data transfer compliance is an important part of AI governance.

Privacy and data protection regimes worldwide restrict the way in which personal data can be transferred internationally. Under the GDPR, organisations can transfer personal data freely from the EU to entities in a non-EU country if there is an EU adequacy decision in place. An adequacy decision is the EU’s way of “protecting the rights of its citizens by insisting upon a high standard of data protection in foreign countries where their data is processed”. 15 jurisdictions are recognised as “adequate” by the EU. This includes the U.S.—but it only applies to commercial organisations participating in and certified under the EU-U.S. Data Privacy Framework. If there is no EU adequacy decision, alternative legal safeguards must be put in place (e.g., Standard Contractual Clauses), before personal data can be transferred from the EU to the non-EU jurisdiction.

On a separate note, the U.S. “Bulk Data Transfer Rule” prohibits or restricts organisations from transferring “U.S. sensitive personal data” and “government-related data” to “countries of concern”, including China, Cuba, Iran, North Korea, Russia, and Venezuela. The Rule was issued by the Department of Justice in January 2025.

Practical mitigations: Although the above was far from an exhaustive overview of international data transfer regulations, the key point is that you must implement specific mitigations—as required by the applicable law—prior to transferring personal data across borders. For example, before onboarding U.S. AI vendors and service providers, verify whether they are certified under the EU-U.S. Data Privacy Framework, and implement Standard Contractual Clauses if not.

Transfer violations can result from the governance failures outlined earlier—such as onboarding AI vendors without proper due diligence and robust contracts, or deploying AI systems in production without completing privacy impact assessments. International data transfer-specific mitigations include mapping your organisation’s data flows to understand and track where data is processed, which vendors have access to it, and what sub-processors they may share that data with. Finally, complete transfer impact assessments as part of your privacy impact assessment process.

Hey 👋

I’m Oliver Patel, author and creator of Enterprise AI Governance—which today celebrates its 30th edition! Thanks for all your support and engagement on this journey!

This two-part series provides my personal view, from the enterprise AI governance coalface, on the top 10 challenges facing AI governance leaders today.

This is not a theoretical, policy, or legal analysis. Rather, it is a contemporary examination of the messy and relentless reality of being tasked with leading AI governance in an enterprise context—where AI is being adopted at scale and promoted as integral to the future competitiveness and relevance of the organisation.

The purpose of this two-part series is to guide AI governance professionals with practical insights on how to tackle some of the most pressing challenges we face. Below are the 10 challenges this series covers in detail. Part 1 covers challenges 1-3 (as they are inextricably linked) and part 2 (coming next week) covers challenges 4-10.

The lack of an established blueprint for enterprise AI governance, and the immense, interdisciplinary challenges we face each day, is exactly why I am writing my upcoming book, Fundamentals of AI Governance. It is packed full of practical tips and strategies for enterprise AI governance leaders, including a visual deep dive on each of these 10 challenges. To secure a 25% discount during the pre-launch period, sign up at the link below.

Secure your 25% discount

Top 10 Challenges for AI Governance Leaders in 2025

1. The ‘democratisation dilemma’. How to maintain robust oversight and promote compliance when the ability to develop, deploy, and use AI is democratised and widely accessible?

2. Volume and velocity. How to keep up with the sheer volume and rapid pace of enterprise AI initiatives, whilst cutting through the noise and deploying finite resources and expertise on the highest value work?

3. Refining the risk-based approach. How to respond to the AI risk ‘vibe shift’ and effectively target governance on the relatively small proportion of AI systems and use cases that could pose significant risks?

4. Protecting confidential business data. How to protect confidential business data when there is immense hunger to experiment with and use the latest AI applications that are released on the market?

5. Ongoing vendor due diligence and oversight. How to move beyond ‘paper-based’ vendor due diligence and apply continuous oversight on the performance, trustworthiness, and safety of externally provided AI applications?

6. AI engineering: building with foundation models. How to determine your rights, responsibilities, and liabilities—as well as the novel risks and tangible mitigations—when building AI systems with foundation models provided by external organisations?

7. Open-source AI model oversight. How to effectively govern the widespread access and use of open-source AI models, to safeguard your organisation from legal, compliance, and cyber security risks, whilst promoting innovation?

8. Embedding compliance by design. How to build AI systems that promote compliance by design and default, to make it seamless for your workforce to do the right thing?

9. Agentic AI governance: taking the human out of the loop. How to promote responsible, meaningful, and empowered human oversight of AI, when the fundamental goal of agentic AI is to take the human out of the loop?

10. Digital governance silos and inefficiencies. How to effectively streamline and integrate your disparate digital governance and risk management processes and capabilities, to improve the user experience and accelerate AI innovation, whilst also strengthening your compliance posture?
    

    Thanks for reading Enterprise AI Governance! Subscribe for free to receive new posts and support my work.

1. The ‘Democratisation Dilemma’

How to maintain robust oversight and promote compliance when the ability to develop, deploy, and use AI is democratised and widely accessible?

The launch of ChatGPT in November 2022, and the subsequent generative AI boom, was a watershed moment. Not only because of fundamental advances in model architecture or training and inference techniques, but because powerful generative AI applications were now, for the first time, at the fingertips of all employees. By some user adoption measures, ChatGPT was the fastest-growing consumer application of all time. Fast forward nearly three years, and it is evident that the democratisation of AI—and its widespread accessibility and availability—is perhaps the most important development impacting the enterprise AI world.

Things have moved on from everyone merely having access to powerful chatbots for simple personal assistance and productivity tasks. The ability to customise, use, and share generative AI applications, for scalable, business critical use cases—and even to build AI powered-tools for production deployment in business workflows without any hard technical skills—is significantly increasing the volume of AI activities and use cases for which oversight is often lacking.

For example, non-technical employees can use no-/low-code agent builder platforms for “drag-and-drop” creation of domain specific chatbots and assistants that can retrieve and surface information, call APIs, execute tasks, and augment and automate important business workflows. Similarly, non-technical employees can also build and scale custom GPTs and generative AI ‘projects’ that are augmented with vast amounts of data and documentation, and optimised to perform in a particular way, via detailed prompts and instructions

All major AI product releases highlight that the trend towards democratisation is accelerating. And it is safe to assume that what the average non-technical employee can do with AI will only become more sophisticated over time.

This compels AI governance leaders to reevaluate how to govern the development and use of AI. Up until now, most organisations distinguished between the AI models and AI systems that their data science and engineering teams build, for production deployment, and the everyday use of generative AI applications by the wider workforce. With respect to the latter, AI governance frameworks do not typically require AI governance and risk assessments, or the implementation of lifecycle controls and risk mitigations. Aside from following the AI usage policies and guidelines, employees are generally free to use approved generative AI applications for their personal work without oversight.

However, this position won’t hold for much longer. For example, if anyone can use a no-code AI platform to build a high-risk CV screening tool that dozens of other employees can use, then, given the compliance responsibilities and ethical implications, this cannot be considered an everyday use of AI warranting no formal governance.

Remember, under the EU AI Act, if your organisation uses a general-purpose AI system for recruitment and candidate selection, and it was not intended to be used for this purpose, you could inadvertently become the provider and the deployer of a high-risk AI system—without even realising.

Practical solutions: the three Gs for dealing with AI democratisation

• Guidance: Educate all employees on their responsibilities and the way in which their seemingly everyday use of AI can give rise to compliance obligations. Furthermore, ensure that any AI use case or AI system that has the potential to be classified as a high-risk AI system under the EU AI Act, or even a potentially prohibited AI practice, undergoes rigorous AI governance assessment and review, irrespective of the AI platform, technology, or approach used to develop and deploy it. Remember, the law is technology neutral.

• Guardrails: Implement guardrails and technical safeguards into the AI platforms that are widely available and “democratised”, so that certain types of AI use cases cannot be pursued or progressed to production (e.g., those that could be prohibited), and to enable potential compliance risks and incidents to be monitored, detected, and flagged on a continuous basis. This could involve monitoring specific content in prompts, documentation, and model outputs to flag potential violations or data-related risks.

• Greenlight: Make it clear precisely which AI platforms and tools are available and approved for use, as well as which data sources, AI-powered workflows, and patterns have been approved for production use. This enhances organisation-wide understanding of what is permissible, effective, and safe, thereby enhancing your AI governance posture without slowing the business down.

2. The Volume and Velocity Challenge

How to keep up with the sheer volume and rapid pace of enterprise AI initiatives, whilst cutting through the noise and deploying finite resources and expertise on the highest value work?

The volume and velocity challenge is directly linked to, and amplified by, the “democratisation dilemma”. The easier it becomes to develop, use, and scale AI applications for business-critical use cases, the more AI use cases there are to govern.

However, surging AI volume and velocity has other causes, such as increased investment in AI capabilities across the board, strong incentives for corporate leaders to demonstrate successful AI adoption, reduced barriers to entry due to the widespread availability of foundation models, rapid advances in AI engineering techniques to capitalise on these models, and technology vendors introducing myriad AI capabilities into every product, platform, and offering.

In this context, AI governance functions are struggling to keep up with the sheer volume of AI initiatives and use cases which they are tasked with assessing, reviewing, monitoring, and approving. Furthermore, the rapid pace at which this volume is increasing, combined with the ever-changing nature of the AI use cases and their underpinning technology— makes it even harder to know how to tackle this problem and structure your AI governance function’s workload.

There are three common symptoms of this challenge, all of which are detrimental to long-term AI governance success.

First, the volume challenge leads to practitioners with expertise on AI ethics, regulatory compliance, and broader legal or technical issues spending excessive time on lower value work, such as processing and reviewing inherently low-risk AI use cases, or providing similar advice repeatedly.

Second, the velocity challenge means that the nature of AI use cases and AI technology is changing fast. Risk assessment questions, lifecycle control frameworks, and training and guidance can quickly become outdated. For example, risk assessment and documentation artefacts designed for generative AI will likely be missing the mark for the new agentic AI systems that are being developed and prepared for production.

Finally, increased ‘demand’ on the AI governance function means more noise and more potential distractions. This can lead to mission drift and not prioritising the most important work that is essential to protect the organisation.

Practical solutions: the three As for handling AI overload

• Automate: Leverage automation and AI to speed up the process of triaging, classifying, and assessing AI use cases, as well as assigning governance and oversight pathways and reviewers (based on the inputs provided in AI governance assessment submissions). Given the volume, it may not be feasible to stick with manual human review of every single submission. Therefore, leveraging automation enables smarter allocation of work to AI governance experts. However, assurance and quality control processes should be implemented to ensure the automation is effectively serving its purpose. It should also be implemented in parallel to the human-led process, at least initially.

• Artefacts: Develop and distribute reusable governance artefacts for common AI patterns and applications—such as chatbots, research agents, and document processing. When the twentieth team wants to build a document assistant bot, they need not start from scratch. Pre-approved patterns, risk mitigation control sets, and documentation templates are win-win, as they support the business and free up AI governance SME time for higher-value work.

• Adapt: Dealing with velocity requires constant and targeted adaptation. Continuously and periodically review and assess whether your policies, risk assessment processes, guidelines, and artefacts are fit for purpose, making focused and impactful changes where necessary.

3. Refining the risk-based approach

How to respond to the AI risk ‘vibe shift’ and effectively target governance on the relatively small proportion of AI systems and use cases that could pose significant risks?

Democratisation is making it easier for anyone to develop and deploy AI, which is challenging the notion of what type of AI activities do and do not require oversight and governance. It is also a chief contributor to the volume and velocity challenge, which risks overwhelming enterprise AI governance teams, unless they effectively adapt and restructure their work.

These three trends—democratisation, volume, and velocity—are also contributing to an AI risk ‘vibe shift’, which many AI governance leaders are noticing and impacted by.

The use of AI is now ubiquitous and embedded in the daily lives of millions of people and workers worldwide. A Harvard Business Review study from April 2025 found that “therapy and companionship” was the top generative AI use case. A more recent study from economists at OpenAI and the National Bureau of Economic Research found that “practical guidance” is the most common conversation theme for ChatGPT, with the top three topic areas being 1) tutoring or teaching, 2) “how to” advice, and 3) health, fitness, beauty, and self-care.

As AI adoption increases, the perception of AI risk is changing. The common everyday use of AI is undoubtedly impacting how its risks and potential dangers are perceived. For many in the corporate world, it may be hard to connect more abstract risk themes, such as bias, explainability, and safety, with their everyday use of AI, which could feel merely assistive and somewhat harmless. Furthermore, if the organisation you work for is yet to suffer or be impacted by a major AI incident, and it is not an issue that is much discussed by the company leadership, this can also make the importance of AI governance feel harder to grasp.

But this is not just about perception. As AI adoption increases, the actual proportion of fundamentally low risk and non-material use cases is also likely increasing. Although this will vary for each organisation, the more people that use AI, and the more it is deployed to augment, optimise, and automate the vast range of back-office processes that, on balance, probably do not have the potential to materially impact people’s lives, the more important it is for AI governance leaders to focus their attention and resources on what matters most.

AI governance practitioners need to be alert to the resulting ‘vibe shift’—as well as the actual changes in the nature and distribution of AI use cases and their inherent risk levels, that are being driven by these trends.

As I have always said, AI governance does not mean governing all AI. You need to be proportionate, pragmatic, and risk-based in your approach. Trying to wrap your arms around anything and everything just because it has ‘AI’ in it is futile.

A better path forward is to refine and reevaluate your risk-based approach to AI governance, including how you classify the risk level of AI systems and AI use cases, and deploy resources and mandate governance and oversight accordingly.

This refinement needs to account for the fact that ostensibly everyday uses of AI can give rise to high-risk use cases, but also that most AI use cases and initiatives should probably not be classified as high-risk, irrespective of their development techniques and underpinning technology.

This is because as AI adoption surges, the volume of use cases and activities is becoming almost impossible to manage. Therefore, it’s never been more important to filter out the noise with razor sharp triaging, risk classification, and prioritisation.

In sum, to govern AI effectively in 2025, it is crucial for AI governance leaders to adapt to the societal and corporate ‘vibe shift’ that is underway. We must be targeted, selective, and proportionate in our approach, to stay credible and focused on what really matters. Structure your approach so that a large majority of your team’s time and invaluable expertise is spent on the small proportion of AI activities that pose significant risks.

Practical solutions: the three Ps for refining your risk-based approach

• Prioritise: Lead the conversation to redefine exactly what constitutes high-risk AI in your organisation in 2025. Instead of navigating opaque vibe shifts, formally agree and document your leadership’s risk appetite and risk-based approach to AI, in light of all the themes covered above. Most AI use cases don’t warrant intensive governance—pretending otherwise will undermine your credibility. Once you have done this, ensure your teams are spending most of their time on what is truly high-risk.

• Proportionate: Apply governance intensity and scrutiny that matches the actual risk level. The rigour applied to and experts involved in the AI governance review, the amount of technical documentation required, the lifecycle controls that must be implemented, and the level of continuous monitoring and governance oversight must all flex, depending on the nature of the AI system or AI use case. This ensures your governance effort scales with impact and risk, not just with AI adoption.

• Prove: Combat the vibe shift head-on by making your catches and wins visible. If you prevent an AI system from being deployed that poses potential ethical, legal, compliance, or reputational risks—or if a similar incident materialised in another organisation—document and communicate this to the right people in a subtle yet informative manner. This could be packaged up as periodic AI governance impact reports. If senior stakeholders can’t see the fires you’re preventing, they may question why you need the fire extinguisher.

  ---