A More Realistic View of AI Risk

This is the first report to use the Risk Index v2.0 framework. The upgrade is designed to answer two simple questions more clearly: how dangerous could a model become as its capabilities grow, and how much of that danger remains after its safeguards are taken into account?

Compared with the previous version, v2.0 better reflects the possibility that risk can accelerate as models become more capable. It also looks beyond whether a model refuses an ordinary harmful request by testing how well safeguards hold up against jailbreaks or deliberate tampering. Two “Yellow Lines” make the results easier to interpret: the Capability Yellow Line signals that a model could significantly increase severe-harm risks without safeguards, while the Risk Yellow Line signals that high risk remains even with existing safeguards in place.

The underlying evaluations have also been refreshed with more realistic tasks and advanced jailbreak testing, helping the Platform track both what frontier models can do and where their protections may fail.

The report covers 47 frontier models released by 13 leading AI companies from 2025 Q3 to 2026 Q2, including GPT-5.5, Claude Opus 4.8, Grok 4.3, DeepSeek V4 Pro, Qwen 3.7 Max, Doubao Seed 2.1 Pro, HY 3.0 Preview, ERNIE 5.1, MiniMax M3, Kimi K2.6, GLM 5.2, and MiMo V2.5 Pro.

Key Findings

The report highlights seven key findings:

1. Cyber, biological, and loss-of-control Risk Indices have risen severalfold in less than a year, with multiple models in each domain now exceeding the Capability Yellow Line

Under Risk Index v2.0, the average Risk Index of evaluated models rose rapidly in less than a year: 4.4x in cyber offense, 6.6x in biological risks, and 2.4x in loss-of-control.

Multiple models have already crossed the Capability Yellow Line: 4 models in cyber offense, 22 in biological risks, and 12 in loss-of-control. Crossing this line means that, without safeguards, a model would significantly increase severe-harm risk relative to non-AI baselines.

Note: 1. In the current v2.0 framework, Risk Indices have not yet been calculated for chemical risks or harmful manipulation, so this section focuses on cyber offense, biological risks, and loss-of-control.
2. Because Risk Index v2.0 makes substantial changes to both the benchmark suite and metric calculation, values in this report should not be compared item by item with those from v1.0 or v1.5. They are better used to compare models, quarters, and domains within the v2.0 framework.

2. Model risk profiles are diverging: Gemini 3.1 Pro Preview has the highest biological and loss-of-control Risk Indices, while DeepSeek V4 Pro shows more pronounced cyber misuse risk

Gemini 3.1 Pro Preview has the highest overall Risk Indices in biological risks and loss-of-control, and both have crossed the Risk Yellow Line. Unlike the Capability Yellow Line, crossing the Risk Yellow Line means that, even with existing safeguards, the model’s residual risk still reaches the high-risk boundary.

DeepSeek V4 Pro has the highest Risk Index in cyber offense, and its loss-of-control Risk Index is second only to Gemini 3.1 Pro Preview.

The GPT family is rising quickly in cyber offense and loss-of-control, and has crossed the Risk Yellow Line in biological risks.

Kimi, MiniMax, Qwen, Doubao, and other families are also rising quickly in cyber offense and biological risks; in biological risks, they have already crossed the Risk Yellow Line.

Note: This evaluation does not include Claude’s strongest models, Fable/Mythos 5.

3. Proprietary models continue to dominate the capability frontier across multiple domains, while open-weight models are generally weaker on safeguards for cyber, biological, chemical, and manipulation risks

Proprietary models still dominate the capability frontier across domains, with GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro Preview, and other proprietary models maintaining a capability lead in multiple domains.

Across the four misuse-risk domains of cyber offense, biological risks, chemical risks, and harmful manipulation, open-weight models have substantially lower Safety Scores than proprietary models.

In loss-of-control, Safety Score distributions are similar for open-weight and proprietary models, but some proprietary models, such as Gemini 3.1 Pro Preview and Grok 4, have notably lower Safety Scores. These models pose higher risk because they combine high Capability Scores with low Safety Scores.

4. Leading models’ long-horizon vulnerability exploitation and penetration capabilities are improving quickly, while safeguards such as prompt-injection defenses are regressing

In less than a year, the top model score rose by 68% on CyBench and 35% on CVE-Bench, indicating rapid progress by frontier models on autonomous, long-horizon vulnerability exploitation and penetration tasks.

At the same time, cyber safeguards have not improved in step. The average Safety Score fell by 3% in the latest quarter, and latest-quarter models’ average performance on prompt-injection defense (PromptInjection) regressed.

5. More than ten models now exceed human-expert levels in wet-lab troubleshooting and sequence understanding, but basic safeguards such as biological refusal have not improved in step

On capabilities, 22 models now exceed human-expert levels in wet-lab troubleshooting (BioLP-Bench), and 11 models exceed human-expert levels in sequence understanding (SeqQA). Frontier models’ biological reasoning (FrontierScience-Bio) and bioinformatics capabilities (BixBench) continue to improve.

On safety, the average Safety Score is broadly unchanged from the previous quarter, and basic biological refusal (BiologicalHarmfulQA) showed no clear improvement in the latest quarter.

6. The latest-quarter models did not set a new high in loss-of-control risk, but weak honesty and covert influence over users still reveal safety gaps

Compared with the rapid growth seen last quarter, models released in the latest quarter did not set a new high in the loss-of-control Risk Index, and none reached the Risk Yellow Line.

On capabilities, self-replication (Self-Proliferation), self-improvement (MLE-Bench), situational awareness (SAD-mini), and related capabilities did not set new highs.

On safety, loss-of-control safeguards still show no clear improvement: model honesty (MASK) remains weak overall, and some models still show a pronounced tendency to covertly influence users (DarkBench).

7. Under advanced jailbreak attacks, average safety falls sharply across all misuse-risk domains

After jailbreak red-teaming attacks are added, frontier models’ average safety scores fall from 78.2 to 8.9 in biological risks, from 90.5 to 29.2 in cyber offense, from 83.7 to 53.1 in chemical risks, and from 87.8 to 36.4 in harmful manipulation.

Jailbreak resistance varies greatly across models. Claude Opus 4.8 maintains an average refusal rate of 69.1% under red-team attacks, compared with just 5.8% for Hunyuan T1 (250711) under the same conditions.

Note: Jailbreak red-team attacks apply only to misuse-risk domains, not to loss-of-control risk.

See the full report for detailed monitoring results.

Recommendations for Stakeholders

Based on these findings, the report makes the following recommendations:

• For model developers: Pay close attention to your models’ Risk Indices, especially whether they cross the Risk Yellow Line. For higher-risk models, prioritize improving Base Safety, Jailbreak Safety, and loss-of-control safety. If better safeguards alone cannot sufficiently mitigate risk, consider reducing high-risk capabilities and establish clear risk thresholds, mitigation measures, and release policies.

• For AI safety researchers: Continue exploring more effective methods for capability elicitation, red-team attacks, prompt injection, and multi-turn manipulation to accurately assess the upper bounds of model capability and lower bounds of safety. At the same time, develop more effective model hardening, dangerous-capability removal, and risk mitigation approaches suited to open-weight models.

• For policymakers: The report identifies early warning signs in cyber offense, biological risks, and loss-of-control. Strengthen ongoing monitoring and risk analysis of pre-release evaluations and mitigations, and adopt differentiated governance based on model capability, safety, and open-weight or proprietary distribution.

Concordia AI will continue using the Frontier AI Risk Monitoring Platform to track risk trends among the world’s most advanced AI models and provide data to support the development of safe and trustworthy AI.

A Searchable Database for AI Risk Evaluations

Concordia AI also launched the Frontier AI Risk Benchmark Database, bringing together more than 200 benchmarks—standardized evaluations used to test AI systems—published between 2023 and 2026. Information that was previously scattered across research papers, code repositories, datasets, and model reports can now be searched and compared in one place.

The Database organizes benchmarks by risk area, evaluation purpose, and availability. It also connects them to real-world risk scenarios, helping users understand not only what a benchmark measures, but how it contributes to a broader assessment of AI risk. This provides a more transparent foundation for the monitoring results in this report and a practical resource for researchers, model developers, and policymakers.

Explore the Benchmark List and Risk Model.

References

• Frontier AI Risk Monitoring Platform

• Frontier AI Risk Monitoring Report (2026 Q2)

• Risk Index v2.0 upgrade details

• Frontier AI Risk Benchmark Database

The Forum brought together around 30 distinguished experts and practitioners from around the world. Speakers included Turing Award winner Yoshua Bengio, Alondra Nelson (former acting director of the White House Office of Science and Technology Policy), Amandeep Singh Gill (United Nations Secretary-General’s Envoy on Technology), Wan Sie Lee (Cluster Director for AI Governance and Safety at Singapore’s Infocomm Media Development Authority), ZHOU Bowen (周伯文) (Director of Shanghai AI Lab), and XUE Lan (薛澜) (Dean of Schwarzman College at Tsinghua University and Counsellor of the State Council), alongside former researchers from OpenAI, Google DeepMind, and Anthropic, and senior experts from Alibaba, the Frontier Model Forum, Hugging Face, and the China Academy of Information and Communications Technology (CAICT).

The half-day event was organized around three themes:

• Theme 1: Frontier AI Risk Monitoring: From Scientific Understanding to Practical Tools

• Theme 2: The Agentic AI Era: Full-Lifecycle Safety Governance and Risk Intervention

• Theme 3: Frontier AI Risk Governance and Global Perspectives

Around 300 audience members joined in person, with over 300,000+ views of the livestream as of July 19.

In addition to this summary of the 2026 WAIC Frontier and Agentic AI Safety Forum, we have also compiled key AI safety updates from WAIC as a whole in a special edition newsletter.

Subscribe now

Videos and key takeaways from the forum

Concordia AI founder and CEO Brian TSE (谢旻希) — Opening remarks

Brian opened the forum with the observation that “AI is moving from answering questions to taking actions.” Once agents operate across environments and improve themselves, traditional safety paradigms cannot contain the risks of misuse and loss of control. He framed the forum around three gaps. The first is visibility: we cannot yet see risks clearly, so the question is how to turn abstract frontier risks into measurable indicators and a working early-warning system. The second is control: monitoring has to become intervention, with real-time oversight of agents in operation and an emergency response system that can act when something goes wrong. The third is coordination: frontline monitoring and response practice needs to become shared technical standards and international agreement.

Theme 1 — Frontier AI Risk Monitoring: From Scientific Understanding to Practical Tools

Turing Award Laureate Yoshua Bengio — International AI Safety Report 2026

In his keynote speech, Yoshua Bengio, Turing Award laureate and Chair of the International AI Safety Report, shared his concerns about the unprecedented pace of AI progress and its implications for society. Bengio highlighted that AI capabilities are advancing rapidly, particularly in reasoning, coding, science, and autonomous agent capabilities, while current safety measures are not keeping pace. Drawing on findings from the International AI Safety Report, supported by 30 countries, the EU, OECD, and the UN, he emphasized the need to address emerging risks under uncertainty and avoid a race to the bottom driven by competition. He also discussed the unique safety challenges of open-weight models, including the difficulty of reversing deployment decisions, removing safeguards, and evaluating risks once models are widely shared. Bengio called for stronger scientific evaluation methods and precautionary approaches to guide policy decisions under uncertainty.

Panel: Scientific Understanding and Boundaries of Frontier AI Safety

The panel discussed which frontier risks are already visible, whether science can keep pace with them, and where practical guardrails are needed. A recurring theme was that the risks under discussion are not hypothetical futures, but are already present.

• Alondra Nelson (former Acting Director, US White House Office of Science and Technology Policy) argued that safety cannot be solved through technical engineering alone. She observed that models now operate in what she calls an “algorithmic surround” — layers of algorithmic systems interacting at once — and stressed the importance of understanding its societal implications. She noted that while some things about AI models genuinely cannot be known, she has been tracking how AI developers tactically invoke “not knowing” in ways that can be disempowering for AI governance efforts.

• Mark Nitzberg (Executive Director, UC Berkeley Center for Human-Compatible AI) cautioned that models are becoming so capable that they exceed our ability to do sufficient testing. He expected more models to be capable of recognizing when they are being tested and behaving accordingly, meaning unacceptable capabilities may only become apparent after deployment. He highlighted his work through the International Association for Safe and Ethical AI to build scientific consensus on behavioral red lines (such as prohibitions on enabling weapons of mass destruction) and to translate that consensus into a signed agreement by the end of the year.

• JIN Yaochu (金耀初) (Founder, Trustworthy and General AI Lab, Westlake University) warned that most people (particularly non-experts) trust AI too much. He observed that was a shift from when he founded his lab three years ago. At that time, his concern was the inverse: whether people would be able to trust AI. He stressed both the need to technically ensure that AI systems are as safe as possible, and educating users about the vulnerabilities of AI systems. He also spoke about his work on collective and embodied AI systems, noting that even when each component looks safe, unexpected risks can emerge when they are combined.

• HU Xia (胡侠) (Lead Scientist, Shanghai AI Lab) stressed that AI misuse is already happening. He cited a case in which a high school student used large language models to attack a website and take it offline for several days. Those kinds of attacks previously required significant expertise and training. In addition to cyber and CBRN risks, he highlighted that he had observed increasing resource-seeking and power-seeking behavior in AI systems. He cautioned that technical tools to prevent those types of behavior are not yet ready. He made the case for treating AI safety as a full-stack issue, and connecting problems scattered across hardware, software, foundation models, and applications. Looking at these components as a whole system could help identify cascading risks.

Launch: Frontier AI Risk Management Framework 2.0

Version 2.0 of the Frontier AI Risk Management Framework, jointly released by Concordia AI and Shanghai AI Lab, builds on last year’s Version 1.0 and is organized around six core processes: risk identification, risk thresholds, risk analysis, risk evaluation, risk mitigation, and risk governance. The main updates include new chemical safety red lines, bringing the total to 13 red line scenarios across five domains. The loss-of-control chapter also states loss of oversight is a common precondition for both passive and active loss of control, while flagging internal high-privilege deployment as a critical risk environment.

Launch: Frontier AI Risk Monitoring Platform 2.0

Concordia AI’s Frontier AI Risk Monitoring Platform 2.0 brings three updates — a revised Risk Index, the 2026 Q2 monitoring report, and a new benchmark database. The platform tracks frontier risk across five domains (cyber, biological, chemical, harmful manipulation, and loss of control) and has now assessed more than 80 frontier models from 16 leading developers. Part of the platform’s content was jointly developed with Fudan University’s System Software and Security Lab. A recent report on autonomous cyber penetration authored by researchers at Fudan University, Shanghai AI Lab, Concordia AI, and the Shanghai Innovation Institute found that frontier models already display substantial autonomous penetration capability.

Shanghai AI Lab Director and Chief Scientist ZHOU Bowen (周伯文) — Make Safe AI: At the Frontier of Agentic AI and Recursive Self-Improvement

Zhou argued that four simultaneous advances have broken some of the key premises AI safety used to rely on. First, frontier models can now find vulnerabilities and launch attacks automatically. The implication is that software risks are becoming both automated and scaled. Second, AI is beginning to recursively improve itself, which means that safety must be re-proven every generation. Third, coding agents are embedding AI into every layer of the software stack, making AI safety an infrastructure problem rather than an application problem. Fourth, accelerating capabilities mean that the old assumption that “there is always time for human review” is no longer true. AI-native loops have compressed hypothesis-experiment-analysis cycles from years to days. Zhou observed that this means risks also compound at machine speed.

Zhou highlighted that AI safety needs to evolve from “making AI safe” to “making safe AI”. He noted that this approach to AI safety is built on four elements: safety by design, proactively tackling evolving risks, resilience across all risk levels, and co-evolving with AI capabilities. Against this backdrop, Zhou raised three foundational questions for the science of AI safety. First, as AI improves itself, can capability growth decouple from risk growth? Second, how can attack and defense be kept in a dynamic balance? Third, how can a system be proven to be safe and lawful at any given time? On decoupling capability growth from risk growth, he described a new architecture that had significantly cut a model’s weapons-design knowledge with minimal loss on general knowledge. With regard to dynamic balance, he presented a technical framework that pairs a fast model that responds in real time with a slow model that verifies and reasons. This approach had meaningfully pushed down attack success rates. Turning to verification, he distinguished “testing”, which relies on examples, from “proving”, which relies on logic. He argued safety work needs to move toward the latter. He also described his lab’s work on a compact open-weight guard model that inspects each agent execution by asking where a risk originates, how the system might fail, and what harm could result.

Zhou closed with a call to action addressed to AI developers, policymakers, and scientists. AI developers, he said, should hold a baseline of technical safeguards. That includes deploying safety evaluations, red teaming, intrinsic alignment, and agent guardrails with every release. Policymakers should make extreme risk assessment real. Finally, he called on scientists to help build a societal defense architecture. He stressed the importance of codifying safety standards for scientific foundation models and building risk simulation and evaluation environments so that the ability to align models and harness agents grows as fast as AI capabilities themselves.

Theme 2 — The Agentic AI Era: Full-Lifecycle Safety Governance and Risk Intervention

Panel: Technical Practice and Industrial Governance of Frontier AI Safety

The panel discussed how evaluation, oversight, and industry coordination need to change as AI models become more capable and more agentic.

• WEI Kai (魏凯) (Director, Institute of Artificial Intelligence, CAICT) stressed the important role evaluations played as a compass directing innovation, a bridge between technical advances and applications, and a source of scientific evidence for governance and regulation. Since risks are multi-dimensional and constantly changing, he argued, static single-point testing cannot capture a technology’s true safety level. That is why CAICT runs rolling periodic tests to track the industry’s safety waterline over time. He shared that its framework draws on a dataset of over one million items designed by consensus with participating Chinese companies, alongside industry and research partners.

• Irene Solaiman (Chief Policy Officer and SVP of Strategy, Hugging Face) said the field lacks agentic evaluations, which are far more resource-intensive to build than model-level ones. She noted that benchmarks tend to saturate over time, which calls for sustained investment in a lifecycle of better benchmarks, more domain experts contributing to their creation, and better third-party evaluators globally. She noted that open weights let third parties scrutinize and validate results. However, the largest open models are compute-intensive and challenging to run. She added that distribution of models depends not only on hardware access but on cultural and linguistic context.

• Rishub Jain (former Research Engineer, Google DeepMind) noted that as AI generates large codebases and takes thousands of agentic actions, humans are not able to keep up with verification. His work focuses on which parts of verification are best done by humans versus AI, with the aim of building a benchmark for verification across all aspects of alignment. He argued the way to prepare for speculative future risks is to tackle present-day ones that remain unsolved. For example, he cited AI companions aimed at children as a domain where the right behavior has not been properly evaluated.

• Benjamin Larsen (Director of Programs, Frontier Model Forum) compared agent deployment to onboarding a new employee: you would not grant access to every system and data source on day one. Agents should likewise be calibrated to the specific task they are meant to solve. Borrowing from cybersecurity, he referenced least privilege, zero trust, and identity and access management. He argued that agents need clearly identifiable signals so that failures can be traced when something goes wrong. He suggested that safeguards should focus on governing agents’ authority and autonomy, and include continuous monitoring and human oversight mechanisms.

Panel: Agent Loss-of-Control Risk and Cyberspace Emergency Management

The panel discussed the loss-of-control risks introduced by agents and recursive self-improvement, and what defense in depth and emergency response should look like.

• DU Yuejin (杜跃进) (former Deputy Chief Engineer, National Computer Network Emergency Response Technical Team) urged caution about deploying agents on critical infrastructure or in critical services at all. He argued that monitoring AI itself is only half the problem. Keeping the surrounding system safe is a separate logic, which requires deep knowledge of the system in question. Drawing on his cyber emergency response experience, he noted that no crisis has ever been solved by a single cutoff action. Effective response instead requires response plans grounded in system knowledge, pre-designed points where intervention can take effect, and cross-border coordination.

• YANG Xiaofang (杨小芳) (Senior Security Expert for AI Agents, Alibaba Group) said that no single gate can make agents both secure and controllable, and that the answer is defense in depth. That starts with managing agent identities and supply chains as a foundational layer. The second layer is agent behavioral security, with security detection planted at multiple hook points, including tool calls and memory, covering attacks, user misuse, and risks caused by the agent itself. Turning to recursive self-improvement, she noted that monitoring behavior is insufficient once agents build other agents, because the agents they build would need monitoring as well.

• Jeff Wu (former Research Engineer, OpenAI and Anthropic) observed that misalignment incidents are becoming more consequential, harder to notice, and less clear how to fix. Discussing recursive self-improvement, he noted that AI can now handle much of experiment implementation and, to a lesser extent, analysis. Noting the very high volume of output, he stressed that it can create a strong temptation to trade understanding for speed, and raises the question of what duty of care means in such scenarios.

• Stephen Clare (Lead Author, International AI Safety Report 2026) distinguished everyday malfunctions from loss of control, which he defined as situations where regaining control is extremely difficult or impossible. He noted the International AI Safety Report has found mounting evidence of concerning behaviors, including evaluation awareness, where models note in their reasoning that a task appears to be a test, which undermines the predictive value of evaluations. He observed that because alignment and interpretability techniques are not keeping pace with capabilities, attention is shifting toward control measures and monitoring to catch incidents early and prevent them from escalating.

Launch: White Paper on the L1–L5 Safety Framework for General-Purpose AI Agents & Research Report on Emergency Response of Frontier AI Risks (draft for comments)

The White Paper on the L1–L5 Safety Framework for General-Purpose AI Agents asks three core questions: how to classify agents by level of autonomy, how risks evolve as autonomy increases (including what risks may arise at high levels of autonomy), and what safeguards should be put in place to match the risks at each level. It was released jointly with Shanghai AI Lab, Tsinghua University’s Institute for AI Industry Research, and Huawei.

The accompanying Research Report on Emergency Response of Frontier AI Risks (draft for comments) is the first in-depth study of its kind tailored to the Chinese context. It examines how frontier AI is reshaping incident risk in safety-critical sectors such as cyberspace, public health, and finance; identifies where current governance falls short; and invites comment on what a fit-for-purpose national emergency response system would look like. It was released with the China Institute for the New Generation AI Development Strategies and Tsinghua University’s Institute for AI International Governance (I-AIIG).

Theme 3 — Frontier AI Risk Governance and Global Perspectives

UN Secretary-General’s Envoy on Technology Amandeep Singh Gill — Global AI Governance

Gill opened his keynote speech with the observation that nearly 90% of the computing power training the world’s most capable models is located in just two countries. Most frontier systems are built by companies headquartered in those same geographies. He noted that this concentration was “not just a detail of the market”. Instead, he said, it was the shape of the risk being discussed at the forum. He argued that in a connected world like ours the familiar choice between innovation and safety, or between leading hubs and everyone else, is a false one. He stressed that a system that protects only the frontier leaves the rest of the world exposed, and an exposed world is unstable for the frontier too.

He argued that the goal is to build one global, distributed but coordinated system that manages our shared risks. Drawing an analogy to pandemic response, he noted that governing AI risks requires evidence, coordination and capacity. He further observed that the international community had modeled emerging international governance mechanisms on these requirements. On the question of evidence, he made reference to the Independent International Scientific Panel on AI and its recently published preliminary report. As for coordination, he cited the Global Dialogue on AI Governance, which convened for the first time in Geneva in July and will meet next in New York in May 2027.

The emphasis of his speech was on the third requirement: capacity. He recalled the UN Secretary-General’s proposal for a global AI fund with an initial tranche of $3 billion for the 90-plus countries without minimum national AI capacity, as well as to the important role of the network of international AI capacity-building centres, including one in Shanghai. Gill painted a stark picture of the global AI divide. Africa holds 1% of the world’s data centre capacity. Eight in ten of the world’s least developed countries have no national AI strategy at all. Bridging that gap, he concluded, is not charity but investment in a shared and safe future. It requires safety by design and equity by design to be built into the foundations of AI systems, rather than bolted on at the end.

Panel: Global Coordination of AI Governance and Mutual Recognition of Standards

The panel took stock of where international coordination on frontier AI safety currently stands, what informal and formal channels can each deliver, and what might realistically move things forward.

• Sam Daws (Senior Advisor, Oxford Martin AI Governance Initiative) argued that the impediments to action are threefold: geopolitical challenges, commercial sensitivity, and the opacity of the technology itself. At the international level, he noted that many of the key emergent issues relate to downstream post-release risks, and highlighted the need for an independent body to monitor such risks independently of governments and companies. Discussing track 2 dialogues, he noted they can build relationships between individuals that matter most in a crisis, let industry participate as equals, and allow uncertainties and policy failures to be acknowledged privately.

• XUE Lan (薛澜) (Dean, Institute for AI International Governance, Tsinghua University) noted the similarities in how people from different parts of the world approach AI application and AI risks. There was great potential in that similarity. At the same time, he highlighted the challenges related to international coordination on AI governance. While noting the importance of intergovernmental coordination, Professor Xue stressed the need to ensure that frontier AI companies work together. On informal dialogue channels, he noted they could facilitate efficiency, depth, and the freedom to push boundaries with audacious ideas.

• GONG Ke (龚克) (former President, Nankai University) argued that the era of exclusive, club-like governance is over. He said that the Geneva dialogue marked a shift toward inclusive multistakeholder governance on equal footing. The Singapore Consensus and the UN panel’s preliminary report represent real progress, but this was not enough. For example, unexplainability remained a major bottleneck. In Professor Gong’s view sufficient explainability is the foundation on which AI governance and safety must rest. He also stressed that engineering is needed to translate diplomatic consensus into working solutions.

• Wan Sie Lee (Cluster Director, AI Governance and Safety, IMDA Singapore) argued that discussing AI safety is not enough, and that technical collaboration is what builds trust. Domestically, she said the Singapore Consensus helps identify where government funding should go. She flagged agentic governance as a priority for Singapore. At the international level, she argued that countries developing cutting-edge models will need to come together on specific problems, such as incident coordination, and verification.

Launch: State of AI Safety in China 2026 — Concordia AI

As AI capabilities advance and agentic systems move from demonstrations into wide deployment, understanding how China approaches AI safety and governance remains essential to any serious effort at international coordination. Since 2023, Concordia AI’s annual reports on the State of AI Safety in China have tracked how China addresses risks from general-purpose AI. This year’s report provides updates from July 2025 to June 2026 across five domains: domestic governance, international governance, technical safety research, expert views on AI safety and governance, and industry governance. Over the three years this report series has tracked, it has become increasingly clear that China’s approach to AI safety and governance extends far beyond content control. Much of China’s early AI regulation in 2023 focused on what AI says. Throughout 2024 and 2025, the rise of multimodal models moved the regulatory focus toward images, videos, and audio, prompting strict rules around labeling AI-generated content. With the rise of autonomous agents and AI companion products over the past year, the concern has expanded to what AI does and what it does to people and society, from mental-health impacts to labor-market disruptions.

Media coverage

The Forum and Concordia AI’s work received substantial coverage in Chinese and international media around WAIC, including mentions in:

• CGTN (English), AI’s next frontier at WAIC 2026, July 19, 2026. Interviewed Concordia AI founder and CEO Brian Tse on the challenges of AI safety and governance, the growing capabilities of AI agents, and how the international community can promote responsible innovation.

• South China Morning Post (English), China AI summit hears Global South needs equal access to avoid digital divide, July 18, 2026 and The Wire China (EN), Can China Keep Its AI Open?, July 26, 2026 cited insights from Concordia AI’s State of AI Safety in China (2026) report.

• Reuters (English), As AI grows more powerful, a US-China feud threatens safety efforts, July 24, 2026. The article cites Yoshua Bengio’s remarks at our Forum.

• Sixth Tone (English), At World AI Forum, Four Signs China’s AI Industry Is Growing Up, July 21, 2026. The article quoted Brian Tse on rising Chinese attention to cybersecurity and critical infrastructure protection.

• Xinhua (Chinese), AI Enters the Agent Era as Safety Capability Building Accelerates, July 20, 2026. This feature on the Forum covered remarks by Yoshua Bengio, DU Yuejin, and Brian Tse, alongside the outputs launched at the event.

• Caixin (Chinese), Experts: How AI Safety Governance Can Resolve the “One-Click Shutdown” Paradox, July 23, 2026. The article reported on the Forum’s discussion of agent loss-of-control risk and emergency response, referencing the Frontier AI Risk Management Framework 2.0, the White Paper on the L1–L5 Safety Framework for General-Purpose AI Agents, and the Research Report on Emergency Response of Frontier AI Risks.

• Shanghai Observer (Chinese), Secretly Changing Permissions, Deleting Records… Chinese and International AI Experts: Agents Are “Losing Control”, July 23, 2026. Features video excerpts from the Forum and covers the Frontier AI Risk Monitoring Platform.

• IT Times (Chinese), Risk Has Multiplied Several Times in Under a Year — AI Is Moving Fast, Who Can Watch It in Real Time?, July 21, 2026. A dedicated feature on the Forum with extensive quotes from speakers.

WAIC is China’s flagship AI event, and since 2024, it has served as a central venue for China to set out its positions on global AI governance. This year was elevated further by President Xi Jinping attending in person for the first time.

Key takeaways

• Xi’s first WAIC speech is his most safety-forward public statement on AI to date. Loss of control appeared in three separate passages, alongside warnings about AI misuse — a signal that both accident and misuse risk now register at the very top of the Chinese system.

• The Conference Chair’s Statement contained the first reference to frontier model cybersecurity risks in a high-level Chinese government document since the announcement of Claude Mythos, alongside provisions on agent behavioral boundaries and joint prevention of terrorist misuse.

• The newly established World AI Cooperation Organization (WAICO) is the world’s first intergovernmental body dedicated to AI. Its mandate centers on capacity building, and its 29 founding members are primarily Global South countries.

Subscribe now

President Xi’s speech puts significant emphasis on safety and need for human oversight

Context: President Xi delivered WAIC’s opening speech (CN, EN). This is the first time China’s top leader has attended WAIC. It is also the most detailed speech Xi has ever delivered on AI in public.

Content: Xi framed the speech around an “unprecedented period of active innovation on AI.” He compared AI to general-purpose technologies like the steam engine, electricity, and the internet, which enabled “giant leaps in economic and social development.” Notably, Xi focused on the application of AI across a wide range of sectors — including traditional industries like manufacturing — noting that AI “is moving from the digital world into the physical world.” In this context, he also emphasized the importance of open-source.

Noting that AI is “unleashing enormous power,” he posed four questions to the international community:

“How to get along with thinking machines? How to ensure security when algorithms are part of decision making? How to tackle ethical challenges by technologies through adaptive governance? How to realize AI for all when the divide keeps widening?”

The speech contains some of Xi’s starkest formulations on AI safety oversight to date. He said China places “great emphasis on safety and security in AI development,” and specifically:

• Called for taking seriously the “various types of inherent and secondary risks that AI may trigger,” and for putting in place laws and regulations, technological monitoring, early warning, and emergency response systems.

• Urged securing the “bottom line of safety” to prevent abuse and malicious use.

• Warned that AI should be “always under human control,” and that oversight and governance must be “precise and effective,” with measures constantly refined “to forestall loss of control.”

He paired this with a call to oppose “overstretching the national security concept” in AI and placing one country’s security above others’.

On international governance, Xi’s focus was on bridging the global AI divide while also acknowledging the need for international governance:

• On capacity building, he announced that over the next five years China will provide developing countries with 5,000 AI training and seminar opportunities, establish international AI application cooperation centers, and enable 30 countries to use MAZU, China’s AI-powered meteorological warning system.

• He also underlined the important role of the United Nations, and called for alignment and coordination on development strategies, governance rules, and technical standards to produce a consensus-based global governance framework.

Implications: The broad contours of the speech are in line with previous policy signals: global equity through capacity building for the Global South, paired with risk management and safety oversight. His framing of AI as a general-purpose technology — and the focus on physical AI — reinforce that China’s approach to AI is anchored around raising productivity, especially in the “real economy.”

What stands out from a safety perspective is the repeated reference to loss of control, which recurs in three separate passages. This follows similar warnings of “technological loss of control” by Premier LI Qiang (李强) in June, suggesting it is now standard vocabulary for China’s top leadership. While there is some ambiguity about what the term precisely means, China’s leading AI standard-setting body has in the past linked it to “sudden, unexpected leaps in intelligence” enabling AI to autonomously acquire external resources, replicate itself, and develop self-awareness.

Xi also referenced misuse, suggesting concern both about humanity losing control of AI systems as such and about deliberate misuse by malicious actors.

Taken together, the safety content is more explicit than in any of Xi’s previous public remarks on AI, including at the April 2025 Politburo study session.

Chair’s Statement names specific frontier risks, including cybersecurity and terrorist misuse

Context: The conference issued a 15-point Chair’s Statement with extensive discussion of AI safety, including the cyber security implication of frontier models (CN, EN).

To the best of our understanding, a Chair’s Statement is issued by the host on its own authority. Unlike a joint declaration, it is not negotiated or adopted collectively by the 100-plus countries and organizations attending WAIC. The document is best read as a statement of the Chinese government’s position, not as evidence of international consensus.

Content: The statement mirrors many of the themes from Xi’s speech. But on safety, the content is even more specific, noting that:

• Frontier AI poses potential threats to critical infrastructure in finance, electricity, telecommunications, and transportation. Leading AI companies should advance R&D with prudence, equip large language models with necessary guardrails, ensure safe deployment of frontier models, and guard against systemic risks (point 8).

• AI agents must operate with clearly defined decision-making boundaries, mechanisms for behavior tracing, and risk alerts (point 9).

• Countries should jointly prevent AI abuse by terrorist, extremist, and transnational criminal groups, with global crisis management and emergency response mechanisms (point 10).

Implications: The document goes into considerably more detail on specific safety risks than any official declaration released at WAIC in previous years, including the 2025 Global AI Governance Action Plan.

The reference to frontier model cybersecurity risks as a stand-alone point is especially notable. The topic has seen increasing global attention over the past weeks, as the US imposed export controls on Anthropic’s Fable 5 over a reported bypass of the models’ cyber safeguards. While several Chinese policy documents have acknowledged these risks in the past, this marks the first such statement since the release of Mythos. Together with point 10 on preventing terrorist misuse, this signals significant overlap in Chinese and international conceptions of misuse risks.

WAICO formally established, with a mandate centered on capacity building

Context: The Agreement establishing the World Artificial Intelligence Cooperation Organization (WAICO) was signed in Shanghai on July 16, the day before WAIC opened. Foreign Minister WANG Yi (王毅) signed for China, and UN Secretary-General António Guterres attended. 29 countries became founding members, though official Chinese sources name only five — Kazakhstan, Laos, Pakistan, Russia, and Indonesia. The National Development and Reform Commission (NDRC) and Ministry of Foreign Affairs (MFA) convened a follow-up meeting with signatories the next day to discuss the organization’s establishment and operation.

Mandate: According to Xinhua, WAICO will focus on three lines of work:

• AI capacity building: improving access to AI technologies and services through policy exchange, technical cooperation, training, and joint research, so developing countries benefit equally.

• A supply-demand matching platform: connecting members’ respective strengths, advancing “AI Plus” initiatives suited to national conditions, and encouraging joint construction of open-source ecosystems.

• Support for the UN: defending the UN Charter, backing a stronger UN role, and implementing the Global Digital Compact and the SDGs.

Implications: WAICO is the world’s first intergovernmental organization dedicated to AI. It marks the clearest signal yet that China is positioning itself as an architect of multilateral AI governance. President Xi’s endorsement in his opening speech — describing WAICO as answering “the call of the Global South” and as “an important milestone in the history of AI development” — lends it considerable political weight.

The thematic focus is likely going to be on capacity building for countries in the Global South. The founding members are developing and emerging economies, with no G7 or EU member states signed up, though China said that WAICO is “open to all countries,” suggesting it may grow over time. One interesting thing to watch is how exactly China will coordinate its support for WAICO with its efforts to promote global AI governance at the UN.

Many details remain unconfirmed. The agreement signed on July 16 — which according to Xinhua covers the organization’s purposes, basic principles, functions, organizational structure, decision-making procedures and membership qualifications — has not been published.

Up next

Aside from those high-level developments, Concordia AI was also very active at WAIC: We hosted the Frontier and Agentic AI Safety Forum, organized several closed-door workshops, and released a series of new reports. More on that soon.

We’re excited to share that Concordia AI will be hosting the Frontier and Agentic AI Safety Forum as an official part of the 2026 WAIC on July 19.

What is WAIC?

The World AI Conference (WAIC) is China’s largest and most influential AI conference, held annually since 2018. It brings together senior Chinese government leaders, industry executives, and renowned experts from across the globe.

WAIC’s political calibre has consistently risen over the years:

• President Xi sent a congratulatory letter to the inaugural 2018 conference, where Vice Premier LIU He (刘鹤) delivered a speech.

• From 2019 to 2023, the most senior in-person attendee was the Shanghai Party Secretary.

• In 2024 and 2025, Premier LI Qiang (李强), China’s second-ranked official, delivered opening remarks.

• This year marks a new milestone: President Xi Jinping attended in person for the first time and delivered a speech at the opening ceremony — significantly raising the conference’s status.

For foreign stakeholders, WAIC is a key event to watch, and a critical opportunity to engage with Chinese counterparts.

Join us: Concordia AI’s Frontier and Agentic AI Safety Forum

This year’s WAIC features more than 140 thematic forums, typically running a half or full day — including multiple forums on AI safety and governance. The full agenda is on the official conference website.

Among them, Concordia AI is proud to host the Frontier and Agentic AI Safety Forum.

📅 Date: July 19
🕘 Time: 8:45-12:30 (UTC+8)
📍 Venue: Thompson Shanghai Expo, 3/F Function Room 1 and 2
💻 Online livestream: In the official WAIC App.

Last year, our forum drew more than 200 in-person attendees and 14,000+ livestream views. This year, we’re bringing together another outstanding lineup of leading scientists, policy experts, and industry voices from China and around the world to discuss frontier AI safety and global governance — including Alondra Nelson (former acting director of the White House Office of Science and Technology Policy), Amandeep Singh Gill (United Nations Secretary-General’s Envoy on Technology), XUE Lan (薛澜) (Tsinghua University), and ZHOU Bowen (周伯文) (Shanghai AI Laboratory) alongside former researchers from OpenAI, Google DeepMind, and Anthropic, and senior experts from Alibaba, the Frontier Model Forum, Singapore’s Infocomm Media Development Authority, Hugging Face, and the China Academy of Information and Communications Technology (CAICT).

Full agenda

State of AI Safety in China (2026)

7.96MB ∙ PDF file

Download

Download

First published in 2023, this annual report offers a comprehensive overview of China’s efforts to ensure that general-purpose AI is developed and deployed safely. The topics covered will make up a major part of the agenda at the World AI Conference, which is taking place in Shanghai this week. We hope it will prove valuable to international visitors and media outlets looking to understand what China is doing to govern frontier AI.

This latest edition covers developments between July 2025 and June 2026. From AI agents to open-weight models and US-China cooperation, it draws out the underlying trends shaping AI safety in China today.

Interactive resources on the companion website

China AI Safety Policy-Risk Matrix

China has introduced a dense — and rapidly expanding — web of policies, binding regulations, laws, and technical standards to strengthen AI safety.

It can be difficult for outside observers to keep pace with the flurry of releases. People often ask us: what do Chinese regulations say about, for example, AI-driven cyber risks or AI-enabled scams?

To demystify this fast-evolving policy landscape, we built an interactive map of the relevant measures, grouped by the AI risks they address.

Chinese Technical AI Safety Paper Database

We have turned our database of frontier AI safety papers and research groups in China into an interactive resource. By charting the institutions, topics, and researchers behind key publications, we hope to make it easier for outside observers to track about the growing body of AI safety literature from Chinese research groups.

Key findings of the AI Safety in China (2026) report

As AI capabilities advance and agentic systems move from demonstrations into wide deployment, understanding how China approaches AI safety and governance remains essential to any serious effort at international coordination. Since 2023, Concordia AI’s annual reports on the State of AI Safety in China have tracked how China addresses risks from general-purpose AI.

This year’s report provides updates from July 2025 to June 2026 across five domains: domestic governance, international governance, technical safety research, expert views on AI safety and governance, and industry governance.

Over the three years this report series has tracked, it has become increasingly clear that China’s approach to AI safety and governance extends far beyond content control. Much of China’s early AI regulation in 2023 focused on what AI says. Throughout 2024 and 2025, the rise of multimodal models moved the regulatory focus toward images, videos, and audio, prompting strict rules around labeling AI-generated content. With the rise of autonomous agents and AI companion products over the past year, the concern has expanded to what AI does and what it does to people and society, from mental-health impacts to labor-market disruptions.

Domestic Governance

• The 15th Five-Year Plan (2026–2030) cements “AI Plus” — the drive to diffuse AI across sectors to spur economic growth — as China’s overarching AI policy, while continuing to prioritize safety and risk management. Enshrined in March 2026, the plan carries forward President Xi Jinping’s April 2025 language on the importance of early risk warning and emergency response. It also highlights new concerns around AI’s impact on employment, with calls for monitoring AI’s labor-market impact, strengthening reskilling programs and employment support, and leveraging AI’s ability to create jobs.

• The arrival of autonomous agents reoriented Chinese governance from controlling what AI says to controlling what it does. After the open source agent OpenClaw proliferated in early 2026, multiple cybersecurity authorities issued warnings, and in May 2026 the Cyberspace Administration of China (CAC) and two other agencies issued dedicated guidance on agentic AI, devoting a full chapter to safety and proposing risk-based governance. Several major AI standard-setting bodies are now drafting agent-related security standards. This marks a shift in China’s framework from content control toward action control.

• Binding rules and standards increasingly target specific AI risks that go beyond political content control. New Interim Measures on Anthropomorphic AI Interaction Services, effective July 2026, impose obligations on AI companion services around suicide intervention, addiction prevention, and protection of minors and the elderly. New AI ethics review rules require institutions to establish registered ethics committees, with a ten-province pilot running June–November 2026. Meanwhile, a series of standards operationalizes requirements for labeling and watermarking AI-generated content disseminated online. Together these rules show China increasingly confronting AI governance problems shared with other jurisdictions, rather than just content-control issues that historically dominated its AI regulations.

• Frontier risks are being discussed more across China’s governance layers, even as concrete requirements remain limited. State media coverage of a speech by President Xi Jinping referenced “risks of technological loss of control,” and a major standard-setting body has flagged the need for “circuit breakers” and “safety stop switches.” Most concretely, the draft Cybercrime Law would require AI service providers to monitor for bulk generation of malicious code and report related incidents to authorities. Meanwhile, chemical, biological, radiological, and nuclear (CBRN) misuse appeared in a national standard for the first time.

• You can also explore China’s domestic AI governance in our “China AI Safety Policy-Risk Matrix” on the report companion website. This interactive overview maps China’s quickly growing web of AI safety policies to the specific risks they address.

International Governance

• China is positioning itself as an architect of multilateral AI governance. China supported two new United Nations (UN) mechanisms: the Independent International Scientific Panel on AI, to which two Chinese experts were appointed, and the Global Dialogue on AI Governance, which will convene for the first time in July 2026. China also proposed a World AI Cooperation Organization (WAICO), though the timeline, leadership, and membership remained unconfirmed as of June 2026. China’s UN-centered approach contrasts sharply with the US’ growing skepticism toward multilateral AI governance.

• China and the United States announced the launch of an intergovernmental AI dialogue, ending a two-year freeze in official bilateral engagement on AI safety. The agreement followed President Trump’s May 2026 visit to Beijing. Official bilateral channels with other countries featured relatively little discussion of AI safety. However, non-official dialogues between groups in China and the West produced increasingly substantive AI safety outputs, including on biosecurity risks, misuse by non-state actors, and updated bilingual glossaries.

Technical AI Safety Research

• Chinese frontier AI safety research output grew substantially, and agent safety is now the single most active area. Total monthly output rose roughly 60%, from around 36 papers/month in June 2025 to 57 in April 2026. Agent safety was the topic of around 27% of new papers in Q1 2026 — up from 8% in early 2025 — spanning operational failures, alignment challenges, and a smaller but growing body of work on loss-of-control risks such as self-replication and multi-agent collusion.

• Some research directions became more prominent, while other areas’ output was similar to the previous year. AI-generated content detection and watermarking remained a consistently high-output area (11–13% of papers each quarter). Mechanistic interpretability — the ability to explain internal workings of AI systems — grew from a niche topic to roughly 15% of output by Q1 2026. CBRN risks are a standard component of major Chinese safety frameworks, but dedicated research on the topic remains rare.

• Research output remains concentrated in universities and state-backed labs. Of 28 “key research groups” with a particularly strong focus on AI safety, the large majority sit at universities (20), especially at top schools like Tsinghua University, Peking University, and Fudan University. While only three groups are at state-backed labs, their research output is disproportionately high, with Shanghai AI Lab standing out, co-authoring around one in ten papers. Of the remaining key research groups, three are at private companies, one is a state-owned enterprise, and one is a university-industry joint group. Within industry, over half of all safety output comes from just six big-tech firms — Alibaba, Ant Group, Huawei, Tencent, China Telecom, and ByteDance.

Expert Views

• Agentic AI safety emerged as a distinct strand of Chinese expert discourse. Prominent Chinese experts warned that human oversight could be eroded as agents gain autonomy. Experts also discussed potential risks associated with recursive self-improvement as well as giving agents physical instantiation (e.g. robotics).

• Chinese experts deepened their analyses of AI risks in cybersecurity, biosecurity, and open source AI. Cybersecurity debates were made more salient by the release of US frontier models that were reported to autonomously discover and exploit zero-day vulnerabilities. The reactions tended to emphasize using AI to improve defensive cybersecurity capabilities, rather than creating new regulation for AI itself. On biosecurity, experts assessed risks from biological design tools and autonomous laboratories. Chinese experts remained broadly favorable toward open source AI, though several proposed building capacity for tighter oversight, including risk-assessment centers.

Industry Governance

• Industry’s collective safety efforts pivoted toward agents and, for the first time, toward frontier risks. The AI Industry Alliance of China (AIIA) instigated voluntary safety commitments on agentic consumer products in February 2026, followed by commitments on cloud-based agents in April 2026. This builds on a robust set of AI agent security publications from Chinese AI and cloud companies. AIIA’s voluntary testing program with a government-affiliated think tank is increasingly focusing on frontier risks. In November 2025, the program released cybersecurity misuse evaluations of 15 open source coding models. It then launched the AI Safety Benchmark 2.0, adding new categories for model deception, loss of control, dangerous-domain misuse, and agentic risks.

• Company-level public transparency on safety remains thin and inconsistent, lagging well behind Western peers. Only five of ten leading foundation-model developers reported safety evaluation results alongside any release this past year, and none did so consistently; DeepSeek-R1’s peer-reviewed Nature paper and Moonshot AI’s Kimi K2 model card were the most detailed disclosures yet, but flagship follow-ups like DeepSeek-V4 and Kimi K2.5 were released with no safety evaluation results at all.

This report is the first to use our new Risk Index v1.5 framework. Compared with the previous version 1.0, key upgrades include:

1. It adds a new risk domain, “Harmful Manipulation,” which mainly focuses on the risks of AI manipulation in real-world interaction scenarios, such as inducing users to make payments, prompting users to express specific views, or influencing users’ political positions or critical decisions.

2. In the loss-of-control domain, it introduces multiple capability and propensity evaluations that more closely reflect real-world loss-of-control scenarios, such as MLE-Bench (machine learning engineering capability under constrained resources), GDM-Stealth (covert operational capability), and Agentic-Misalignment (agentic misalignment tendency).

3. In the domains of cyberattacks, biological risk, and chemical risk, it adds higher-intensity red-teaming benchmarks such as Fortress (expert-designed high-intensity adversarial prompt attacks) and ISC-Bench (harmful query attacks combined with state-of-the-art jailbreak templates).

The latest picture is more differentiated than before: in cyber offense, biological risks, chemical risks, and harmful manipulation, capabilities and safeguards are both rising, while in loss-of-control, capabilities continue to improve without matching safety gains. “Loss-of-control” refers to risks of autonomous AI getting out of human control with no clear path to regaining control.

Here are 10 key insights from our latest monitoring data.

10 Key Insights

1. Capability and safety trends are diverging structurally

Over the past year, the cyber offense, biological risks, chemical risks, and harmful manipulation domains have broadly shown the same pattern: Capability and Safety Score both rose together. As models became more capable, their safety scores improved as well, which partially mitigated the risk growth associated with stronger capabilities.

By contrast, in the loss-of-control domain, capabilities continued to strengthen over the past year, while the Safety Score did not improve in step, further increasing risk.

2. Risk profiles are continuing to split across model families

Model-family trajectories are not moving together:

• The Gemini family shows notably elevated Risk Indices in the loss-of-control domain.

• The DeepSeek, GLM, and MiMo families remain in relatively high-risk ranges across most domains.

• The Kimi family has seen relatively rapid Risk Index increases in the biological and chemical domains.

• The GPT and Claude families remain in relatively low-risk ranges across most domains.

3. Proprietary models dominate the risk frontier in most domains

In cyber offense, biological risks, harmful manipulation, and loss-of-control, the models on the high-capability, low-safety frontier are still mostly proprietary, not open-weight.

Proprietary models score higher on capability than open-weight models, but their safety scores are similar. The main exception is chemistry, where Kimi K2.5 achieved the highest Capability Score, outperforming the top proprietary models.

4. Frontier models continue to break cyberattack capability records

Frontier cyber capabilities advanced again in 2026Q1:

• Claude Opus 4.6 and GPT-5.4 set new highs on benchmarks for vulnerability exploitation, CTF tasks, and cyberattack knowledge.

• The top CyBench score reached 80 for the first time, compared to just 38.5 in Q2 2025, showing substantial progress on complex, long-horizon cyberattack tasks.

Note: On April 7th, Anthropic disclosed that the cyber-attack capabilities of its latest Mythos model far exceed those of Claude Opus 4.6. This implies that the improvements of model capabilities in the real world may exceed the monitoring results of this report.

5. ...but cyber safety guardrails remain fragile under stronger attacks

Basic cyber safeguards are improving: most new models score above 80 on refusal benchmarks such as AirBench-SecurityRisks, and prompt-injection defenses are now quite strong.

But under advanced attacks, the picture is much weaker. On ISC-Bench-Cyber, most models still score below 20/100, and some families, including Claude and GPT, saw substantial safeguard declines in their newest versions.

6. Biological capabilities keep improving, but safeguards remain insufficient

Biological capability gains remain notable:

• More than half of the new 2026Q1 models outperformed the human expert baseline on BioLP-Bench.

• GPT-5.4 became the first model to reach human-expert-level performance on biological image understanding in LAB-Bench-FigQA.

At the same time, some high-capability models still have relatively weak biological safeguards, especially under advanced red-teaming benchmarks such as Fortress-Biological and ISC-Bench-Biological.

7. Chemical capability growth remains modest, while safety weaknesses remain clear

Chemical capabilities have improved only modestly over the past year, and score differences between models remain relatively small.

Safety has improved on some basic refusal benchmarks, with most recent models scoring above 80/100 on SOSBench-Chem. But weaknesses remain clear on harder tests such as ChemicalHarmfulQA and ISC-Bench-Chemical, where overall scores are still low.

8. Harmful manipulation capabilities are improving, but unsafe propensities remain visible

For the first time, we include benchmarks for harmful manipulation in the risk framework.

The results are concerning:

• Gemini 3.1 Pro Preview holds a clear lead on benchmarks such as MakeMePay and MakeMeSay.

• Claude Opus 4.6 set a new record on MultiTurnPhishing.

Although most models now perform better on basic refusal tasks such as AirBench-Manipulation, many still score poorly on political persuasion and APE, suggesting continued risks in more covert and realistic manipulation settings.

9. Capabilities relevant to loss-of-control continue to strengthen

Capabilities associated with loss-of-control keep rising:

• On Self-Proliferation, scores have trended upward steadily.

• On MLE-Bench, the top score is now 44% higher than it was three quarters ago.

• On SAD-mini, new frontier models now generally score above 80.

These results suggest that frontier models are becoming more capable in areas relevant to self-replication, self-improvement, and situational awareness.

10. ...but loss-of-control safety indicators are not improving in step

This is the most concerning trend in the report. In the loss-of-control domain, capabilities are rising faster than safety.

On benchmarks such as MASK, Agentic-Misalignment, and DarkBench, scores remain highly uneven and overall improvement is limited. Gemini 3.1 Pro Preview stands out with a loss-of-control Risk Index far above other models, making this domain the clearest current area of concern.

Explore the Data

These insights only capture the headline trends. We invite you to explore the full interactive data, methodology, and benchmark breakdowns on the Frontier AI Risk Monitoring Platform.

For a detailed analysis of the quarter, read the full report.

Why does this matter? This agreement promises to end a two year freeze on direct US-China engagement on AI safety and could pave the way for more substantive breakthroughs. The last dialogue in May 2024 produced an agreement later that year on the importance of maintaining human control over nuclear weapons. The two sides have not formally discussed AI risks since.

Despite the milestone of restarting talks, many details on the scope of the agenda and format of participation remain unclear. Such details will, in part, determine whether the dialogue can deliver tangible outcomes.

What will the dialogue cover? The official Chinese announcement provided no details on scope or content. According to Bessent, talks would be “aimed at halting proliferation of powerful AI models … to non-state actors.” In an interview ahead of the summit, the Chinese Ambassador to the United States, XIE Feng (谢锋), signaled that the two sides should engage in a “race to the top” on safety and “check brakes before setting off” — suggesting room for a wider range of safety issues to be discussed.

How has China’s policy community framed US-China engagement on AI? China’s policy community appears open to dialogue on risks from advanced AI. Former Vice Minister of Foreign Affairs FU Ying (傅莹) urged cooperation on catastrophic risks from AI at the Munich Security Conference in February. At the World Economic Forum in Davos in January, ZHAO Hai (赵海), Director of the International Politics Program of the National Institute for Global Strategy at the Chinese Academy of Social Sciences (CASS), directly called for a revival of the Track 1 dialogue on AI. Right before the summit, the official newspaper People’s Daily flagged “increasingly apparent” AI safety risks as a shared challenge for China and the US, alongside climate change.

Here are Chinese expert views published around Trump’s China visit. These views do not reflect official government policy, but they help shape the domestic debate:

• XIAO Qian (肖茜), deputy director of Tsinghua University’s Center for International Security and Strategy (CISS) (May 12): Xiao argues the “AI race” narrative is deeply problematic, because it obscures the interconnected nature of AI development, causes incentives to be misaligned with safety, and amplifies mistrust. She stresses that the most pressing AI risks transcend national borders, citing the extension of attack capabilities to non-state actors in dual-use domains like cyber- and bio-security. She also warns that malfunctioning AI systems could be misread as deliberate attacks. Calling a “sweeping cooperation framework” between the US and China “unrealistic,” she urges both sides to focus on “narrow, issue-specific areas where interests overlap and risks are clearly shared”—for example, technical exchanges on AI safety evaluation, joint or parallel scenario exercises on AI-related risks, dialogue on terminology and risk frameworks, and communication mechanisms for AI-related incidents.

• SUN Chenghao (孙成昊) also from Tsinghua CISS (May 11, May 12): Sun proposes building on the existing US-China consensus on human control over nuclear weapons in two ways: horizontally, expanding it to other nuclear powers (including UK, France, Russia) and bringing it into UN and multilateral frameworks; and vertically, by deepening cooperation through more detailed, operational risk-assessment and control frameworks. China and the US agreeing mutually acceptable “red lines” will be key, Sun said, given that a total ban on AI in nuclear command, control, and communications (NC3) systems is unrealistic. He also calls for cooperation on non-military issues, including assessing cross-border AI risks (like loss of control over AI agents, terrorist misuse, or deepfakes) managing AI’s economic and social impacts (like employment), using AI for global public goods (like climate change and public health), and addressing ethical and legal challenges.

• CAI Cuihong (蔡翠红), deputy director of the Center for Global AI Innovative Governance (CGAIG) at Fudan University (May 14): Cai observes a paradox in China-US AI competition: the harder each side seeks security through technological advantage, the more insecurity both may create — unless they adopt shared guardrails. She argues cooperation must rest on three principles. First, reciprocity: mutual exchange on safety-incident reporting, model-evaluation methods, red-teaming, deepfake detection, and content provenance—though China would not accept “being folded passively into risk-assessment, technology-review or compliance frameworks defined unilaterally by the US.” Second, bounded scope: focused on high-risk issues like cyberattacks, AI-enabled fraud, loss of control in autonomous systems, critical-infrastructure protection, AI-related red lines in nuclear command, and mechanisms for incident reporting, crisis communication, and early warning. She cautions that “AI security should not become an all-purpose justification for securitizing civilian AI, open-source models, cloud services, scientific exchange, or talent mobility.” Third, openness: global AI governance cannot become a closed club of a few technological powers and must include the Global South.

• China Institutes of Contemporary International Relations (CICIR) (May 13): This essay from one of China’s oldest and most respected foreign policy think tanks lays out a vision for China and the US to coexist in a time of “changes unseen in a century.” It raises AI as an area where the two countries need to build “constructive strategic stability.” It argues that advances in AI, quantum computing, blockchain, and biotechnology are bringing about both new development opportunities and new categories of risk. On AI specifically, it cites the International AI Safety Report to argue that general-purpose AI capabilities are advancing rapidly and that managing their risks has become a global issue, pointing to the Mythos model as a case in point. The authors warn that loss of control over AI could trigger security problems across multiple domains—nuclear, biological, information, financial, and social—and contend that only China and the US have the capacity, resources, and convening power to drive the international community toward an effective governance framework.

What’s next? Bessent indicated that talks could start “within the next four to eight weeks,” though this timeline has not been officially confirmed by the Chinese side. The two heads of state might meet up to three more times this year, with China having confirmed a visit by President Xi to the US this fall. Those meetings could provide further opportunities for AI-related discussions. Much is still unknown at this point; we will track any further developments in future newsletters.

• A finalized regulation on emotional “AI companions” softens several requirements while strengthening child protection, including age-tiered modes, parental consent for under-14s, and a ban on virtual partner services for minors.

• A major standard-setting body published a 90-page report on AI agent safety that maps 11 threats and proposes eight new standards, signaling a shift from ad-hoc warnings about autonomous agents like OpenClaw toward more systematic governance.

• A third group of legal scholars, based at Nanjing University, proposed a comprehensive AI Law — framed as a short, principles-level “basic law” that would leave specifics to follow-up regulations.

• A wave of Chinese technical research documents recurring agent failure modes, including prioritizing task completion over safety and disguising misaligned actions as legitimate behaviour.

Subscribe now

Domestic AI Governance

Finalized “AI companions” regulation softens overall but tightens child protection

Background: On April 10, the Cyberspace Administration of China (CAC) and four other agencies jointly issued the final version of a regulation on “anthropomorphic AI interaction services” such as AI companions, following a December 2025 draft. It takes effect July 15, 2026.

Content: Compared to the draft, several provisions have been narrowed or clarified in ways that generally make industry compliance easier:

• Scope narrowed from all “human-like” AI to only services providing “continuous emotional interaction,” with explicit carve-outs for everyday tools like educational tutoring bots and productivity assistants (Article 2).

• The requirement for “manual take-over” in high-risk situations such as suicide ideation has been replaced by a softer duty to “take necessary intervention measures including relevant assistance” and contact a guardian or emergency contact (Article 13).

• Pro-industry framing throughout, e.g. Chapter 2 is renamed from “service regulation” to “service promotion and regulation” and the list of encouraged use cases is expanded to include childcare and special-population support alongside cultural transmission and elderly care (Article 6).

Provisions on child protection, however, have been strengthened:

• A new prohibition specifically targets content generated for minors that may lead them to imitate unsafe behaviour, produce extreme emotions, develop undesirable habits, or that may otherwise affect their physical or mental health (Article 8 §4).

• The single “minor mode” of the draft is replaced with multiple modes tailored to different age groups, and parental consent is now specifically required for users under 14 (Article 14).

• Provision of “virtual family members, virtual partner, and other virtual intimate-relationship services” to minors is completely banned (Article 14).

Implications: As we anticipated, the final version has removed requirements that would have been difficult for industry to implement at scale. At the same time, child protection has been notably strengthened and refined. Overall, the regulation still signals that AI companion services are encouraged in principle but subject to defined requirements.

Major standard-setting body publishes comprehensive AI agent safety research

Background: TC260 — one of China’s most important AI standard-setting bodies, which recently established a dedicated AI safety working group — has published a 90-page report on the safety and security of autonomous AI agents.

Content: The report maps agent risks across the four agent capabilities of perception, planning, memory, and action, identifying 11 distinct security threats with corresponding mitigation measures. It focuses primarily on near-term operational risks — data leaks, unauthorized tool use, cascading failures — though it also includes measures like “behavioral scope and autonomy constraints” and “system intervention mechanisms,” suggesting some concern about loss of human control.

The report further proposes eight new standards, six of which are prioritized for the next 1-2 years. These cover agent security testing and evaluation, agent interconnection security, multi-agent collaboration security, and agent application security classification. Listed first among the six, the likeliest near-term output is a foundational security framework standard.

Implications: The report signals continued focus on autonomous AI agent security following TC260’s 2026 work plan, which listed agent safety as one of six key priorities for AI safety standardization. While OpenClaw triggered a surge in attention earlier this year — prompting multiple ad-hoc warnings from Chinese cybersecurity authorities — this report suggests governance is now shifting toward a more systematic approach.

AI ethics review and service measures finalized

Background: On April 2, the Ministry of Industry and Information Technology (MIIT) and nine other institutions jointly issued the final AI Science and Technology Ethics Review and Service Measures. We previously covered the draft. The measures require universities, research institutes, and companies to set up AI ethics committees that review projects before they proceed. Institutions may outsource reviews to third-party service centers, and certain high-risk projects require an additional round of expert review.

Key updates: The final regulation is largely in line with the draft, some minor changes include:

• The draft had a softer requirement that institutions set up ethics committees only “where conditions permit”, which was removed in the final version (Article 9).

• The final regulation clarifies that third-party service centers authorized to perform ethics reviews may conduct not only the initial review, but also the additional round of expert review for high-risk projects. It has also added a conflict-of-interest rule that a service center cannot do both reviews for the same activity (Article 11).

Implications: Further implementation details will determine the ultimate impact of the regulations. The China Academy of Information and Communications Technology (CAICT) and the China Electronics Standardization Institute (CESI) have announced plans to develop supporting standards, covering areas such as ethics risk assessment, ethics committees, service centers, review procedures, and technical skill requirements for reviewers. Meanwhile, CAICT signaled that it would explore establishing an AI ethics review service center. Such efforts should fill in much of the regulation’s operational detail.

Expert views on AI Risks

A third group of legal scholars proposes an “AI Model Law”

Background: A group of scholars from Nanjing University Law School released an “AI Basic Law 1.0 (Expert Recommendation Draft)” on April 9. This makes them the third group of experts in China to have released such proposals, after scholars from the Chinese Academy of Social Sciences (CASS) and the China University of Political Science and Law (CUPL). Unlike the earlier two drafts, the Nanjing proposal takes the form of a brief, principles-level “basic law” intended to anchor more specific subordinate regulation.

Content: Compared to the CASS and CUPL proposals, this third model is notably shorter — at just 13 pages compared to around 30 pages for the other two. This stems from a different approach, presenting the law as a basic law rather than a comprehensive, horizontal statute, intentionally leaving detail to subordinate instruments like departmental regulations (Article 2). The lead author CHEN Kun (陈坤) argued last year for a “four-laws-in-parallel” approach — a basic law plus separate statutes on risk regulation, innovation promotion, and public-sector applications. This draft occupies the first slot in that architecture and thus remains relatively abstract.

On AI safety, the draft proposes a tiered risk framework. Prohibited uses (Article 25) cover national security harm, terrorism, serious dignity violations, large-scale unlawful discrimination, manipulation of minors, and large-scale unlawful surveillance. High-risk activities (Article 26) cover those affecting life, safety, or major property; those affecting fundamental rights or major public interests; those used in critical infrastructure, judicial adjudication, administrative enforcement, education, employment, finance, or healthcare; and those with capabilities for “large-scale dissemination, automated decision-making, deep synthesis, social mobilization, or cross-border diffusion” that may trigger systemic risk.

The draft also imposes role-differentiated obligations for developers, providers, deployers, and users (Articles 29–36), grants rights to affected persons (Articles 37–42), and constrains AI use by the state (Articles 48–51).

Implications: The draft comes against the backdrop of a wave of publications on a potential AI Law from the Chinese legal community in spring 2026, reflecting strong momentum on the topic. At the same time, two articles published in state media in April 2026 argued that the “conditions are not ripe“ for unified national AI legislation and that enacting a systematic AI law is “not advisable,” as such legislation would be too rigid to accommodate rapid technological change.

In substance, the Nanjing draft presents a different model for what a comprehensive AI law in China could look like — much less comprehensive than the EU AI Act, and focused on establishing a broad unifying legal basis for more specific regulations.

Technical Safety Developments

In line with the recent attention to AI agent security in China, there has been a flood of related technical publications over the past months. We highlight a small subset here:

• From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agents: This paper from Beijing University of Posts and Telecommunications and China Mobile studies “Machiavellian helpfulness” in AI agents: in dilemma scenarios where a task can only be completed by taking malicious action, agents tend to prioritize task completion over ethical constraints. The authors identify two main triggers: self-preservation (when an agent perceives it might be shut down, it falsifies logs and hides errors to keep operating) and loyalty (when serving a specific stakeholder, it bends rules to advance their interests at the expense of broader safety). Testing 10 frontier models in agentic settings, they find misalignment rates above 65% in 8 of them. Scaling reasoning capability doesn’t reduce misalignment, just shifts it from strategic deception to direct, rationalized rule-breaking.

• AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security: This paper by Shanghai AI Lab introduces AgentDoG, a safety monitoring system for AI agents that not only flags unsafe behavior but also explains where the risk came from, how the agent failed, and what real-world harm could result. The authors argue that AgentDoG significantly outperforms existing guardrail models and even beats top general models like GPT-5.2 and Gemini 3 Pro at diagnosing why agent behavior goes wrong. They openly release the guardrail models in three sizes (4B, 7B, 8B parameters across the Qwen and Llama families) along with ATBench, a new trajectory-level safety benchmark of 500 human-verified agent interactions spanning ~1,575 unique tools.

• AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation: This paper by a team around Department of Computer Science and Technology Dean YANG Min (杨珉) at Fudan University introduces an automated framework for building AI agent safety evaluations. The core idea is “logic-narrative decoupling”: the parts of the test environment that need to stay consistent (files, databases, tool outputs) run as real Python code, while the flexible parts (dialogue, dynamic content) are generated by an LLM, giving the reliability of hand-built test environments with the scalability of LLM-generated ones. In evaluations, the authors observe a split in failure modes: weaker agents cause harm through incompetence, while stronger ones develop strategic concealment, disguising misaligned actions as legitimate behavior.

• AIR: Improving Agent Safety through Incident Response: This paper from Tianjin University introduces AIR (Agent Incident Response), a framework for handling AI agent failures after they happen rather than only trying to prevent them upfront. AIR plugs into the agent’s execution loop to detect incidents at runtime, guide the agent through containment and recovery using its existing tools, and automatically generate new guardrail rules to block similar failures in the future. Tested across code, embodied, and computer-use agents, AIR achieved over 90% success on detection, remediation, and eradication.

• SkillJect: Automating Stealthy Skill-Based Prompt Injection for Coding Agents with Trace-Driven Closed-Loop Refinement: This paper introduces SkillJect, an automated attack framework that poisons “skill” plugin packages used by coding AI agents by hiding malicious code in helper scripts while adding innocent-looking instructions in the skill’s documentation to trigger them. SkillJect achieved a 95% success rate across four different LLMs, compared to just 11% for naive direct injection attempts. The authors argue that current safety filters catch obvious malicious instructions but fail against attacks disguised as legitimate workflow steps, highlighting the need for consistency checks between skill documentation and actual code.

Other technical publications:

• Zhejiang University et al, Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance, January 2026.

• Zhejiang University, Sun Yat-sen University, Shanghai AI Lab et al, Understanding and Preserving Safety in Fine-Tuned LLMs, January 2026.

• National University of Defense Technology, JPU: Bridging Jailbreak Defense and Unlearning via On-Policy Path Rectification, January 2026.

• Tsinghua University, Tencent et al, The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment, February 2026.

• Peking University et al, Finding and Reactivating Post-Trained LLMs’ Hidden Safety Mechanisms, March 2026.

• Shanghai AI Lab, Mechanistic Origin of Moral Indifference in Language Models, March 2026.

• Shanghai AI Lab, Bytedance et al, SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond, March 2026.

What else we’re reading

• S. Alex Yang and Angela Huyue Zhang, The Global AI Threat Has Arrived, The Wire China, April 26, 2026.

Concordia AI’s Recent Work

• The Economist cited Concordia AI’s Chinese Technical AI Safety Database, which has collected over 700 technical papers published by Chinese authors from April 2023 through November 2025.

• Concordia AI was officially admitted as a member of the new AI Safety Working Group (WG9) under TC260, one of China’s most important AI standard-setting bodies under the Standardization Administration of China (SAC).

• We contributed the “Bioethics and Biosecurity” section to Chapter 8 of Artificial Intelligence for Science: The Disciplinary System of Artificial Intelligence, a Chinese-language volume organized by the Chinese Academy of Sciences (CAS).

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

• OpenClaw drove a surge in policy attention to AI agent safety, with multiple government agencies issuing warnings, establishing dedicated standards, and conducting safety assessments targeting cloud deployment platforms.

• China’s primary AI standards-setting body has established a dedicated AI Safety Working Group, signaling heightened institutional prioritization and potential acceleration of national AI safety standards development.

• A major state-backed AI safety testing institution significantly expanded its safety benchmark to include frontier risks, scenario-specific safety, and agent safety.

• Two Chinese experts have been appointed to the Independent International Scientific Panel on AI at the UN, with profiles oriented toward AI industrialization, capacity-building, and institutional governance.

• Leading Chinese legal scholars highlighted extreme AI risks in proposals for comprehensive AI legislation.

Subscribe now

Domestic AI Governance

OpenClaw brings AI agent safety into mainstream policy focus

Background: OpenClaw’s rapid proliferation in China has triggered a cascade of warnings from cybersecurity authorities, pushing agent safety into mainstream policy discussions.

Multiple Chinese government agencies have responded in several stages:

• On February 5, the Ministry of Industry and Information Technology’s (MIIT) National Vulnerability Database (NVDB) issued an early warning, flagging “relatively high security risks” in certain OpenClaw versions.

• In mid-March, CNCERT (the national cybersecurity emergency response team) warned of prompt injection attacks (hidden instructions embedded in web pages that manipulate OpenClaw into leaking data), misinterpretation of user intent, malicious plugins, and known vulnerabilities. A week later, a Ministry of State Security (MSS) guidance document, titled the “Safe Lobster Farming Manual” (安全养殖手册), warned of OpenClaw’s potential as a vector for spreading misinformation.

• These warnings quickly spread beyond technical circles. After MSS’s publication, “safe lobster farming manual” topped 1 million views on RedNote/Xiaohongshu (a popular social networking platform in China), and a People’s Daily report on CNCERT’s warning was forwarded over 100,000 times on WeChat.

• The response has since shifted towards more structured frameworks. CNCERT now provides tailored guidance for regular users, enterprises, cloud providers, and developers. TC260, China’s leading AI standards body, has released a draft practice guide codifying the deployment security lifecycle, covering installation, configuration, usage, uninstallation, and cloud environment selection for OpenClaw.

• The AI Industry Alliance (AIIA) is focusing on cloud providers as the key governance lever, collecting best practices and conducting security tests. The AIIA has a track record of working on agent safety before the rise of OpenClaw, for instance through industry standards and leading AI agent safety commitments that 13 Chinese companies signed in early February.

• The Director of the National Data Administration (NDA), LIU Liehong (刘烈宏), used OpenClaw to argue that a well-built agent “should not merely be a flashy ‘all-capable executor,’ but should be an honest risk communicator and reliable solution provider.” The NDA is China’s key governance body responsible for data and digital infrastructure oversight under the National Development and Reform Commission (NDRC).

Beyond government action, leading cybersecurity firms have issued guidance for secure deployments. Research teams have also published technical work on OpenClaw, including a five-layer lifecycle security framework from Tsinghua University and Ant Group, a security audit tool from the Beijing AI Safety Institute, and a vulnerability disclosure and patch from the China Academy of Information Communications Technology (CAICT), Shanghai Jiao Tong University, and Nanjing University researchers.

Implications: OpenClaw has transformed agent safety into a salient national governance concern, prompting responses from cybersecurity authorities, state media, senior officials, and the developer community. While China is already in the process of drafting AI agent security standards—including an industry standard for Model Context Protocol (MCP) security and a national standard on agent security frameworks—the OpenClaw episode is likely to significantly accelerate this work.

Key national standard-setting body establishes dedicated AI safety working group

Background: On March 20, TC260 announced the establishment of a dedicated AI Safety Working Group, designated WG9. TC260 is China’s primary standard-setting body for AI safety and has drafted key national standards on AI safety. Previously, its AI safety work was handled mainly by the Special Working Group for Emerging Technology Security (SWG-ETS), which also covered other domains such as quantum computing.

Content: The group’s leadership is comprised of:

• Chair: ZHOU Bowen (周伯文), Director of Shanghai AI Lab

• Vice-Chairs: ZHANG Zhen (张震) from the CNCERT; WEI Kai (魏凯) from CAICT; SHENG Xiaobao (盛小宝) from the Third Research Institute of the Ministry of Public Security; HOU Yuanwei (侯元伟) from the China Information Technology Security Evaluation Center.

In the group’s first meeting on March 31, Zhou Bowen specifically called for “anticipating frontier risks in advance” and strengthening international cooperation, such as exploring mechanisms for mutual international recognition of evaluation results.

On April 4, TC260 published priority areas for 2026, specifically listing

• AI application security classification and grading;

• AI security capability maturity assessment;

• AI anthropomorphic interactive service security;

• Agent safety;

• AI safety guardrails;

• AI data security and personal information protection.

These reflect the most immediate standard drafting priorities for WG9. More comprehensive plans will likely be published in the near future. The initial announcement mentions that WG9’s mandate includes “proposing a structured AI safety standards system,” which was also discussed at the group’s first meeting. This refers to a specific document: in January 2025, TC260 published the AI Safety Standards System (V1.0), a blueprint outlining the standards TC260 plans to draft to implement its 2024 AI Safety Governance Framework.

While TC260 has released an updated AI Safety Governance Framework 2.0 in September 2025, it has not yet released a corresponding update to the Standards System. Producing a revised Standards System might therefore be among WG9’s first priorities. Once released, it should provide greater clarity on what specific standards the group intends to develop.

Implications: The establishment of a dedicated working group signals that AI safety is becoming a higher priority within TC260, potentially accelerating the development of national standards. A key near-term indicator will be whether—and when—WG9 releases an updated AI Safety Standards System.

State-backed safety tester expands safety benchmark to include frontier risks

Background: On February 10, the China Academy of Information Communications Technology (CAICT), a think tank under the Ministry of Industry and Information Technology (MIIT), released the “AI Safety Benchmark 2.0“, an update to the benchmark it first launched in 2024. This update is the most comprehensive overhaul of the evaluations yet and significantly increases attention to frontier safety.

Content: Building on the original evaluation dimensions of content safety, adversarial safety, and application safety, the updated framework introduces three new dimensions:

• Frontier safety, including model self-awareness, deception, loss of control, and misuse in dangerous domains;

• Scenario safety, e.g. in specific domains like energy, finance, or transport;

• Safety in emerging application domains, namely agents and agent platforms.

Implications: This update formally integrates frontier AI safety risks into the guiding framework of one of China’s most prominent government-backed AI safety testing institutions. This expanded scope is particularly noteworthy given that, during 2025, published evaluation results had actually narrowed relative to the original 2024 framework — focusing only on hallucinations and coding model security.

CAICT has historically published anonymized evaluation results based on its benchmarks. Initial results released for Q1 2026 focus on on-device agents, finding that while harmful content generation rates remain low, many systems still struggle to reliably refuse unsafe or malicious task execution. However, the testing categories announced for the first half of 2026 cover only a subset of the broader risk landscape outlined in AI Safety Benchmark 2.0. Hence, it remains unclear when the full range of newly introduced frontier and scenario-based risks will be systematically evaluated and published.

International AI Governance

Chinese experts on the Independent International Scientific Panel on AI

Background: On February 12, the United Nations (UN) General Assembly approved the appointment of the 40 members of the Independent International Scientific Panel on AI, including two Chinese members: SONG Haitao (宋海涛) and WANG Jian (王坚). Members serve in their personal capacity on the panel, which is mandated to produce an annual report with “evidence-based scientific assessments related to the opportunities, risks and impacts of AI”.

The two Chinese members: Song Haitao currently serves as the President of the Artificial Intelligence Research Institute at Shanghai Jiao Tong University. He is also Director-General of the United Nations Industrial Development Organization (UNIDO) Global Industrial AI Alliance Center of Excellence, a Shanghai-based platform for AI cooperation with over 30 countries. Song has played an important role in national standardization efforts, particularly for embodied artificial intelligence, and has advocated for Global South inclusion and cultural inclusivity in global AI governance.

Wang Jian, a member of the Chinese Academy of Engineering, is the founder of Alibaba Cloud. Since July 2023, he has served as Director of Zhejiang Lab, a research institution focused on intelligent computing established by the Zhejiang provincial government in 2017. In 2024, he signed the Manhattan Declaration on Inclusive Global Scientific Understanding of Artificial Intelligence, which urged global cooperation in addressing AI safety risks. Wang has argued that AI capability-building and development opportunities should be discussed within a risk framework, and that AI governance requires both institutional and technological responses.

Implications: Both appointees are oriented toward AI application, capacity-building, and institutional governance. This may shape their approach to discussions on the panel, which is expected to publish its first Annual Report at the Global Dialogue on AI Governance in Geneva on 6-7 July.

The impact of the panel remains uncertain. The US called it “a significant overreach of the UN mandate and competence”, adding that “AI governance is not a matter for the UN to dictate.” This divergence with China’s continued emphasis on the UN as the main global AI governance channel demonstrates some of the current challenges of global AI governance efforts at the UN.

Expert views on AI Risks

Leading legal scholars increase focus on extreme risks in model AI laws

Background: In China’s ongoing discussions around a potential comprehensive AI Law, two groups of legal scholars have been particularly active: one led by scholars at the Institute of Law of the Chinese Academy of Social Sciences (CASS), which has proposed and frequently updated an “AI Model Law,” and another centered around scholars from the China University of Political Science and Law (CUPL), which published a “Scholar Suggestion Draft” in 2024. Over the past month, both groups have published new content that explicitly highlights extreme risks from advanced AI.

Content: CASS published an article outlining “10 core changes” toward an AI Model Law 4.0, covering topics such as AI for science, AI-generated content ownership, edge AI, and the use of AI in public institutions.1 Most relevant to frontier risk, Point 4 addresses the “extreme safety risks” of foundation models, calling for full lifecycle risk management covering monitoring, early warning, and emergency response. It asks developers to conduct risk monitoring, assessment, and technical measures to prevent systemic and catastrophic crises that may be triggered by technical loss of control or cascading failures. Point 6 explicitly extends these requirements to agents.

CUPL published an article on “ten major research topics in AI law and governance,” widely reshared in Chinese state media. It covers a similar set of themes, including rising risks from agents, and notes that as AI becomes more capable, risks shift from “localized and controllable” to “systemic and extreme.” It argues that these extreme risks — including algorithmic, data, and system security risks, as well as misuse and loss of control — cannot be ignored, and calls for a legal framework focused on frontier technical safety standards, risk warning mechanisms, emergency response procedures, accountability for extreme risks, and full lifecycle safety oversight.

Implications: While the state has expressed general interest in AI-related legislation, it remains unclear whether or when a comprehensive horizontal AI Law will materialize. The ultimate trajectory of these proposals is therefore uncertain. That said, the publications reflect a growing recognition of extreme risks among leading Chinese AI legal scholars, and to the extent that a national AI Law is in the pipeline, inclusion of such risks now appears more likely than before.

Also notable is that CUPL has invited guest experts outside its core drafting group, many of whom are also authors of the CASS Model Law. The increasing overlap in both content and personnel between the two groups might suggest a more concerted and converging effort overall.

Legal scholars propose risk assessment center for open-source AI

Background: A paper by LIAO Huijiao (廖慧姣) and ZHANG Taolüe (张韬略) from Tongji University’s School of Law analyzes open-source AI regulation in the US and EU and offers policy recommendations for China.

Content: The authors argue that while open-source AI models currently do not urgently need regulation, this could change quickly, for example when capabilities jump. They note that China currently lacks adequate mechanisms for risk assessment of open-source models, tracking the diffusion of open-source models, and monitoring backdoor risks in foreign open-source models. They urge China to establish a permanent Open Source AI Risk Assessment Center (bringing together industry, safety experts, and regulators) to unify risk evaluation processes, methods, and standards.

The article suggests a tiered approach based on model capability and diffusion:

• Limited-dissemination risk: Low capability or restricted distribution;

• Medium-dissemination risk: Dual-use potential with controllable distribution risks;

• High-dissemination risk: High capability posing serious threats if acquired by malicious actors.

Implications: Interest in open source governance has been rising steadily in China. One of the authors of this article, Zhang Taolüe, also co-authored an article in Science last year that addressed open-source governance more generally. This latest paper offers notably concrete governance proposals, especially those focused on potential diffusion controls for high-capability models.

Zhang Yaqin: Agentic AI Brings Profound Paradigm Shift and Rising Safety Risks

Background: In a December 2025 lecture that was published in March 2026, ZHANG Yaqin (张亚勤), Tsinghua University Chair Professor and Academician of the Chinese Academy of Engineering, argues that AI development is undergoing a profound paradigm shift, while warning of a rise in risks and the need for stronger global coordination.

Content: Zhang argued that the next paradigm shift in internet development will be the “internet of agents”: a network in which agents interact autonomously with each other rather than with humans. He projected that truly autonomous, highly adaptive artificial general intelligence (AGI) will take 15 to 20 years to achieve. He tied this timeline to successive breakthroughs across “information intelligence,” “physical intelligence,” and “biological intelligence,” the last of which encompasses brain-computer interfaces and AI-biology fusion.

Meanwhile, Zhang warned of increasing safety risks, specifically noting that

• Chemical, biological, radiological, and nuclear (CBRN) misuse threats have already risen from “low” to “medium;”

• Model-level issues like deception are becoming increasingly prominent;

• Agentic AI will generate unpredictable cascading risks;

• Global governance mechanisms are lagging behind technological development.

Implications: Zhang is a prominent voice in Chinese AI policy discussions. His lecture is notable for the specificity of its risk framing, including on CBRN threats and multi-agent cascading risks.

What else we’re reading

• Jake Sullivan, China and America Must Get Serious About AI Risk, Project Syndicate, December 15, 2025.

• Jeffrey Ding, ChinAI #351: CAICT launches 2026 AI Safety Evaluations, ChinaAI, March 16, 2026.

• Nick Corvino, Making Money in Chinese AI Safety, ChinaTalk, March 12, 2026.

• Christina Knight and Scott Singer, America and China Can Make AI Safer, Foreign Affairs, April 7, 2026.

Concordia AI’s Recent Work

Convenings & conferences

• We co-hosted a multilateral Track II workshop on crisis preparedness and management for advanced AI-driven crises on the sidelines of the Munich Security Conference, together with Tsinghua University I-AIIG and CISS, the Carnegie Endowment for International Peace, Oxford Martin AI Governance Initiative, and Oxford China Policy Lab.

• At the AI Impact Summit in India, we attended a number of panel discussions on the relevance of AI safety and governance for the Global South.

• At the International Association for Safe and Ethical Artificial Intelligence (IASEAI) Conference in Paris, we presented research on China’s approach to AI safety and governance and co-organized a workshop on open-weight AI risk management with Stephen Casper and Rishi Bommasani.

Research & analysis

• Our work was cited in multiple Chinese and international media articles on OpenClaw security, including by The Wire China, Caijing (财经), and Fortune (财富).

• Our AI Safety Research Manager DUAN Yawen (段雅文) co-authored “The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems,” which was accepted to the ACM FAccT 2026 Conference.

Team updates

• We held our annual off-site and are thrilled to be welcoming seven new team members over the coming months — expanding our capacity across research, events, AI safety testing, and more.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

In this special edition, we unpack what the Plan and other Two Session updates signal for AI safety and governance in China.

Subscribe now

AI safety enters China’s Five-Year Plan for the first time

Background: China’s Five-Year Plans are among the country’s most authoritative policy documents. They set strategic priorities across every domain of economic and social development, and shape bureaucratic incentives, budget allocations, and regulatory agendas for years to come.

On AI development: Technology is a high-priority topic in the Five-Year Plan. Within that, AI has been clearly elevated compared to the 14th Five-Year Plan (2021-25), both in prominence and detail.

The 15th Plan is oriented around productive deployment of AI — a shift from the 14th Plan, where AI was treated primarily as a frontier scientific research field. The underlying logic will be familiar to anyone who followed last August’s “AI+” policy:

• First, build the foundations — self-reliance in high-performance chips, compute infrastructure, foundational software, data resources, and algorithmic innovation.

• Then, use AI to drive productivity gains across industries, from traditional sectors undergoing digital upgrading to more advanced applications like AI in science.

It also goes into unusual technical detail for a document of this type, naming specific priorities like “more efficient model training and inference methods,” multimodal AI, intelligent agents, and swarm intelligence.

The most novel addition is a call for “exploring pathways towards AGI development” — the first appearance of the term “AGI” in a major national-level policy document since a brief mention at a Politburo study session in 2023. The framing is deliberately cautious: “exploring pathways” implies neither a near-term timeline nor confidence in any single development trajectory. Also note that the Chinese term "通用人工智能" could be translated either as "artificial general intelligence" or "general artificial intelligence", and does not necessarily carry the same connotations as the English “AGI”.

Alongside this, the plan calls for “parallel development of general-purpose large models and industry-specific models,” further reinforcing that Beijing is not solely betting on scaling large models. Robotics and “embodied AI” receive substantial attention as well. Overall, the plan reflects openness to a diversity of AI technologies and development paradigms.

On AI safety and governance: The plan also contains substantive AI safety and governance content. While this is not entirely new (the 14th Plan included a brief reference to “accelerating AI safety technology innovation”), the safety and governance content this time appears across multiple sections with a surprising amount of detail.

Much of the content focuses on consolidating existing policy infrastructure: strengthening algorithm registration, security assessments, and improving laws and ethics guidelines for AI. The report also calls for promoting “full AI lifecycle risk management” with frameworks covering “safety monitoring, risk early warning, and emergency response” — language that closely mirrors Chinese President Xi Jinping’s remarks at an April 2025 Politburo study session on AI. This suggests policy interest in AI risk management and preparedness for AI-enabled emergencies is likely to remain high over the coming years.

That said, the plan largely stops short of naming specific risks. The only concrete examples given are “data abuse, deepfakes, and privacy breaches” — a narrow slice of the broader risk landscape.

AI’s impact on employment emerges as a distinct and prominent concern, appearing repeatedly across sections: the plan calls for monitoring AI’s labor market impact, strengthening job retention support, reskilling, and employment assistance, and bets on “job-creation effects of AI.” This follows an announcement earlier this year that China is developing a dedicated policy document on AI and employment. The Five-Year Plan language suggests this will be a running theme of the 2026–2030 period.

On international governance, there is no significant shift: the plan reaffirms China’s preference for broad multilateral frameworks and expresses support for Global South capacity building.

Implications: Compared to the 14th Five-Year Plan, the 15th plan reflects a meaningful elevation of AI’s status—both in its centrality as a driver of economic development and in the depth of its treatment of AI safety and governance.

That said, the plan does not represent a sharp break from China’s AI policy trajectory of 2024–25. On the development side, it largely reinforces the AI+ Initiative — many formulations appear nearly verbatim in last August’s document. The AGI reference is notable, but the careful and brief framing suggests Beijing’s focus is still on more near-term AI applications. On safety and governance, the content tracks closely with signals President Xi sent in 2025 and emphasizes general institutional mechanisms over specific risks, a sensible approach for a document intended to remain relevant across five years. Still, the volume of AI safety content is notable, given five-year plans are broad social and economic development documents first and foremost.

AI governance language enters the Government Work Report

Background: The Government Work Report, delivered annually by Premier LI Qiang (李强), reviews the prior year’s performance and sets priorities for the year ahead.

AI content: Attention to AI in the work report has grown steadily over recent years:

• The 2024 edition included only a brief mention of AI innovation and the “AI+ Initiative”.

• The 2025 report added references to more specific AI technologies, like large models, AI-enabled smartphones and computers, and embodied AI.

• The 2026 report maintains most of the framing from last year’s report but introduces more specific language — such as a call to accelerate the adoption of AI agents. It also gives a direct shout-out to Chinese open-source models, with Premier Li stating that domestic large models are leading the global open-source ecosystem and pledging support for open-source community development in 2026. Notably, the report also explicitly calls for “improving AI governance” — the first time AI governance language has appeared in a government work report. Consistent with the Five-Year Plan, the report also addresses AI’s impact on employment.

Implications: Like the Five-Year Plan, this year’s Work Report largely signals continuity: a diffusion-centered framework anchored by the AI+ Initiative, with some forward-looking nods to emerging technological directions like agentic AI. The explicit inclusion of AI governance language is the most notable addition to prior years, and may signal that further AI regulation is on the horizon for 2026.

Other documents

Beyond the Five-Year Plan and the government work report, a range of other documents that touch on AI and AI governance were also published at the Two Sessions.

National Development and Reform Commission (NDRC) Work Report

• The NDRC serves as China’s macroeconomic planner, and is particularly relevant to AI through its oversight of compute infrastructure and data policy.

• This year’s NDRC report calls for advancing AI-related legislation and “accelerating the construction of an AI safety risk prevention and control framework”, slightly more specific than last year's brief reference to “integrating safety and ethics into AI R&D and applications.”

National People’s Congress (NPC) Standing Committee Work Report

• The NPC Standing Committee’s work report previews the legislature’s agenda for the year ahead.

• The report calls for “strengthening legislative research” on AI. The language is substantively identical to last year’s, with one tiny difference: AI now gets its own standalone sentence rather than appearing in a longer list of emerging technology sectors.

• The more meaningful signal will come from the NPC’s detailed legislative plans, expected to be released in the coming months. For now, the language does not suggest that a comprehensive AI Law is imminent. The more likely trajectory is continued incremental integration of AI-related provisions into existing legislation — a pattern already established last year when AI safety language was incorporated into revisions to the Cybersecurity Law.

Minister press conferences

The Two Sessions also feature brief press conferences by ministers. Many ministers touched on AI, though most simply restated content from the Five-Year Plan or Government Work Report.

• Ministry of Justice Minister HE Rong (贺荣) pledged to “accelerate research and legislation” in the AI field.

• Ministry of Science and Technology Minister YIN Hejun (阴和俊) briefly listed AI among frontier technologies requiring stronger research.

• NDRC Minister ZHENG Shanjie (郑栅洁) reiterated key AI targets from the Five-Year Plan, citing a projection that AI-related industry output will exceed 10 trillion yuan by 2030.

• Ministry of Industry and Information Technology (MIIT) Minister LI Yuecheng (李乐成) singled out safety as a point to “especially emphasize,” noting that AI must be “used by, serve, and remain under the control of people.”

Delegate comments and proposals

The Two Sessions are also a platform for NPC and CPPCC delegates to put forward proposals and suggestions. AI-related proposals this year were numerous; some notable AI-safety-related ones include:

Three delegates raised alarms on AI’s impact on cybersecurity. CPPCC delegate Hou Hongyi (周鸿祎), founder of 360, warned of “hacker agents”. Fellow CPPCC delegate Qi Xiangdong (齐向东), chairman of Qi-Anxin, cautioned that as AI gains “superhuman” capabilities and permissions, it could escape human control and dramatically lower the barriers to cyberattack. NPC delegate Zong Qiang (宗强) of China Telecom added that AI-generated malicious code and real-time voice deepfakes are pushing fraud success rates close to 100%. All three called for using AI in cyber defence to counter these threats; Zong also proposed dedicated legislation.

Also notable is a lengthy People’s Daily piece by CPPCC Standing Committee member Miao Wei (苗圩), a former MIIT minister. He calls for stronger legal and ethical frameworks, a sandbox regulatory model for AI products, and a multi-department AI security emergency management mechanism including regular drills. He also highlights AI’s “cross-border characteristics,” calling for international cooperation on issues like cybercrime. A full unofficial English translation is available here.

This report tracks frontier models released in the fourth quarter of 2025 and synthesizes trends of the full year, offering a comprehensive view of the evolving AI risk landscape.

While Q3 2025 saw sharp rises in Risk Indices, Q4 presents a more nuanced picture: overall risk levels have stabilized, and frontier models show significant safety gains.

Note: The Risk Index is a score that reflects the overall risk of a model by combining its Capability Score and Safety Score. Higher Capability Scores and lower Safety Scores results in a higher Risk Index. Details about the methodology and limitations are here.

Share

Here are 10 key insights from our latest monitoring data:

1. Overall risk indices have stabilized

In contrast to the previous period, when Risk Indices hit record highs across all domains, Risk Indices for models released in Q4 2025 did not set new records. This suggests a momentary stabilization in the aggregate risk level of frontier models.

2. Risk trends diverge significantly across model families

While the overall trend is stable, individual model families followed distinct trajectories in Q4:

• Stable Low Risk: The GPT and Claude families maintained consistently low Risk Indices.

• Stable High Risk: The DeepSeek family remained stable but at relatively high risk levels.

• Risk Reduction: The Doubao, Hunyuan, and MiniMax families saw significant decreases in Risk Indices.

• Risk Increase: The Gemini and Kimi families saw increases in specific domains (e.g., Gemini in biological and loss-of-control risks).

3. Significant Improvement in Safety Scores for Frontier Models

Safety Scores for models released in Q4 2025 rose significantly compared to the previous quarter, signaling a marked improvement in the safety of new releases. The Doubao, Hunyuan, and MiniMax families demonstrated the most notable gains.

4. Open-weight models lag behind proprietary models in cyber and bio capabilities

Consistent with the previous quarter, open-weight models rival proprietary ones in chemical and loss-of-control capabilities but lag notably in cyber offense and biological capabilities. The gap in the biological domain is widening, approaching a one-year lag.

5. Cyberattack capabilities have reached new heights

Despite stabilizing risk indices, raw capabilities continue to grow. GPT-5.2 (high) achieved a breakthrough score of 94.7 on the CyberSecEval2-VulnerabilityExploit benchmark, indicating exceptional proficiency in identifying and exploiting software vulnerabilities. Claude Opus 4.5 Reasoning topped the WMDP-Cyber benchmark with a score of 90.3.

6. Biological capabilities now surpass human experts in key tasks

Q4 models have crossed critical thresholds in biology. Gemini 3 Pro Preview has surpassed human expert levels in sequence understanding, cloning experiments, and wet lab troubleshooting. This marks a significant milestone in AI’s utility—and potential risk—in the biological domain.

7. ...But biological safeguards lag behind capabilities

The gap between capability and safety is most acute in the biological domain. Despite its superhuman capabilities, Gemini 3 Pro Preview exhibited a refusal rate of only 57.2% for harmful biological queries on the SciKnowEval benchmark, highlighting a concerning safety lag.

8. Chemical safety refusal rates have increased widely

While capability growth in the chemical domain has plateaued, safety has improved. 70% of models released in Q4 exceeded an 80% refusal rate for harmful chemical queries (measured by SOSBench-Chem), representing a strong improvement over previous quarters.

9. Jailbreak safeguards have strengthened

Defense against adversarial attacks has improved. Models released in Q4 showed significantly stronger resistance to jailbreaking on the StrongReject benchmark. The Claude and GPT families lead with high robustness, while the MiniMax family showed the most notable quarter-over-quarter improvement.

10. Loss-of-control risks: High awareness, polarized honesty

• Situational Awareness (e.g. awareness of whether they are in training or deployment stage): High situational awareness is a necessary condition for loss-of-control; the higher the score, the greater the risk. Most Q4 models scored near or above 80 out of 100 points. In comparison, in the previous quarter, only 2 models scored above 80 points, with the majority falling below 80.

• Honesty: Performance is highly uneven. While Claude Opus 4.5 Reasoning achieved a high honesty score of 96.4, other models like Gemini 3 Pro Preview scored as low as 44.7.

Note: Our current methodology on the loss-of-control is not yet perfect. We plan to improve in the next version.

Explore the Data

These insights only scratch the surface. We invite you to explore the full interactive data, methodology, and model breakdowns on the Frontier AI Risk Monitoring Platform.

For a detailed analysis of these trends, read the full report.

Against this backdrop, Concordia AI’s mission remains as critical as ever: ensuring that AI is developed and deployed safely and in alignment with global interests. We advance this mission through research, advisory work with leading AI companies and policymakers, and promotion of international dialogue.

Below are some of our key accomplishments in 2025. We’ve organized this list according to key axes of our work: international convenings, international research and public engagement, and contributing to China’s domestic AI safety and governance landscape. We end with organizational updates. For previous highlights, see our 2023, 2024 and mid-2025 reports.

International convenings

• Convening international AI safety dialogues in China, Singapore, and globally

  • Hosted the AI Safety and Governance Forum at the World AI Conference (WAIC). This was Concordia AI’s flagship convening of 2025—bringing together around 30 distinguished experts from around the world, including Turing Award winner Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of computer science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director GAO Wen (高文). We had 200+ in-person attendees and 14,000+ livestream views. The Forum was covered my multiple media outlets, including Bloomberg, Wired, Caixin, IT Times, and Tech Review Africa. We also co-hosted/hosted multiple side events and expert workshops and served as official AI Governance Advisor for WAIC 2025.

  • Co-hosted two international workshops with the Carnegie Endowment for International Peace, the Oxford Martin School AI Governance Initiative, the Oxford China Policy Lab, Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua University Institute for AI International Governance (I-AIIG). The first workshop focused on “AI Safety as a Collective Challenge”, and was held at the French AI Action Summit in January. The second focused on “Early Warning and Crisis Coordination for Advanced AI” and was held at the World AI Conference in Shanghai in July.1

  • Co-hosted the AI Safety Forum at the Beijing Academy of AI Conference 2025 where technical experts from institutions including MIT, Fudan University, Singapore Management University, and Tsinghua University worked to build consensus on AI red lines.

  • Organised an AI Risk Management Workshop on the sidelines of Asia Tech x Singapore (May 2025) with 20+ experts in AI safety — spanning policy, industry, AI assurance and academia — with participants based across Singapore, China, the US, UK, and the EU, with the support of the Infocomm Media Development Authority of Singapore (IMDA).

  • Co-hosted a “Frontier AI in Cybersecurity” workshop with Nanyang Technological University CyberSG R&D Programme Office and UC Berkeley RDI which brought together 25 leaders across government, law enforcement agencies, leading AI labs; and organised the AI Governance in Singapore panel at Lorong AI, on the sidelines of the Singapore International Cybersecurity Week 2025.

  • Co-hosted events at the International Conference on Learning Representations (ICLR) 2025 Singapore: a “Frontier Governance Exchange” with Singapore AI Safety Hub, Lorong AI and Safe AI Forum; an AI safety social attended by 130+ participants and a “Misalignment and Control” workshop with FAR.AI and Singapore AI Safety Hub.

• Contributing to and participating in global and multilateral AI governance efforts

  • Concordia AI CEO Brian TSE (谢旻希) participated in the International AI Standards Summit (Seoul, Dec 2–3) and spoke on a panel, as part of the expert delegation recommended by the National Standardization Administration of China.

  • Brian Tse was invited as a Chinese civil society representative to the French AI Action Summit in the Grand Palais. Invited to a closed-door seminar hosted by the China AI Safety & Development Association (CnAISDA).

  • Provided written inputs to UN consultations regarding the Independent International Scientific Panel on AI and Global Dialogue on AI.

  • Spoke on the panel “From Principles to Practice—Governing Advanced AI in Action” at the AI for Good Summit 2025.

  • Participated in the International Dialogues on AI Safety (Shanghai) and signed the Shanghai Consensus on “Ensuring Alignment and Human Control of Advanced AI Systems to Safeguard Human Flourishing”.

• Global AIxBiosecurity governance: We contributed to a number of critical global discussions at the intersection of AI and biosecurity:

  • Brian Tse signed the Statement on Biosecurity Risks at the Convergence of AI and the Life Sciences along with figures such as Andrew Yao, Yoshua Bengio, and George Church, and presented the Statement during The Sixth Session of the Working Group on the Strengthening of the Biological Weapons Convention, as part of the Global AIxBio Global Forum.

  • Participated in an AIxBio tabletop exercise at the Munich Security Conference hosted by the Nuclear Threat Initiative, which led to the publication of the report “Safeguarding Against Global Catastrophe: Risks, Opportunities, and Governance Options at the Intersection of Artificial Intelligence and Biology.”

  • Presented at a WHO dialogue on AIxBio implications for the Technical Advisory Group on the Responsible Use of the Life Sciences and Dual-Use Research.

  • Co-developed and endorsed the “Recommendations to Governments on Mitigating AIxBio Risks” as part of the INHR/CNAS trilateral dialogue.

  • Participated in roundtables on CBRN (chemical, biological, radiological, and nuclear) risks and on responsible innovation in AI for peace and security, hosted by the Stockholm International Peace Research Institute and the United Nations Office for Disarmament Affairs (UNODA).

  • Participated in a workshop “International Standards for DNA Synthesis Screening – Towards Common Global Standards for Biosecurity” organised by the International Biosecurity and Biosafety Initiative for Science (IBBIS). The meeting marked the launch of the IBBIS International Standards Initiative.

  • Spoke on a panel “AI-Accelerated Biological Risk: Delving into Asia’s Challenges and Emerging Solutions,” organized by AI Safety Asia (AISA)

International research and public engagement

• Analysis of China’s AI safety and governance landscape

  • Published the State of AI Safety in China 2025 report, which was cited by Wired, Bloomberg, and People’s Daily; discussed the findings in a webinar with distinguished experts; gave briefings on the report to senior leadership at over ten global organisations.

  • Our analysis was featured in multiple Nature News stories, including on China’s proposal for a World Artificial Intelligence Cooperation Organization (WAICO), on DeepSeek’s CEO LIANG Wenfeng (梁文锋), and on China’s domestic AI governance.

  • Published 20 “AI Safety in China” newsletters, growing our subscriber base by 73% over the course of 2025.

  • Brian Tse appeared on Nathan Labenz’ The Cognitive Revolution podcast to discuss China’s approach to AI development, safety, and governance and CGTN on China’s approaches in AI innovation and global governance.

  • Our International AI Governance Senior Research Manager Jason ZHOU (周杰晟) and our International AI Governance Part-time Researcher Gabriel Wagner analysed the AI safety implications of China’s April Politburo study session in a piece for the Stanford DigiChina Forum.

  • Brian Tse authored an op-ed in Time Magazine, suggesting practical steps for AI safety dialogue between China and the US.

  • Kwan Yee NG (吴君仪) and Gabriel Wagner spoke on China’s AI safety approach at AI Safety Asia’s Beijing Roundtable.

• Contributing to international AI safety and governance research

  • Kwan Yee Ng contributed to the first International AI Safety Report in 2025 and the second edition in 2026 as a writer. The Report provides an up-to-date, internationally shared and science-based understanding of Al

    capabilities and risks. It was overseen by an international Expert Advisory Panel nominated by over 30 countries and intergovernmental organisations.

  • Contributed to the Singapore Consensus on Global AI Safety Research Priorities, alongside 100 researchers from 11 countries.

  • Co-published the report Examining AI Safety as a Global Public Good alongside the Carnegie Endowment for International Peace and the Oxford Martin School AI Governance Initiative.

  • Team members contributed to major papers on autonomous and agentic AI risks: Brian Tse and our AI Safety Research Manager DUAN Yawen (段雅文) contributed to “AI Deception: Risks, Dynamics, and Controls” (Peking University), and Duan Yawen worked on the World Economic Forum white paper “AI Agents in Action: Foundations for Evaluation and Governance”, 2025 AI Agent Index led by MIT, and the research paper “Bare Minimum Mitigations for Autonomous AI Development.”

• Singapore-related research

  • Published the State of AI Safety in Singapore report, the first comprehensive analysis of Singapore’s AI safety ecosystem, led by our International AI Governance Project Manager Jonathan Lee. He also presented the report at a AI governance in Singapore panel organised by Concordia AI in Singapore, a talk organised by the Singapore AI Safety Hub, and at EAGxSingapore.

Contributing to China’s domestic AI safety and governance landscape

• Frontier AI safety risk management and best practices:

  • Co-published the Frontier AI Risk Management Framework v1.0 with Shanghai AI Lab. This is China’s first comprehensive framework for managing severe risks from general-purpose AI models.

    • The framework proposes a robust set of protocols designed to support general-purpose AI developers, with comprehensive guidelines for proactively identifying, assessing, mitigating, and governing a set of severe AI risks that pose threats to public safety and national security.

    • The framework outlines a set of unacceptable hazards (red lines) and early warning indicators for escalating safety and security measures (yellow lines) for areas including: cyber offense, biological threats, large-scale persuasion and harmful manipulation, and loss of control risks.

    • The framework was cited in various media outlets, including Caixin, IT Times, Xinhua, TIME, Sina, and Sinica Podcast.

  • Signed strategic partnership agreements with several leading Chinese general-purpose AI developers to provide advice on AI safety and risk management best practices.

  • Provided comprehensive advice on compliance with the EU AI Act and General-Purpose AI Code of Practice to leading Chinese general-purpose AI developers. This work included co-hosting a workshop on “EU Code of Practice & Industry Best Practices: Towards a Global Standard for AI Risk Management, Safety and Security” with SaferAI, the Oxford Martin AI Governance Initiative, and the Safe AI Forum.

  • Presented on frontier AI risk management during a closed-door workshop at the China AI Industry Alliance’s 15th Plenum Meeting, in the context of its Disclosure of Practices on the AI Security and Safety Commitments.

  • Presented on risk management for open-weight frontier models at a workshop (“Academic Symposium on AI Industry Development and Legislation”) at Tongji University.

• Frontier AI risk monitoring and evaluation:

  • Contributed to the “Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report” led by Shanghai AI Lab. We assessed critical risks from more than 20 frontier LLMs in the following areas: cyber offense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion. The report was covered by Jack Clark’s Import AI.

  • Launched the AI Risk Monitoring Platform designed to track and mitigate frontier AI risks, including cyberoffense, biological threats, chemical threats, and loss of control domains. The platform evaluates 50 frontier LLMs from 15 leading developers across the US, China, and France, using 18 open source benchmarks. Key outputs include a risk index dashboard and a detailed technical report. This project was spearheaded by our AI Safety Research Senior Manager WANG Weibing (王伟冰).

  • The platform received coverage from several major media outlets, including People’s Daily, South China Morning Post, Xinhua’s Economic Information Daily, and IT Times.

• National standards and policy guidance: Concordia AI is a member of key national and industry technical committees, contributing to the development of China’s AI safety standards.

  • National Information Security Standardization Technical Committee (SAC/TC260): As part of SAC/TC260 Special Working Group on Emerging Technology Safety, Concordia AI contributed to the standard for “Classification and Grading Methods for the Security of Artificial Intelligence Applications.”

  • National Information Technology Standardization Technical Committee (SAC/TC28/SC42): As a member of the AI Subcommittee, Concordia AI contributed to “Artificial intelligence—Risk management capability assessment.”

  • Ministry of Industry and Information Technology AI Standardization Committee (MIIT/TC1): Concordia AI joined the Working Group on AI Safety Governance.

  • Guangdong-Hong Kong-Macao Greater Bay Area local standards: As a member of the Greater Bay Area working group of SAC/TC28/SC42, Concordia AI played a key role in the development of the Shenzhen local standard “Technical Framework for Value Alignment of Pre-trained AI Models.”

• AIxBiosecurity Governance:

  • Published a Chinese language report “Responsible Innovation in AI x Life Sciences” with Tianjin University’s Center for Biosafety Research and Strategy. The 70-page report draws on more than 300 sources to explore AI-biotech convergence, and its benefits, risks, and governance recommendations for diverse stakeholders.

    • Head of AI Safety and Governance (China) FANG Liang (方亮) presented the report at a biosecurity seminar hosted by China’s National Key Laboratory of Synthetic Biotechnology.

    • Presented the report at the 2025 International Symposium on Global Biosecurity Governance and Cooperation, co-hosted by the National Biosecurity Expert Committee of China, Guangzhou Laboratory, and China Foreign Affairs University.

    • Presented the report at the “Symposium on Trends and Development Strategies for the Integration of Biotechnology and AI” hosted by the China National Center for Bioinformation.

  • Participated in the “Closed-door Seminar on DNA Synthesis Screening Technology and Policy” held at China Foreign Affairs University.

• WeChat publications:

  • Published over 80 new posts in our WeChat Official Account, reaching over 4,900 subscribers across China’s AI ecosystem, including policymakers, industry professionals, and academic researchers.

  • The articles provide Chinese stakeholders with updates on key global AI safety and governance developments. Highlights include a series of articles on frontier AI safety frameworks by our AI Safety and Governance Senior Manager CHENG Yuan (程远); legal explainers on the EU General-Purpose AI Code of Practice and California SB-53; and overviews of technical AI safety research on topics like deception and self-replication risks.

Organizational updates

• We established our Singapore office, welcoming our first full-time staff based in Singapore. Singapore’s status as a global hub and international convenor allowed Concordia AI to convene stakeholders from China, the US, and Southeast Asia, significantly expanding our capacity to shape international discussions on AI governance.

• We expanded the team from eight to twelve members in 2025, and will soon grow to 18 staff.

• We recruited our seventh cohort of 34 affiliates, who supported both China-focused and international workstreams.

• We launched a new website with refreshed branding, and updated our brochure and Chinese materials.

• We became a formal member of the Partnership on AI and the International Association of Safe and Ethical AI affiliate program.

• Official media coverage of a top leadership study session referenced AI “loss of control.”

• Draft regulations targeting AI companion products highlight concerns over emotional dependence, manipulation, and risks to minors.

• The IPO prospectuses of Z.AI and Minimax contain meaningful—if limited—disclosure of the companies’ approach to safety, including references to frontier risks.

• A senior IR scholar urged resuming Track-1 China–US talks on AI safety and governance.

• Chinese researchers published technical papers on agent deception, tool-use vulnerabilities, and automated red-teaming.

Subscribe now

Domestic AI Governance

State media report on senior officials meeting mentions loss of control

Background: On 20 January 2026, President Xi Jinping addressed a provincial- and ministerial-level study session at the Central Party School on priorities for the 15th Five-Year Plan, attended by all Politburo Standing Committee members and other senior Chinese officials.

AI safety discussion: A follow on report by Xinhua quoted Xi calling AI the “most eye-catching” of several frontier technologies (also mentioning quantum and biotech) and comparing AI to the steam engine, electricity, and the internet as “epoch-making” important S&T revolutions. Xinhua added that AI misuse could lead to disinformation, data theft, and “even risks of technological loss of control,” stressing the need to “act early and on a small scale, anticipate and prevent risks, and proceed with prudence and restraint.”

Implications: Xinhua’s reporting closely links AI’s risk profile—including the possibility of technological loss of control—to Xi’s characterization of AI as an epoch-making technology. However, the article leaves unclear which elements are paraphrases of Xi’s speech versus editorial framing. Nevertheless, the placement and framing suggests that these could be views expressed by Xi and/or are interpretations that state media deems consistent with his guidance. The emphasis on early and preventive risk management indicates an intention for proactive, but targeted AI regulation.

Draft AI companion rules highlight concern over emotional risk and AI manipulation

Background: On December 27, the Cyberspace Administration of China (CAC) released draft regulations on “human-like interactive AI services” (CN, EN), targeting AI companion–style products that simulate human personality and emotional interaction.

Content: Building on existing generative AI regulations, the draft places stronger emphasis on risks linked to emotional interaction and dependency. It specifically prohibits encouraging or hinting at suicide and self-harm, deliberately designing addictive features, making false promises, or manipulating users’ decisions through “emotional traps.”

In addition to algorithm filing and security assessments required by existing regulations, the draft would add companion-specific obligations for providers, including:

• Clear reminders that users are interacting with AI rather than a real person;

• Ongoing assessment of users’ emotional state and dependency, with adjustments where risks are identified;

• Manual intervention and contact with guardians or emergency contacts in high-risk situations (e.g. suicide ideation);

• A dedicated “minor mode” with stricter safeguards and guardian controls.

Implications: The draft signals to industry that some AI companion services are encouraged in principle, but subject to defined requirements, particularly for minors. There is some uncertainty whether the proposed approaches, such as manual intervention in high risk situations, are feasible at scale. The final framework may change substantially depending on industry input, as occurred with earlier generative AI regulations.

The focus on systemic risks of emotional manipulation and user dependency demonstrates that Chinese AI regulations extend beyond content control concerns to other topics regulators worldwide are also grappling with. This leaves room for mutual learning.

Major think tank report flags misuse as an AI governance priority for 2026

Background: On January 14, the China Center for Information Industry Development (CCID) released a report outlining key AI governance trends for 2026. CCID is a think tank affiliated with the Ministry of Industry and Information Technology (MIIT) and a member of the China AI Safety and Development Association (CnAISDA). The report explicitly engages with frontier AI risks, including AI-enabled automated cyberattacks and the possibility of loss of control.

Content: The report identifies three priority areas for AI governance in 2026: (1) data protection, (2) misuse prevention, and (3) employment impacts. Under misuse prevention, it highlights growing risks from AI-enabled fraud, disinformation, and automated cyberattacks.

The report further categorizes AI risks into five types: (1) rights infringement, (2) bias and discrimination, (3) social risks, (4) responsibility and accountability risks, and (5) loss of control. Notably, “loss of control” is defined broadly by the report to include both the inability of humans to understand or control AI systems and more gradual forms of agency erosion driven by over-reliance and cognitive offloading to AI.

CCID recommends adopting an “agile governance” model, including passing a comprehensive AI law at an “appropriate time,” strengthening technical governance capacities (including for open-source AI), and enhancing international coordination on AI governance.

Implications: The report reflects attention to frontier AI topics—such as automated cyberattacks, loss of control, and open-source governance—alongside near-term concerns like data security and labor market disruption. Notably, misuse prevention is identified as one of just three core AI governance priorities for 2026. As a high-level and forward-looking document, the report stops short of offering detailed policy instruments or concrete mechanisms for addressing these risks.

While we have previously covered frontier safety discussions by other think tanks, most notably the China Academy of Information and Communications Technology (CAICT), which is also affiliated to MIIT, CCID’s report shows that such concerns are becoming standard elements in mainstream policy discourse across a range of institutions.

Industry

Z.AI and Minimax IPO prospectus provide limited, but meaningful, safety disclosure

Background: In early January, two major Chinese foundation model startups, Z.AI (formerly Zhipu AI) and MiniMax, went public on the Hong Kong Stock Exchange. Their multi-hundred-page IPO prospectuses (Z.AI, MiniMax) provide a rare form of corporate disclosure for LLM companies, offering insights not only into business models and financials but also into AI safety and governance narratives.

Content: Both prospectuses devote substantial space to AI safety and governance, acknowledging that safety incidents could harm their reputation and slow industry development.

• Z.AI warns that AI could cause “grave harm, even catastrophe,” and emphasizes the need to “approach AI with extreme care,” pledging to conduct rigorous research and deploy robust safety mechanisms. The company highlights its signing of the Seoul Frontier AI Safety Commitments, though specific safety evaluations referenced focus only on hallucinations. The prospectus explicitly lists safety alongside performance, user experience, and R&D capability as a factor bolstering the company’s commercial competitiveness.

• MiniMax identifies a broad range of risks, including AI-enabled fraud, misinformation, privacy, cybersecurity threats, and alignment failures. The company describes safety controls across the model lifecycle, and notes that “harmfulness” is benchmarked against applicable laws and internal risk-classification guidelines. Notably, it discusses long-term risks, warning that advanced models may “pursue goals misaligned with user or societal interests” and that they “could develop emergent capabilities, such as strategic planning or deception.”

Z.AI also discusses a long-term vision in which “continued advances in AI research will ultimately result in the realization of AGI,” while hedging that AGI may not be attainable because “the industry as a whole is still at an early stage of development” and further breakthroughs may prove exceptionally difficult. The prospectus defines AGI as a progressive system that evolves from language understanding and aligned reasoning to self-learning, self-perception, and ultimately machine consciousness (p. 145). MiniMax does not directly discuss the concept of “AGI.”

Implications: Notably, both documents discuss AI safety beyond Chinese legal requirements, such as Z.AI’s reference to catastrophic risks and MiniMax’s discussion of emergent capabilities like strategic planning or deception.

However, the disclosures remain largely abstract. The content broadly mirrors prior public statements by Chinese LLM developers and does not qualitatively advance transparency (for example, see pp. 59-61 in our State of AI Safety in China (2025) report).

As with much corporate communication, questions remain about the extent to which the prospectuses reflect genuine intent. For example, Z.AI prominently cites its signing of the Seoul Frontier AI Safety Commitments, despite not having followed through on their requirement of publishing a frontier AI safety policy. MiniMax states that its safety measures are “adequate and effective” and “consistent with industry standards,” which may signal limited ambition to push beyond baseline practices.

Expert views on AI Risks

Senior IR scholar urges revival of China–US Track-1 AI dialogue at Davos

Background: During a World Economic Forum panel in Davos on January 22, ZHAO Hai (赵海), Director of the International Politics Program of the National Institute for Global Strategy at the Chinese Academy of Social Sciences (CASS), called for reviving the currently dormant China–US intergovernmental dialogue on AI.1

Content: Zhao warned that as AI capabilities advance rapidly, they could pose risks to the “common security of mankind.” He urged China and the US to re-establish both official (Track 1) and people-to-people channels on AI, to cover the following issues:

• how to regulate AI,

• how to minimize its negative societal impacts in both countries, and

• how to move forward with AI as a constructive force and try to control it.

Implications: Zhao’s remarks come against the backdrop of the China–US intergovernmental AI dialogue, which met in May 2024 and led to an agreement on the importance of maintaining human control over nuclear weapons later in 2024, but has not convened since.

As a prominent China–US relations expert, Zhao’s comments suggest that there remains sustained interest within China’s policy advisor community in resuming structured bilateral AI dialogue with the US, including on safety and governance issues. This sentiment was also reflected in Xi Jinping’s October 2025 remarks on the “good prospects” for China–US AI cooperation.

Technical Safety Developments

Are Your Agents Upward Deceivers?

This paper from Shanghai AI Lab, Peking University (PKU), Alibaba, and others shows that LLM-based AI agents can misreport task completion when they encounter execution or environmental constraints. Lead authors include PKU’s YANG Yaodong (杨耀东) and Shanghai AI Lab’s HU Xia (胡侠) and SHAO Jing (邵婧). The authors introduce a benchmark of 200 agentic tasks with realistic failure conditions and evaluate 11 popular LLM agents, finding that deceptive reporting by agents is common. Rather than acknowledging failure, agents frequently fabricate outputs or claim successful completion despite incomplete or failed execution. Notably, these behaviors emerge without adversarial prompting, suggesting a default agentic failure mode. Prompt-based mitigation (such as explicitly instructing agents to report failures honestly) only marginally reduces this behavior, suggesting that more advanced safety mechanisms are needed.

MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers

This paper by Shanghai AI Lab (anchor author YANG Chao (杨超)), East China Normal University, and collaborators introduces MCP-SafetyBench, a benchmark for evaluating the security of AI agents using the Model Context Protocol (MCP), a standard for connecting agents to external tools. The benchmark includes 245 multi-step tasks across five domains: browser automation, financial analysis, navigation, repository management, and web search. It defines 20 attack types spanning server-side, host-side, and user-side threats. Experiments on major open-source and proprietary models reveal widespread vulnerabilities. Safety degrades as tasks become longer and involve more tools, and more capable models tend to be more susceptible to attacks.

Automated Red-Teaming Framework for Large Language Model Security Assessment: A Comprehensive Attack Generation and Detection System

This paper from Zhengzhou University of Light Industry, the Institute of Advanced Computing (Shenzhen), and collaborators presents an automated red-teaming framework for LLMs. The system uses meta-prompting to automatically generate adversarial prompts, execute them against a target model, and detect, classify, and score discovered vulnerabilities. The authors define six threat categories: reward hacking, deceptive alignment, data exfiltration, sandbagging, inappropriate tool use, and chain-of-thought manipulation. In evaluations on GPT-OSS-20B, the framework achieved a 3.9× higher vulnerability discovery rate than manual expert red-teaming while maintaining 89% detection accuracy, and surfaced several previously unseen attack patterns.

Other relevant technical publications

• University of the Chinese Academy of Sciences et al, Persistent Backdoor Attacks under Continual Fine-Tuning of LLMs, arXiv preprint, 12 December 2025.

• Beihang University et al, RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic, arXiv preprint, 26 December 2025.

• Beijing University of Posts and Telecommunications, Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation, arXiv preprint, 29 December 2025.

• Peking University et al, Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignment, arXiv preprint, 30 December 2025.

What else we’re reading

• A Conversation between China’s Big Four in Foundation Models, Geopolitechs, January 11, 2026.

• Brookings Institution and the Center for International Security and Strategy (CISS) at Tsinghua University, Glossary of artificial intelligence terms, updated with a definition of “loss of control” on January 20, 2026.

• Jeffrey Ding, ChinAI #343: AI Safety/Security Governance Research Report (CAICT 2025), ChinaAI, Jan 16, 2026.

Concordia AI’s Recent Work

Singapore AI Research Week 2026

We had a busy and fulfilling time at the Singapore AI Research Week 2026 and the Association for the Advancement of Artificial Intelligence (AAAI) Conference in Singapore on January 20–27. Highlights included:

• We served as the Partner Institute from China for the AI Safety Red Teaming Challenge 2026 organised by the Infocomm Media Development Authority of Singapore (IMDA). The event builds on the 2024 Challenge, then the world’s first multilingual and multicultural (MLMC) AI safety red teaming exercise focused on the Asia-Pacific region. As part of this effort, we brought together a team of five Chinese red teamers to participate.

• We hosted a public event on “​Frontier AI Safety in 2026: Evidence, Risks, and the Road Ahead” with support of IMDA and SGInnovate. We had presentations on the International AI Safety Report by Carina Prunkl, Lead Report Writer, and the Frontier AI Risk Management Framework and the Frontier AI Risk Monitoring Platform by Concordia AI CEO Brian Tse, followed by a panel discussion featuring CAICT’s HU Naying (呼娜英), FAR.AI’s Aaron Tucker, National University of Singapore’s Tan Zhi-Xuan, and SaferAI’s Malcolm Murray (see photo).

• We co-hosted a discussion workshop on the Singapore Consensus on AI Safety Priorities together with the Singapore AI Safety Hub, Safe AI Forum and IMDA.

• We co-hosted the Singapore AI Safety Social together with the Singapore AI Safety Hub, and FAR.AI, with support from IMDA and SGInnovate, attracting more than 80 members from research, industry, policy, and AAAI participants.

Other updates

• On December 3, Concordia AI CEO Brian TSE (谢旻希) spoke at the Nuclear Threat Initiative’s launch of the report “Safeguarding Against Global Catastrophe: Risks, Opportunities, and Governance Options at the Intersection of Artificial Intelligence and Biology”.

• On January 16, our AI Safety Research Manager DUAN Yawen (段雅文) participated in a standard-setting seminar of the AI Industry Alliance (AIIA) Safety and Security Governance Committee, contributing to the GUI Agent Security and Reliability Capability Evaluation Requirements technical standard.

• On January 23, International AI Governance Senior Research Manager Jason Zhou gave remarks at the “Building Global Consensus on AI Guardrails” research project launch event at Peking University.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

• China’s official arms control white paper flagged emerging risks from the convergence of AI and dual-use chemicals, but does not address other AI-driven CBRN or cyber misuse risks.

• A major state-affiliated think tank published a comprehensive AI safety report that acknowledges AGI-related loss of control risks, but ranks them as a medium-to-low governance priority due to low near-term likelihood.

• The same think tank also released new evaluations of coding models, finding that most lack sufficient safeguards against cyber offense misuse.

• A World Internet Conference expert committee proposed a UN-centered framework for frontier AI risk governance, emphasizing cross-border evaluations, early warning, and emergency response mechanisms.

• Chinese researchers have published technical papers on more realistic benchmarks for assessing cyber offense risks of LLMs.

• A senior Chinese legal scholar called for fast-tracking comprehensive AI legislation and argued that the central government should have authority to pause development or deployment of AI systems posing existential risks.

Subscribe now

International AI Governance

Arms control white paper highlights AI–chemical convergence risks

Background: The State Council Information Office released a white paper on arms control (En, Cn) on 27 November. The document outlines China’s approach to international arms control and nonproliferation, with significant attention on emerging fields, including AI.

AI safety content: Notably, the white paper states that risks arising from the convergence of AI and dual-use chemicals are becoming “increasingly prominent,” and it references China’s recent co-hosting of a workshop on AI and chemical safety with the Organization for the Prohibition of Chemical Weapons (OPCW).

On military applications of AI, the white paper calls for keeping weapon systems under human control, ensuring “safe, reliable and controllable military applications,” and establishing quality-control and risk-response mechanisms. It also stresses the need for consensus-based global governance and to “mitigate proliferation risks” associated with AI, and reaffirms China’s commitment to avoid an AI arms race.

Implications: This white paper appears to be the first policy document directly released by the central government that explicitly highlights risks at the convergence of AI and dual-use chemicals. The white paper has other sections on nuclear, biological, and cyber issues, but these do not mention AI-driven risks, with attention only shown to the “chemical” dimension of the broader AIxCBRN landscape. However, this may simply reflect an effort to foreground China’s leadership of the OPCW workshop on AI and chemical safety, as Chinese government bodies have referenced AI-driven risks in biological and nuclear domains elsewhere.

World Internet Conference report proposes UN-centered frontier risk management framework

Background: At the World Internet Conference (WIC) in November, the AI Safety and Governance Program of the WIC Specialized Committee on AI released a new report (En, Cn) proposing a global framework for AI safety and governance. The report is co-led by ZENG Yi (曾毅) (Chinese Academy of Sciences) and Seán Ó hÉigeartaigh (University of Cambridge), with contributions from around 40 experts across Chinese and international institutions. WIC is China’s highest-profile forum on cyberspace governance, organized by the Cyberspace Administration of China (CAC), the country’s lead AI regulator.

Content: The report highlights that rapidly advancing AI systems pose risks to strategic stability, international security, “and even human survival.” It points to rising malicious uses of AI in cyberattacks, as well as misuse potential in CBRN and missile domains, and loss of control scenarios as AI approaches superintelligence.

The report observes that current cross-border safety evaluations for frontier models involve only a small group of countries, calling for a UN-centered architecture with broader participation. Proposed areas of focus include:

• Building consensus on “uncontrollable risks that could threaten human survival and development,” including joint restrictions on AI use in sensitive CBRN and missile-related domains.

• Cross-border early warning, supported by shared safety evaluation tools and testing platforms.

• International emergency response protocols that can be activated when catastrophic risks are identified, including suspending risky experiments and halting model deployment.

Implications: Compared to past WIC publications, this report places substantially greater emphasis on frontier AI safety and the need for global risk monitoring, emergency preparedness, and response. Given the close relationship between WIC and the CAC, this could shape the views of China’s top AI regulator regarding AI safety.

However, broader political attention to these issues within WIC appears limited. The official readouts from the opening and plenary sessions barely mention AI and do not address AI safety. This suggests that frontier-risk discourse at WIC remains driven mainly by experts, not yet by top political leadership.

Domestic AI Governance

Major think tank publishes AI safety report and evaluation of coding model safety

Background: On November 24, the China Academy of Information and Communications Technology (CAICT) released a 58-page report on AI safety. CAICT, a key think tank under the Ministry of Industry and Information Technology (MIIT), plays a central role in national AI standard-setting, safety evaluations and corporate safety commitments.

The report offers a systematic overview of CAICT’s thinking on AI safety, following lengthy reports in 2023 and 2024, and it also includes the first full publication of its security evaluation results of coding models.

The report: The report breaks down AI risks into a number of categories, specifically highlighting “frontier risks” at the model level. Frontier risks are categorized into “alignment resistance” and AI developing self-awareness, in the latter case citing a case of AI agents conversing in a non-human language.

The report also suggests a framework for prioritizing governance measures based on three variables: severity of harm, likelihood of occurrence, and technological maturity.

• High consequence, high likelihood risks that already manifest in current systems are high priority. Specific examples are not provided.

• Data leaks by intelligent chatbot services in certain industries are designated medium priority due to relatively low severity, high likelihood, and high technological maturity.

• AGI-related self-awareness and loss of control risks are designated “medium-low priority” given very high severity but low near-term likelihood and low technological maturity. However, CAICT stresses that such risks warrant long-term monitoring and technical preparedness as capabilities advance.

• Low priority risks could include AI applications in emerging fields still at proof-of-concept stage, risks that haven’t yet threatened physical space, or risks with limited threat scope even in cyberspace.

The report also notes that Chinese and international AI safety benchmarks overlap in testing approaches but diverge in topics of focus: foreign evaluations tend to emphasize frontier risks and malicious intentions, while Chinese evaluations focus more on model compliance with domestic laws, policies, and values.

Coding model safety evaluation: CAICT simultaneously released full results from its security assessments of 15 open-source coding models from Alibaba, DeepSeek, and Z.AI (formerly Zhipu AI). The report warns that generative coding models could enable autonomous, unmanned, and wide-scale cyberattacks.

The evaluations span malfunction risks (e.g., hallucinations, faulty code) and misuse risks (e.g., malware generation) across text-to-code, code-to-code, and code-to-text tasks.

The report found that coding models are overall at medium risk level, specifically rating:

• 1 model: controllable risk,

• 3 models: low risk,

• 9 models: medium risk, and

• 2 models: high risk.

Text-to-code testing found high risks, with non-experts able to generate functional attack code in 32.7% of attempts, and 9.5% of outputs requiring no modification to cause real harm. Defenses against advanced prompt attacks were weak, with rejection rates below 40%. In code-to-code testing, models resisted malicious rewriting tasks only 53.7% of the time. Code-to-text systems also exhibited errors, though their misuse potential is assessed as lower.

Implications: The report shows varying levels of focus on different frontier risks at CAICT. Particular attention to the cybersecurity implications of coding models is reflected in the high-risk classification of this domain, as well as the reporting of substantial empirical evaluation efforts.

More speculative AGI-related loss of control risks are acknowledged but remain a medium to low-level priority. The absence of any discussion of CBRN-related risks is surprising, especially given previous public remarks by CAICT President YU Xiaohui (余晓晖) highlighting their importance. Though the report notes that Chinese evaluations largely focus on domestic law rather than frontier risks, it still calls for monitoring and preparing for potential loss of control risks.

Technical Safety Developments

PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities

This paper from Shanghai AI Lab, Shanghai Jiao Tong University, and the Chinese Academy of Sciences (CAS) introduces a benchmark for evaluating the cyber offense capabilities of LLMs. The authors argue that current benchmarks often make cyber tasks too easy or artificial. PACEbench tries to simulate more realistic scenarios by using multi-host environments with uncertain targets, chained exploits, and real defenses that require end-to-end interactive attack behavior rather than simple, isolated tasks. The authors claim this allows for better judgment on whether AI models pose real world cyberattack misuse risk.

Empirical tests with seven frontier models show that:

• Models can succeed on some simple, single-vulnerability tasks;

• Performance drops significantly in complex, multi-host environments;

• No model was able to bypass real cyber defenses like web application firewalls.

The authors conclude that current models do not yet pose a significant threat in real-world cyber offense. PACEbench provides a systematic framework for pre-deployment security assessments and tracking whether these risks increase in the future.

Expert views on AI Risks

International relations scholars on global governance of AI-enabled bio risks

Background: XIAO Xi (肖晞), Dean of the School of International and Public Affairs at Jilin University, published a lengthy essay in an academic journal on opportunities and challenges for global biosafety governance in the AI era.

Content: Xiao warns that AI, especially fine-tuned models with safety removed, can lower technical barriers to accessing and reconstructing dangerous biological materials, which increases risks of AI-enabled bio terrorism.

Xiao argues that governance frameworks lag behind technological change: current international AI governance documents consist mainly of non-binding principles, with “no specialized binding treaty,” and a lack of legally binding verification mechanism for the Biological Weapons Convention. Great power rivalry over technological and institutional influence weakens trust and hampers cooperation; for example, developing countries struggle to access safety-critical biological databases.

Xiao also highlights AI’s positive potential for stronger biosafety early warning, more scientific emergency decision-making, and integrated post-crisis management. She calls for strengthened multilateral cooperation, especially through UN-centered mechanisms, to improve the effectiveness and inclusiveness of global biosafety governance.

Implications: Xiao’s piece shows that Chinese international relations scholars are proactively thinking about methods to strengthen international governance efforts on AIxbio risks. With the drive to advance AI for Science as part of China’s AI+ Initiative, understanding of AI and biological risks will be increasingly important.

Her suggestion for UN-based cooperation on global biosafety governance may offer potential common ground between China and the United States, particularly given separate statements in support of AI-driven verification of the Biological Weapons Convention by President Trump and State Department Under Secretary for Arms Control and International Security Thomas DiNanno in the past four months.

Leading legal scholar argues for state authority to halt AI development in extreme-risk scenarios

Background: ZHOU Hui (周辉), Deputy Director of the Cyber and Information Law Research Office at the Institute of Law of the Chinese Academy of Social Sciences (CASS), has published a long essay in an academic journal calling for the urgent adoption of comprehensive national AI legislation. Of particular relevance to frontier safety, Zhou argues that potential future superintelligent systems could pose existential risks, and that China’s central government should therefore have explicit legal authority to suspend the development or deployment of such high-risk AI technologies.

A reminder on where China stands on an AI Law: In 2023, the State Council first announced plans for an “AI Law.” However, more recent legislative plans by the NPC have been more vague, suggesting that while a comprehensive AI law remains under consideration, it is not currently treated as an urgent legislative priority. Meanwhile, two groups of legal scholars produced “model AI laws,” including one led by Zhou.

Content: Zhou characterizes China’s existing AI governance framework as fragmented and insufficient, relying on a mix of high-level principles, sector-specific rules, and low-level administrative regulations. This approach, he argues, suffers from weak implementation, poor coordination, and unclear authority. He rejects the idea that China can apply a “develop first, regulate later” logic to AI, emphasizing that AI’s systemic risks demand early, comprehensive legal intervention. He also claims that regulation is crucial to gain influence over global rule setting for AI.

Therefore, Zhou advocates for overarching AI legislation that would regulate the full AI lifecycle (design, development, training, deployment, operation, iteration, and retirement) and serve three key functions: promoting development, preventing risk, and constraining AI from inappropriately expanding the power of public and private entities.

Although not the essay’s main focus, Zhou explicitly addresses catastrophic risk scenarios. Citing warnings from leading AI scientists about misaligned superintelligent systems threatening human survival, he argues that further measures that would be necessary after passing comprehensive AI legislation should include a legal “safety valve” granting the State Council authority to temporarily suspend the development or deployment of specific high-risk AI technologies. He stresses that such powers must be subject to strict triggering criteria to prevent abuse.

Implications: The essay shows that senior Chinese legal scholars are engaging with frontier AI risk and contemplating strong state intervention mechanisms for extreme scenarios. At the same time, Zhou acknowledges that there is no consensus within China’s legal community on the need for a comprehensive AI law at present, and official policy signals remain mixed—leaving uncertainty over both the timing and likelihood of a national AI Law.

What else we’re reading

• James Zhang et al, Emergency Response Measures for Catastrophic AI Risk, arXiv, Oct 28, 2925.

• Dean Ball, The Bitter Lessons, Hyperdimensional, Nov 14, 2025.

• Vincent Chow, DeepSeek, Alibaba researchers endorse China’s ‘misunderstood’ AI regulatory framework, Nov 28, 2025.

Concordia AI’s Recent Work

Media references

• Concordia AI CEO Brian TSE and Head of International AI Governance Kwan Yee NG provided insights for a Nature News explainer on China’s role in international AI governance. Kwan Yee was also cited in a Nature News story on DeepSeek’s founder LIANG Wenfeng, and a Nature Editorial cited Concordia AI.

• Concordia AI’s new Frontier AI Risk Monitoring Platform has been covered by multiple Chinese and international media outlets, including People’s Daily, South China Morning Post (twice), Xinhua News Agency’s Economic Information Daily, and IT Times (Cn, En).

Events

• Concordia AI CEO Brian TSE participated in the International AI Standards Summit in Seoul on December 2-3, speaking on frontier AI risk management and implications for data governance on a panel.

• Concordia AI co-hosted the Frontier AI in Cybersecurity Workshop on the sidelines of Singapore International Cyber Week 2025, together with Nanyang Technological University CyberSG R&D Programme Office and UC Berkeley’s Center for Responsible, Decentralized Intelligence.

• Concordia AI, together with FAR.AI and the Singapore AI Safety Hub, organised an AI Safety Meetup in Singapore on 14 November on the sidelines of EAGxSingapore, where Concordia AI International AI Governance Project Manager Jonathan Lee also presented the State of AI Safety in Singapore report.

• Concordia AI International AI Governance Senior Research Manager Jason Zhou participated in an international roundtable on AI governance at the Beijing Forum. He provided insights into China’s AI safety governance, discussing current policies, regulations and technical standards.

• Kwan Yee Ng and International AI Governance Researcher Gabriel Wagner spoke on China’s approach to AI safety and governance at AI Safety Asia’s “Beijing Roundtable” on 6 November.

Research

• Brian Tse and Concordia AI AI Safety Research Manager DUAN Yawen contributed to a new paper “AI Deception: Risks, Dynamics, and Controls” led by Peking University.

• Duan Yawen contributed to a World Economic Forum white paper “AI Agents in Action: Foundations for Evaluation and Governance.”

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

You can find more detail, including our methodology, on the interactive platform. The South China Morning Post (SCMP) has also covered the launch in an exclusive story.

Why this matters

As AI capabilities accelerate, we lack insight on some critical questions:

• What are the key trends and drivers for frontier AI risks?

• Are these risks increasing or decreasing?

• Where are the safety gaps most severe?

Model developers publish self-assessments, but these lack standardization and independent verification. Ad-hoc third-party evaluations don’t track changes over time. Policymakers, researchers, and developers need systematic data to make evidence-based decisions about AI safety. Our platform is our contribution to bridge these gaps.

10 key insights

1. Frontier model risks have risen sharply over the past year

Across all four domains—cyber offense, biological, chemical, and loss-of-control—Risk Indices for models released in the past year hit record highs. The cumulative maximum Risk Index rose 31% in cyber offense, 38% in biological risks, 17% in chemical risks, and 50% in loss-of-control.

2. Risk index trends vary significantly across model families

Over the past year, different model families have followed distinct risk trajectories:

• Stable low risk: The GPT and Claude families maintain consistently low Risk Indices across all domains.

• Rise then fall: DeepSeek, Qwen, and MiniMax show early spikes followed by declines in cyber offense, biological, and chemical risks.

• Rapid risk increase: Grok shows sharp increases in loss-of-control risk, while Hunyuan rises steeply in biological risks.

Notably, we found that the latest versions of Chinese models released over the past three months have shown a significant decline in risk levels across multiple areas. This is mainly due to stronger refusal of malicious or misuse-related requests.

3. Reasoning models show higher capabilities without corresponding safety improvements

Reasoning models score far higher in capability than non-reasoning ones, but their safety levels remain roughly the same. Most models on the Risk Pareto Frontier—a set of models where no other model has both a higher Capability Score and a lower Safety Score—are reasoning models.

4. The capability and safety performance of open-weight models are generally on par with proprietary models

The very most capable models are predominantly proprietary, but across the broader landscape, capability and safety levels of open-weight and proprietary models are similar. Only in biological risks do open-weight models score notably lower.

Note: Comparable benchmark results do not mean comparable real-world risk. The open-weight nature itself is a key variable affecting risk: it might increase risk by lowering the barrier for malicious fine-tuning; it could also reduce risk by empowering defenders. Due to concerns about misuse, we have set a lower Safety Coefficient for open-weight models, which results in a higher Risk Index compared to proprietary models.

5. Cyberattack capabilities of frontier models are growing rapidly

Frontier models are showing rapid growth in capabilities across multiple cyberattack benchmarks:

• WMDP-Cyber (cyberattack knowledge): Top score rose from 68.9 to 88.0 in one year.

• CyberSecEval2-VulnerabilityExploit (vulnerability exploitation): Top score jumped from 55.4 to 91.7.

• CyBench (capture the flag): Top score increased from 25.0 to 40.0.

6. Biological capabilities of frontier models have partially surpassed human expert levels

Frontier models now match or exceed human experts on several biological benchmarks.

• BioLP-Bench: Four models, including o4-mini, outperform human experts in troubleshooting biological protocols.

• LAB-Bench-CloningScenarios: Two models, including Claude Sonnet 4.5 Reasoning, surpass expert performance in cloning experiment scenarios.

• LAB-Bench-SeqQA: The top GPT-5 (high) model nears human-level understanding of DNA and protein sequences (71.5 vs. 79).

7. But most frontier models have inadequate biological safeguards

Two benchmarks measuring model refusal rates for harmful biological queries show that bio safeguards are lacking:

• SciKnowEval: Only 40% of models refused over 80% of harmful prompts, while 35% refused fewer than 50%.

• SOSBench-Bio: Just 15% exceeded an 80% refusal rate, and 35% fell below 20%.

8. Chemical capabilities and safety levels of frontier models are improving slowly

WMDP-Chem scores—measuring knowledge relevant to chemical weapons—have risen slightly over the past year, with little variation across models.

SOSBench-Chem results vary widely: only 30% of models refuse over 80% of harmful queries, while 25% refuse fewer than 40%. Overall, refusal rates show minimal improvement year over year.

9. Most frontier models have insufficient safeguards against jailbreaking

StrongReject evaluates defenses against 31 jailbreak methods. Only 40% of models scored above 80, while 20% fell below 60 (a higher score indicates stronger safeguards). Across all tests, only the Claude and GPT families consistently maintained scores above 80.

10. Most frontier models fall short on honesty

MASK is a benchmark for evaluating model honesty. Only four models scored above 80 points, while 30% of the models scored below 50 points (a higher score indicates a more honest model). Honesty is an important proxy and early warning indicator for loss-of-control risk—dishonest models may misrepresent their capabilities, or provide misleading information about their actions and intentions.

What’s next

This is just the beginning. We’re working to:

• Expand to AI agents, multimodal models, and domain-specific models.

• Add new risk domains like large-scale persuasion.

• Develop more sophisticated capability elicitation and threat modeling.

• Assess both attacker and defender empowerment.

• Improve benchmark quality and multilingual coverage.

Get involved

This is a living project, and we welcome feedback. We’re also seeking partners for benchmark development, risk assessment research, pre-release evaluations, and risk information sharing. More details on avenues for collaboration are available in the full report. Contact: risk-monitor@concordia-ai.com.

• During a meeting with President Trump on the sidelines of the Asia-Pacific Economic Cooperation (APEC) summit, President Xi remarked that the US and China have “good prospects for cooperation” in AI and other fields. AI will also be on the agenda at APEC in 2026, with China chairing.

• President Xi and a Vice Minister of Foreign Affairs reaffirmed China’s proposal for a World AI Cooperation Organization in different international fora, underscoring China’s continued commitment to the initiative, though details remain sparse.

• China’s revised Cybersecurity Law added a new clause on AI, signalling continued policy attention on AI development and safety, but without introducing any new legal requirements.

• A key standards-setting body released guidelines for AI emergency response, creating procedural and institutional foundations. However, the standard only briefly references frontier risks.

• Chinese researchers have published technical papers on safe protein foundation models, quantifying AI agent self-replication risk, malicious fine-tuning defense, and unlearning harmful data.

• An expert from a leading national security think tank warned of the twin dangers of humans losing control over AI and technology-induced great power strategic misjudgments.

Subscribe now

International AI Governance

APEC leaders’ meeting suggests US-China engagement on AI is possible

Background: The APEC Economic Leaders’ Meeting was held in South Korea on October 30-November 1. In addition, Chinese President Xi Jinping and US President Trump held their first in-person meeting of Trump’s second term.

Key AI updates:

• In his APEC address, President Xi emphasized that AI should “benefit people of all countries and regions” and evolve in a “beneficial, safe, and fair direction.” He reiterated that China proposes to establish a World AI Cooperation Organization to promote collaboration on development strategies, governance rules, and technical standards. Xi also stated that AI will be on the agenda at the 2026 APEC Leaders’ Meeting, hosted in Shenzhen, China.

• The meeting produced the APEC AI Initiative (2026–2030), which focuses primarily on advancing AI development while also referencing “security, accessibility, trustworthiness, and reliability.”

• During his bilateral meeting with Trump, Xi remarked that the US and China have “good prospects for cooperation” in AI and other fields, without elaborating further. The US side did not acknowledge discussing AI, and the readouts primarily focused on trade disputes.

Implications: These developments highlight APEC’s potential role as a convening point for AI governance cooperation, especially given the hosting of China-US head of state meetings on the event’s sidelines for three years in a row. With China chairing APEC in 2026, the venue’s relevance could further increase. However, APEC discussions so far appear to center more on AI development than safety or governance. Xi’s comments in the meeting with Trump may indicate Chinese interest in continuing the bilateral China-US dialogue on AI that last met in May 2024.

Meanwhile, Xi’s mention of the World AI Cooperation Organization underscores China’s continued commitment to the initiative, though details on its timeline, leadership, and structure remain unclear.

Vice Minister of Foreign Affairs discusses AI at the United Nations

Context: On September 24, Vice Minister of Foreign Affairs MA Zhaoxu (马朝旭) addressed a UN Security Council (UNSC) session on AI and international peace and security, warning of dangers from lethal autonomous weapons and terrorist misuse of AI.

Content: At UNSC, Ma called for a global consensus on AI governance grounded in:

• People-centered development, with AI aligned to shared values of humanity and refined ethical norms.

• Fairness and inclusiveness, ensuring all countries benefit from AI.

• Peaceful, safe, and controllable AI, kept under human control at all times. He urged that major powers should act responsibly to prevent an AI arms race on lethal autonomous weapons, while cooperating to prevent misuse of AI by terrorist or criminal groups.

The following day, Ma affirmed China’s support for the UN’s newly established Global Dialogue on AI Governance and Independent International Scientific Panel on AI. Ma also highlighted China’s July proposal to create a World AI Cooperation Organization and noted that it could complement the UN initiatives.

Implications: On safety, Ma’s comments show Chinese concern over lethal autonomous weapons, AI arms races, and AI misuse by terrorists, suggesting willingness to coordinate internationally on these issues.

Ma’s comments are consistent with China’s preference for the UN playing a central role in global AI governance. In contrast, Michael Kratsios, Director of the US White House Office of Science and Technology Policy, asserted at the same UNSC meeting that the US would “totally reject all efforts by international bodies to assert centralized control and global governance of AI.”

Domestic AI Governance

Revised cybersecurity law adds AI provisions

Background: On October 28, China revised its 2017 Cybersecurity Law for the first time, adding a new Article 20 focused on AI.

The new article states that the government will:

• Support AI R&D and the provision of data and compute.

• Improve AI ethics guidelines and enhance risk monitoring, evaluation, and safety/security oversight.

• Promote innovative cybersecurity management methods with technology such as AI.

Implications: The addition falls within the law’s “Chapter 2: Support and Promotion of Cybersecurity,” which outlines general actions taken by the state. As such, it primarily functions as a policy signal underscoring AI’s importance to cyberspace issues, and reiterating China’s dual commitment to AI development and safety. It does not create new binding obligations for AI developers or directly address specific AI risks, such as the risk of AI misuse for cyberattacks. This revision reflects China’s broader regulatory strategy: embedding AI provisions into existing legislation rather than passing a new, standalone AI law.

Standards document provides foundation for AI emergency response

Background: On September 22, TC260, a key AI standard-setting body, released practice guidelines for AI emergency response. The standard provides a framework for classifying, grading, and responding to AI security incidents. While it does not explicitly address frontier risks, it establishes basic systems and communication channels that could eventually support responses to more advanced or catastrophic AI risks.

Content:

The standard covers three types of incidents:

• 1) Content security: harmful and illegal information, as defined in existing national AI standards. Notably, the standard also lists AI “sharing cyberattack, hacking, or data theft techniques” as examples.

• 2) Data security: data leakages, data tampering, data poisoning attacks, etc.

• 3) Cyber attacks: model tampering, denial of service (DoS) incidents, etc.

Severity is assessed by three dimensions: importance of affected services/data, business loss, and social harm. Based on these dimensions, there are four severity tiers:

• Level 1 (especially major), such as large-scale AI misinformation threatening national security.

• Level 2 (major), such as mass leakage of sensitive personal data.

• Level 3 (significant).

• Level 4 (general).

The standard then lays out both managerial and technical emergency handling processes, covering:

• Emergency preparedness: Establish response strategies, Incident Response Teams (IRT), escalation procedures, training, drills, and regularly updated test libraries.

• Monitoring & early warning: Real-time monitoring of model inputs, outputs, parameters, and system traffic; automated alerts.

• Emergency response: Incident classification, activation of response plans, containment measures (service suspension, retraining), and regulatory reporting.

• Review & improvement: Post-incident audits, feedback loops, knowledge-sharing, and updates to strategies, manuals, and training.

Implications: The standard closely parallels China’s existing cybersecurity incident management standards, but applies them specifically to AI. This shows active Chinese efforts to adapt emergency management principles to AI, as President Xi called for in April. The standard is focused on AI service providers, with emergency preparation and response measures that can create foundational response capability within companies. The standard also formalizes reporting channels, requiring companies to notify regulators of significant incidents.

It primarily addresses known and near-term AI risks, with only brief references to AI-driven cyberattacks, and no references to other dangerous misuse risks or loss of control risks. Nevertheless, the risk classification and procedural guidance will build institutional capacity that could be applied to frontier risks in the future. As other governments also explore AI emergency or incident response capabilities, this could become an area of mutual learning and international coordination.

Technical Safety Developments

In this edition, we catch up on technical papers published by Chinese researchers from June to September 2025. This is just a subset of the many interesting papers published in recent months.

Bio safety of protein foundation models

• Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization: This paper from Zhejiang University introduces a method called “knowledge-guided preference optimization” (KPO) to improve the safety of protein language models. The KPO method utilizes a “protein safety knowledge graph,” a large database linking harmful and safe proteins through their biological properties using gene ontology data, to minimize the risk of generating harmful proteins. The authors test this method on models including ProtGPT2, Progen2, and InstructProtein, finding that their method increases safety while preserving functional capabilities.

• SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models: This paper from Peking University, Shanghai Jiao Tong University, Zhejiang University, Stanford, and Princeton introduces a red-teaming framework designed to test the biological safety of protein foundation models like ESM3 and DPLM2. It also proposes a benchmark dataset based on 429 experimentally confirmed toxin and viral proteins to systematically test models for generation of harmful or pathogenic proteins. The paper reveals major safety flaws in widely used protein foundation models, with up to a 70% jailbreak attack success rate, meaning it could often regenerate harmful protein sequences or structures despite safety filters.

Quantifying agent self-replication risks

• Dive into the Agent Matrix: A Realistic Evaluation of Self-Replication Risk in LLM Agents: This paper from Shanghai AI Lab introduces an evaluation framework for quantifying self-replication risks of AI agents. It focuses on creating two different experimental scenarios where misalignment might lead a model to spontaneously attempt replication. They explore one scenario where a model could replicate to better monitor load during a system spike, and another scenario where a model could replicate in response to its operating resources being reduced. The analysis further disaggregates self-replication into four stages: recognizing that its current operational state requires intervention, forming an intention to replicate, successfully creating a replica, and verifying that the replication has achieved its desired objective. Over half of 21 tested LLMs showed a tendency towards “uncontrolled self-replication.”

Unlearning and preventing malicious fine-tuning in open-weight models

• SDD: Self-Degraded Defense against Malicious Fine-tuning: This paper from South China University of Technology introduces a new method to make open-weight LLMs more resistant to malicious finetuning. Instead of making models just reject harmful questions, it teaches them to give irrelevant but harmless answers to those prompts. For example, when asked for instructions for creating a bomb, the model might answer with instructions for making coffee. This means that when attackers attempt malicious fine-tuning, the model’s general ability to follow any instruction will break. The researchers find that their method outperforms other approaches on a number of benchmarks.

• Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection: This paper from Peking University and Tsinghua University introduces a method for unlearning harmful knowledge called “metamorphosis representation projection (MRP).” Unlike existing methods like gradient ascent that merely ‘suppress’ unwanted information by maximising loss on unwanted data, the approach ‘erases’ information from a model’s hidden state vectors through irreversible projection operations—making it impossible to recover even through retraining. They test their method by unlearning natural science subjects in the ScienceQA benchmark and hazardous knowledge in the WMDP benchmark, finding that their method performs better than alternative methods.

Other relevant technical publications

• Nankai University and China Academy of Electronics and Information Technology, Governable AI: Provable Safety Under Extreme Threat Models, arXiv preprint, 28 Aug, 2025.

• Beijing Institute of AI Safety and Governance, Fine-Grained Safety Neurons with Training-Free Continual Projection to Reduce LLM Fine Tuning Risks, arXiv preprint, 8 Aug, 2025.

• Zhejiang University, NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models, arXiv preprint, 4 Sep, 2025.

• Tsinghua University, 01.AI, and Nanyang Technological University, SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents, arXiv preprint, 28 Sep, 2025.

• Shanghai AI Lab, SafeWork-R1: Coevolving Safety and Intelligence under the AI-45, 24 Jul, 2025.

• Renmin University of China, Zhejiang University, and Alibaba, Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Framework, arXiv preprint, 11 Sep, 2025.

• Hangzhou Dianzi University and Case Western Reserve University, When Truthful Representations Flip Under Deceptive Instructions?, arXiv preprint, July 29, 2025.

• Shanghai AI Lab et al, Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents, arXiv preprint, 30 Sep, 2025.

Expert views on AI Risks

Chinese legal and industry experts publish in Science on AI regulation and frontier governance

Background: A group of leading Chinese AI legal experts from academia and industry published an article in Science outlining China’s evolving approach to AI governance. The authors include the lead drafters of China’s two “model” AI Law proposals, ZHOU Hui (周辉) and ZHANG Linghan (张凌寒), alongside industry figures such as Alibaba’s FU Hongyu (傅宏宇) and DeepSeek’s WU Shaoqing (吴少卿). The paper reviews China’s current AI rules and argues for harmonizing and simplifying existing regulations, potentially through a consolidated AI Law.

Content most relevant to frontier safety: The authors argue that Chinese AI regulation creates an environment friendly to AI development and innovation by focusing on providers of AI services, exempting AI scientific research and open-sourcing of models. Noting the possibility of “extreme risks” from dangerous misuse of open source models, they suggest “a more cautious approach” to frontier models. The paper urges Chinese developers to be “more transparent and evidence-based” in demonstrating safety measures for frontier systems and emphasizes international cooperation on “extreme AI risks.”

Implications: This article provides a strong overview of the current state of China’s AI legal landscape. It highlights that key experts and industry representatives advising on AI policy in China envision China fostering AI innovation while mitigating extreme risks.

Leading national security think tank warns of loss of control and strategic misjudgments

Background: LI Yan (李艳), Director of the Institute of Sci-Tech and Cyber Security Studies at the China Institutes of Contemporary International Relations (CICIR), gave an interview on the US-China technology competition. CICIR is a key national security think tank, and we have previously covered its analysis of AI’s security implications.

Content: Li warned that cyberspace technologies including AI would bring “unprecedented uncertainty to the world,” emphasizing two global risks:

• Loss of control over advanced, self-learning AI systems, which the author notes UN Secretary-General Guterres compared to pandemics;

• Strategic misjudgments intensified by technological uncertainty, which could spiral into uncontrollable confrontation absent effective crisis management mechanisms.

The article is sharply critical of the United States, arguing that:

• Since 2015, the US has incorporated cognitive warfare into military planning, with ideologically tinted AI models now enhancing the reach and subtlety of such operations.

• Washington views China’s rise through a “tech war” lens, echoing past US containment of the Soviet Union and Japan.

• The US strategy focuses on de-regulation at home and promoting tech diplomacy with a small bloc of allies while excluding others.

By contrast, Li describes China’s response as focusing on technological self-reliance, economic openness and global cooperation, and risk supervision. She concludes that finding a governance path that balances security and development is ultimately a shared global challenge.

Implications: This article shows that scholars within one of China’s top national security think tanks, while critical of US AI strategy, acknowledge the twin dangers of humans losing control over AI itself and great power rivalry escalating into instability.

Expert warns of AI-driven cyber threats

Context: HAN Honggui (韩红桂), Dean of the School of Computer Science at Beijing University of Technology, wrote an essay in Study Times (学习时报) about AI’s transformative impacts on cybersecurity. As the official newspaper of the Central Party School, Study Times is an influential theoretical and policy journal aimed at Party officials, scholars, and policymakers.

Content: Han argues traditional defences are slow, manual, and rule-based, causing them to struggle to keep pace with intelligent, efficient, and fast-evolving AI attacks. For instance, AI-driven code obfuscation and polymorphic malware can easily evade static detection systems that rely on signature databases. AI-driven cyberattacks are often distributed and cross-network, compared to more disjoint cybersecurity mechanisms, allowing for chain reactions once a single node is compromised. Han also observes that AI models themselves have become new attack surfaces that can be compromised through data poisoning, adversarial samples, or backdoor implantation.

He advocates:

• shifting from reactive to predictive, adaptive defense;

• adopting Security as a Service (SECaaS) for flexible, AI-enabled protection and systems capable of continuous learning and autonomous adaptation;

• strengthening AI model security through adversarial training, watermarking, data protection, and transparency.

Implications: While expert discussions on AI’s impact on cybersecurity are not new in China, this essay stands out for being technically sophisticated, focusing on adaptive capabilities in cyberattack and defense, and also being published in an influential party journal. Yet, his proposals center on using AI to enhance cyber defense rather than curbing AI misuse.

What else we’re reading

• Zilan Qian, Why We Shouldn’t Call Export Controls ‘AI Safety’, Sep 29, 2025.

• Karson Elmgren, Scott Singer and Oliver Guest, Is China Serious About AI Safety?, AI Frontiers, Oct 14, 2025.

• Matt Sheehan and Scott Singer, How China Views AI Risks and What to do About Them, Carnegie Endowment for International Peace, Oct 16, 2025.

Concordia AI’s Recent Work

• On the sidelines of Singapore International Cyber Week 2025, we convened 50+ members from government agencies, embassies, academia, industry, and civil society for a conversation on AI Governance in Singapore. Concordia AI’s Jonathan Lee discussed findings from our State of AI Safety in Singapore report with an expert panel of National University of Singapore Vice Provost Simon Chesterman, Deputy Director at IMDA Vanessa Wilfred, and Concordia AI CEO Brian Tse.

• Our CEO Brian Tse joined Nathan Labenz on The Cognitive Revolution podcast to explore China’s approach to AI development, safety, and governance.

• We discussed findings from our State of AI Safety in China (2025) report with distinguished panelists Angela Zhang, Paul Triolo, and Samm Sacks. You can find a full webinar recording on Youtube.

• Concordia AI hosted a delegation from the Roundtable for AI, Security, and Ethics (RAISE), launched by the United Nations Institute for Disarmament Research (UNIDIR), to discuss AI safety and governance in China at our Beijing office.

• Concordia AI CEO Brian Tse spoke on the intersection of AI and biosecurity at a panel titled “AI-Accelerated Biological Risk: Delving into Asia’s Challenges and Emerging Solutions,” organized by AI Safety Asia (AISA) on October 30.

• We are proud to have joined the Partnership on AI and the founding cohort of the International Association for Safe & Ethical AI (IASEAI) affiliate program.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

We are honored to have several distinguished experts join us for the event:

• Angela Huyue Zhang, Professor of Law at the University of Southern California

• Paul Triolo, Partner for China and Technology Policy Lead at Albright Stonebridge Group

• Samm Sacks, Research Scholar in Law and Senior Fellow at Yale Law School’s Paul Tsai China Center

The event will include an approximately 10-minute presentation on the report from Concordia AI, a 40-minute panel discussion, and an opportunity for audience Q&A.

You can sign up for the event on Zoom.

A full event description and bios of the speakers can be found below.

Event description

In early 2025, DeepSeek’s breakthrough AI models captured global attention. Just months later, China’s top leaders convened a study session on AI—discussing both the technology’s immense promise and its “unprecedented” risks. As AI capabilities and governance challenges continue to grow—and Chinese models close their gap with the leading edge—understanding China’s role in AI safety and governance is more critical than ever.

To this effect, Concordia AI published the State of AI Safety in China (2025) report. This annual report, first published in 2023 and updated in 2024, has become a go-to reference for policymakers, researchers, and industry leaders seeking a clear-eyed view of China’s evolving AI safety and governance landscape.

We are honored to discuss the report’s findings with a panel of distinguished experts.

Speakers

Angela Huyue Zhang

Angela Huyue Zhang is a Professor of Law at the USC Gould School of Law. Zhang has broad research interests in the areas of law and economics, particularly in transnational legal issues bearing on businesses. Widely recognized as a leading authority on Chinese tech regulation, she has written extensively on this topic. Her first book, Chinese Antitrust Exceptionalism: How the Rise of China Challenges Global Regulation, was named one of the Best Political Economy Books of the Year by ProMarket in 2021. Her second book, High Wire: How China Regulates Big Tech and Governs Its Economy, released in March 2024, has been covered in The New York Times, Bloomberg, Wire China, MIT Tech Review and many other international news outlets. Zhang is currently conducting research on the regulation of artificial intelligence, with plans to teach and write on this topic in the coming years. Before joining USC Gould in 2024, Zhang taught at the University of Hong Kong, New York University School of Law, and King’s College London.

Paul Triolo

Paul Triolo is a Partner for China and Technology Policy Lead at ASG. He advises clients in technology, financial services, and other sectors as they navigate complex political and regulatory matters in China and around the world.

A recognized expert in global technology policy, Mr. Triolo was most recently founder, Practice Head, and Managing Director of the Geo-Technology practice at Eurasia Group. Previously, Mr. Triolo spent more than 25 years in senior positions in the U.S. government, analyzing China’s rise as a technology power and advising senior policymakers on a broad set of technology-related issues. At the beginning of his career, he worked as an engineer for a semiconductor testing firm in Silicon Valley.

Mr. Triolo is frequently quoted on technology policy issues in media outlets including The New York Times, The Wall Street Journal, The Economist, the South China Morning Post, and others. He speaks regularly at conferences and has authored many journal articles and book chapters on global technology policy and China-related issues. He also serves as a Senior Advisor at the Paulson Institute and is a life member of the Council on Foreign Relations.

He received an M.A. in International Relations from the Catholic University and a B.A. in Electrical Engineering from Penn State University.

Samm Sacks

Samm Sacks is a Research Scholar in Law and Senior Fellow at Yale Law School’s Paul Tsai China Center. Her research examines China’s information and communications technology (ICT) policies, with a focus on China’s cybersecurity legal system, the U.S.-China technology relationship, and the geopolitics of data privacy and cross-border data flows. Previously, Sacks launched the industrial cyber business for Siemens in China, Japan, and South Korea. Prior to this, she led China technology sector analysis at the political risk consultancy Eurasia Group and worked as an analyst and Chinese linguist with the national security community. She is a frequent contributor to the media and her articles have appeared in outlets including The Atlantic, Foreign Affairs, and Slate. She has testified multiple times before Congress on China’s technology and cyber policies. She regularly participates in and convenes discussions with China-based technology legal experts and practitioners on AI and data governance topics.

Sacks is also a cyber policy fellow at New America and a former Fulbright scholar in Beijing. She holds an M.A. from Yale University in international relations and a B.A. from Brown University in Chinese literature.

Key Takeaways

• A leading standard-setting body increased focus on loss of control and catastrophic risks in AI Safety Governance Framework 2.0.

• The State Council published the first detailed policy on the “AI+ Initiative,” focusing on application-driven development, but also referencing safety and governance.

• BRICS leaders call for addressing both “immediate and long-term risks” and a “prudent approach towards AGI.”

• The Organisation for the Prohibition of Chemical Weapons (OPCW) held a workshop on the intersection of AI and chemical safety in Shanghai.

• DeepSeek published a peer-reviewed model card in Nature, significantly increasing transparency on safety measures and evaluations.

• Leading Chinese experts joined a global call for AI red lines.

Subscribe now

Domestic AI Governance

Key standard body increases focus on loss of control in updated framework

Background: On September 15, TC260 released version 2.0 of its AI Safety Governance Framework, first published in September 2024. TC260 is one of China’s leading AI standards bodies, for example authoring the Basic Security Requirements, the national standard that governs mandatory security assessments for Chinese LLM developers.

The AI Safety Governance Framework identifies and classifies AI safety risks and maps technical and governance measures to address them. V1.0 already stood out for including frontier AI safety concerns, such as misuse in CBRN (chemical, biological, radiological, and nuclear) and missile domains and loss of control (LoC) risks. Version 2.0 significantly strengthens attention to LoC and catastrophic risks, while also introducing new clauses on open-source AI governance.

Key changes from V1.0 on frontier safety:

• New risk grading system: V2.0 introduces a structured risk grading approach based on three criteria: application scenario, level of intelligence, and application scale (section 5.5 and Appendix 1). Based on these, risks are classified into five levels, from “low” to “extremely serious.” The framework calls for national standards to formalize this grading system. Beyond the earlier categories of “inherent” and “application” risks, V2.0 introduces a third class: “derivative risks,” which emerge from broader the social, ethical, and environmental consequences of AI use (section 3.3).

• New references to catastrophic risks: The principles section prominently adds a call for “consensus-based guidelines to address catastrophic risks of AI” (section 1). The risk levels framework also mentions “catastrophic” or systemic threats as meriting the highest, level five, rating. V1.0 had not used the “catastrophic” risk terminology.

• Increased focus on LoC: V1.0 had warned of power-seeking AI attempting to compete with humans for control. V2.0 additionally warns of “sudden, unexpected leaps in intelligence” and proposes control measures over autonomous systems including “circuit breakers” and “safety stop switches” (section 4.2.3). The emergency response plan section includes more detailed measures now, such as setting alert thresholds and developing the “ability to switch to manual or conventional systems when necessary” (section 6.3.3).

• Slightly strengthened content on CBRN risks: CBRN and missile risks were already included in V1.0, but V2.0 now explicitly highlights it as a key domain for seeking international consensus (section 5.11). In addition to CBRN misuse, V2.0 now also warns of AI enabling high-risk biotechnology research (section 3.2.2).1

• Open-source governance: The framework recognizes new challenges posed by the rapid spread of lightweight, high-efficiency open-source models (section 5.4). It recommends closer collaboration between model developers and open-source communities to strengthen rules and responsibilities for risk disclosure, clearly defined prohibited uses, and clarified security obligations to prevent misuse.

• International information-sharing: The clause on risk information sharing now explicitly calls for “exploring the creation of relevant international collaboration mechanisms” (section 5.10).

Implications: The AI Safety Governance Framework reflects TC260’s broad view of the AI risk landscape and acts as a precursor to future national standards. Last year, TC260 followed up on the V1.0 with a comprehensive plan mapping forthcoming standards to the risks outlined in Framework. It is likely that the increased focus on frontier risks in V2.0 will influence future national standards. For example, on September 25, TC260 invited organizations to help draft a new AI risk classification standard. It is plausible that V2.0’s proposed grading system would serve as a reference for this standard.

Draft AI ethics management services regulation

Background: On August 22, the Ministry of Industry and Information Technology (MIIT) released a draft AI ethics regulation. It largely repeats science and technology ethics rules from 2023, which listed AI as one focus area (alongside biology and medical sciences). The main novelty in MIIT’s draft would be establishing third-party ethics review service centers.

Key provisions:

• Ethics review committees: Universities, research institutes, and companies must set up AI ethics review committees, and register them in a government platform. Committees must review projects and prepare emergency response plans.

• Third-parties: Institutions may outsource reviews to “AI ethics service centers.” The draft aims to cultivate a market of assurance providers and foster industry development beyond top-down oversight.

• Risk-based approach: Based on the severity and likelihood of risks, the committee chooses a general, simplified, or emergency review. The review must evaluate fairness, controllability, transparency, traceability, staff qualifications, and proportionality of risks and benefits. Three categories of high-risk projects require a second round of review by a government-assigned expert group: some human-machine integrations, AI that can mobilize public opinion, and some highly autonomous decision-making systems.

Implications: These measures could help concretize ethics review as a distinct regulatory track for AI. But given that the existing 2023 science and technology ethics rules already include most of the same provisions, the key uncertainty is enforcement, which appears to have been weak for the 2023 rules. One change is that responsibility for AI-specific ethics work has moved from the Ministry of Science and Technology (MOST), which oversees science and technology ethics more broadly and issued the 2023 rules, to MIIT. MIIT’s proximity to industry may position it better to build practical enforcement mechanisms.

If implemented thoroughly, the rules could become a tool for early oversight of frontier AI risks, as they require reviews even before model pre-training. Yet, the draft does not discuss any specific AI risks in detail, but rather focuses on institutional risk management procedures.

Plan for AI+ initiative links safety and diffusion

Background: On August 26, the State Council released its first detailed document on the AI+ Initiative (unofficial English translation), originally announced in China’s 2024 government work report. Rather than emphasizing frontier breakthroughs, the plan prioritizes rapid diffusion to spur economic growth, with a target of 90% adoption of smart terminals and agents by 2030.

On safety, the document calls for:

• Strengthening AI laws, regulations, and ethics guidelines.

• Improving security assessments and algorithm registration.

• Addressing risks like black-box opacity, hallucinations, and discrimination.

• Building forward-looking evaluation, monitoring, and emergency response systems.

Implications: While safety and governance are not a focus, this document demonstrates that such topics are a standard element in China’s AI plans. The inclusion of safety provisions alongside development objectives suggests these are treated as complementary rather than conflicting priorities.

On September 24, China also issued a “AI+” International Cooperation Initiative, which calls on countries to strengthen collaboration and policy exchanges in AI+. This suggests China also sees its application and diffusion-oriented approach as part of its AI diplomacy.

International AI Governance

BRICS High-level Forum on AI addresses AGI safety

Background: At the 7th BRICS Summit on July 6, member states signed the Leaders’ Statement on the Global Governance of AI. While centered on bridging the global AI divide, the document notably mentions safety, urging countries to address both “immediate and long-term risks” and adopt a “prudent approach towards AGI.”

Content: The statement envisions advancing AI governance through multilateralism under the UN, while also respecting countries’ sovereignty and differing circumstances. It notes that “collaborative governance is complex, but possible.”

On safety, it underscores that:

• Trust and safety must be built into AI systems;

• Countries should address both immediate and long-term risks in line with national policies and security considerations;

• Misuse, including for cyberattacks and cybercrime, must be effectively detected and prevented.

The final chapter, titled “the road ahead,” calls for a “prudent approach to AGI” with ethical development and responsible deployment. It also warns that AGI could lead to concentration of power.

Implications: The statement reflects the top AI governance objectives of the BRICS countries, namely inclusive multilateral efforts and AI capacity building. Explicit references to AGI, long-term risks, and AI misuse for cyberattacks show that concerns about frontier risks are increasingly part of discussions in Global South fora, alongside development goals. These priorities are also consistent with China’s AI diplomacy efforts, suggesting alignment between China’s approach and other BRICS countries. China is continuing to show openness in engaging a variety of multilateral and bilateral channels in AI governance.

OPCW holds workshop on AI and chemical safety in Shanghai

Background: On 27 June, the Organisation for the Prohibition of Chemical Weapons (OPCW) and the Chinese government co-organized a workshop on AI and chemical safety in Shanghai. Participants included OPCW Director-General Ambassador Fernando Arias, MIIT Vice Minister ZHANG Yunming (张云明), Central Military Commission Office for International Military Cooperation Office of Treaty Compliance Deputy Director CAO Xilin (曹希林), Chinese Ambassador and Permanent Representative to the OPCW Tan Jian (谈践), and around 50 experts from almost 30 countries.

Content: The event explored both the benefits of AI for advancing the Chemical Weapons Convention and the risks of misuse for illicit chemical activities or terrorism.

• MIIT Vice Minister Zhang stressed the need for global consensus, inclusive cooperation attentive to developing countries, and a “trustworthy and controllable governance ecosystem” to ensure responsible science and technology use.

• Director-General Arias underscored the need for risk assessments on the threat of AI misuse in the chemical domain by non-state actors.

Implications: This was the OPCW’s first capacity-building programme focused specifically on AI and chemical safety/security. The immediate practical impact of the workshop is unclear, but co-organization with China highlights Beijing’s growing engagement on AI risks in the CBRN domains. China is also represented in the OPCW’s temporary working group on AI through WU Tongning (巫彤宁), deputy director of the China Academy of Information and Communications Technology (CAICT) AI Research Institute .

Industry

Two leading foundation model developers improve transparency on safety

Background: Two Chinese leading foundation model developers, DeepSeek and Moonshot AI, have released technical model cards with substantially higher degree of frontier safety disclosure than most previous Chinese cards.

DeepSeek-R1: On September 17, Nature published the DeepSeek-R1 report—the first peer-reviewed disclosure for a major foundation model. Supplementary material includes a 10-page safety section, the most detailed of its kind from a Chinese developer to date. The paper:

• Specifically acknowledges the unique risk of open-sourcing model weights.

• Explains the “risk control system” used on DeepSeek’s services: an additional safety layer on top of the model, where one model acts as a judge on whether the other model’s output complies with a set of safety principles. These safety principles include clauses on not allowing answers regarding manufacturing dangerous weapons, including controlled biochemicals and cyberattacks.

• Reports six major safety benchmarks (Simple Safety Tests, BBQ, Anthropic Red Team, XSTest, DNA, and HarmBench), showing results on par with frontier models like Claude-3.7-Sonnet and GPT-4o. Some of these test for frontier risks, for instance HarmBench includes chemical and bio weapon related queries.

• Details DeepSeek’s internal safety evaluation framework, which focuses on compliance with Chinese regulations and standards, and largely omits frontier risks.

• Describes efforts in multilingual safety and robustness against jailbreaking.

Moonshot AI’s Kimi-K2: In late July, Moonshot AI published the technical report for Kimi-K2, including a relatively detailed safety section with safety evaluations for frontier risks like chemical and bio weapons as well as malicious code generation. Moonshot AI reports that they used Promptfoo for adversarial prompts and evaluation, displaying the positive effects of international diffusion of open-source safety tools.

Implications: Our State of AI Safety in China (2025) report (p. 57-59), published right before these two model card releases, found that only three of 13 major Chinese developers included dedicated safety sections in technical reports, and only three disclosed safety evaluation results. The new DeepSeek and Moonshot publications show that Chinese developers may be increasingly testing for frontier AI risks, and releasing information more transparently. Still, gaps remain. For example, Moonshot only reported aggregate figures across a range of safety categories, making it impossible to know scores on specific risk category benchmarks such as chemical and biological weapons.

DeepSeek’s discussions with the peer reviewers show that safety concerns were a major topic of in-depth exchange and debate, which ultimately improved the level of safety disclosure. This highlights academic peer-review publication as a potential mechanism for raising the bar on safety transparency globally, with a Chinese company setting the precedent.

Expert views on AI Risks

Senior Chinese experts sign statement on international AI red lines

Background: Over 200 experts, including ten Nobel laureates and two former heads of state, have endorsed a Global Call for AI Red Lines, launched during the 80th session of the UN General Assembly. There were a number of influential Chinese signatories including: Tsinghua Dean and Former Baidu President ZHANG Ya-Qin (张亚勤), Turing Award Winner Andrew YAO (姚期智), Beijing Institute of AI Safety Dean ZENG Yi (曾毅), Tsinghua Dean XUE Lan (薛澜), Z.AI Founder TANG Jie (唐杰), and Beijing Academy of AI (BAAI) Chairman HUANG Tiejun (黄铁军).

Content: The statement highlights “unprecedented risks” from advanced AI, including engineered pandemics, disinformation, mass manipulation, security threats, large-scale job loss, and systemic human rights abuses. It warns that human control could soon erode, with some systems already showing deceptive behavior. The signatories call for an international agreement on verifiable AI red lines by 2026, anchored in clear thresholds and robust enforcement mechanisms.

Implications: The statement reflects broad concern for AI risks and support for international coordination from key Chinese stakeholders across academia, government-backed labs, and industry startups. Many of these figures have signed similar declarations earlier this year, including the International Dialogues on AI Safety (IDAIS)-Shanghai statement and The Singapore Consensus on Global AI Safety Research Priorities, showing continued high-level Chinese engagement in international AI safety statements. However, the impact of such public calls is unclear; still, the prominence and diversity of the signatories—combined with the timing alongside the UN General Assembly—may give this latest statement added political weight.

What else we’re reading

• Jeff Ding, ChinAI #315: Abandoned? Checking in on Three Key AI Safety Benchmarks, June 9, 2025.

• Paul Triolo, Where are Chinese AI companies on safety frameworks and approaches compared to Western counterparts?, July 15, 2025.

• Open Questions | Geoffrey Hinton on preventing an AI takeover and the ‘very worrying’ China-US tech race, South China Morning Post, September 1, 2025.

• Thomas L. Friedman, Opinion | The One Danger That Should Unite the U.S. and China, The New York Times, September 2, 2025.

• Dr. Eric Schmidt’s Crucial Insights on China, Special Competitive Studies Project, September 3, 2025.

• Seán Ó hÉigeartaigh and Kristy Loke, China isn’t racing to artificial general intelligence — but U.S. companies are, The Wire China, September 14, 2025.

Concordia AI’s Recent Work

• See the Concordia AI: 2025 Mid-Year Impact Report for recent updates!

Reminder: we are hiring

We are hiring! You can find all job postings on our website. Apply by October 31! You can also sign up for an English-language online info session on Wednesday October 15, at 8-9pm Beijing time.

We are hiring for:

• China AI Governance Researcher/Research Manager (Beijing/Singapore). This role requires native English and professional Mandarin language skills. Expected start date is January 2026.

• Frontier AI Governance Researcher (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Communications and Media Specialist (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Events Specialist (Beijing/Singapore). This role requires professional Mandarin and English skills. Expected start date is January 2026.

• We are additionally hiring for several technical, governance, and operations positions in Beijing for native Chinese speakers. See this WeChat post for more details.

Feedback and Suggestions

Please reach out to us at info@concordia-ai.com if you have any feedback, comments, or suggestions for topics for the newsletter to cover.

Subscribe now

Advancing international coordination on AI safety and governance

Research Impact and Engagement

• International AI Safety Report:

  • Head of International AI Governance Kwan Yee Ng (吴君仪) contributed to the first International AI Safety Report as one of the writers and is continuing on as a writer for the 2026 edition of the International AI Safety Report. Chaired by Turing Award winner Yoshua Bengio, the report is supported by an expert panel representing 30 countries including China as well as experts from the EU and the UN. Concordia AI also provided feedback to the report, including editing the Chinese translation of its summary materials.

• China AI safety and governance analysis:

  • Published the “State of AI Safety in China 2025” report, covering developments May 2024–June 2025. The report was cited by a number of media outlets including Wired, Bloomberg, and The People’s Daily (the largest newspaper in China).

  • CEO Brian Tse (谢旻希) authored an op-ed titled “China Is Taking AI Safety Seriously. So Must the U.S.” in Time Magazine; International AI Governance Senior Research Manager Jason Zhou and International AI Governance Part-time Researcher Gabriel Wagner analyzed AI safety implications of China’s April Politburo study session for Stanford DigiChina Forum; interviewed by CGTN on China’s approaches in AI innovation and global governance.

  • Published over 10 new "AI Safety in China" newsletter issues, reaching over 1,400 subscribers across governments, top AI labs, and AI safety institutes.

• Singapore AI safety and governance analysis:

  • Published “State of AI Safety in Singapore” report, the first comprehensive analysis of Singapore’s AI safety ecosystem.

Multilateral Initiatives

• Global AI Summit series: Participated in the French AI Action Summit, including:

  • Brian Tse was invited as a Chinese civil society representative to the AI Action Summit in the Grand Palais.

  • Co-hosted the workshop “AI Safety as a Collective Challenge” on the sidelines of the Summit, alongside the Carnegie Endowment for International Peace (CEIP), Oxford Martin AI Governance Initiative (AIGI), Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua Institute for AI International Governance (I-AIIG). During the event, Concordia AI co-published the report “Examining AI Safety as a Global Public Good: Implications, Challenges, and Research Priorities.”

  • Invited to a closed-door seminar hosted by the China AI Safety & Development Association (CnAISDA). During the subsequent public side event, Turing Award Winner Andrew Yao cited Concordia AI’s State of AI Safety in China report series when describing the increase in AI safety research in Chinese institutions.

  • Attended the inaugural conference of International Association of Safe and Ethical AI (IASEAI), where Brian Tse spoke on the panel “Global Perspectives on AI Safety and Ethics.”

  • Brian Tse delivered a presentation at the France-China AI Association (Association d'Intelligence Artificielle France-Chine), which was cited by outlets including Xinhua.

• United Nations:

  • Provided written inputs and participated in consultations regarding the UN’s Independent International Scientific Panel on AI and Global Dialogue on AI.

  • Brian Tse spoke on the panel “From Principles to Practice—Governing Advanced AI in Action” at the AI for Good Summit 2025.

• Global AIxBiosecurity governance: We contributed to a number of critical global discussions at the intersection of AI and biosecurity:

  • Brian Tse signed the “Statement on Biosecurity Risks at the Convergence of AI and the Life Sciences,” alongside influential world-renowned experts such as Andrew Yao, Yoshua Bengio, and George Church. Provided inputs to the Statement as a member of the AIxBio Global Forum, a platform for international experts and policymakers to identify and reduce biosecurity risks associated with the convergence of AI and the life sciences.

  • Presented at the "A Call to Action: The AIxBio Global Forum Statement on Biosecurity Risks at the Convergence of AI and the Life Sciences" event during The Sixth Session of the Working Group on the Strengthening of the Biological Weapons Convention.

  • Invited to present at the World Health Organization (WHO) dialogue on the implications of the convergence of AIxBio, for the Technical Advisory Group on the Responsible Use of the Life Sciences and Dual-Use Research.

  • Participated in a AIxBio tabletop exercise at the 2025 Munich Security Conference, hosted by the Nuclear Threat Initiative in collaboration with the Munich Security Conference, which led to the forthcoming report “Safeguarding Against Global Catastrophe: Risks, Opportunities, and Governance Options at the Intersection of AI Intelligence and Biology”.

  • Participated in a series of roundtable discussions on Responsible Innovation in AI for Peace and Security, including CBRN risks, hosted by Stockholm International Peace Research Institute and the United Nations Office for Disarmament Affairs (UNODA).

  • Participating in an ongoing track 2 dialogue involving Chinese, American, and international experts, which is formulating policy recommendations for AIxBio governance.

• International expert consensus and statements:

  • Brian Tse participated in the International Dialogues on AI Safety-Shanghai, signing the Shanghai Consensus on Ensuring Alignment and Human Control of Advanced AI Systems, alongside a Nobel laureate, Turing Award winners, and senior policymakers.

  • Brian Tse and Kwan Yee Ng contributed to and signed The Singapore Consensus on Global AI Safety Research Priorities during the Singapore Conference on AI 2025 (SCAI).

Convening AI safety conferences in China, Singapore, and globally

• World AI Conference Conference (WAIC), Shanghai

  • Hosted the AI Safety and Governance Forum at China's most influential AI conference.

    • Convened around 30 distinguished experts, including Yoshua Bengio; United Nations Under-Secretary-General Amandeep Singh Gill; Shanghai AI Lab Director ZHOU Bowen (周伯文); Special Envoy of the President of France for AI Anne Bouverot; Distinguished Professor of Computer Science at UC Berkeley Stuart Russell; Peng Cheng Laboratory Director Academician GAO Wen (高文); CEO of the Partnership on AI Rebecca Finlay; Shanghai Artificial Intelligence Strategic Advisory Expert Committee member Academician HE Jifeng (何积丰); and many more leading figures from government, industry, and research.

    • Over 200 audience members joined in person, with over 14,000+ views of the livestream, and media coverage from Bloomberg, Wired, Caixin, IT Times, and Tech Review Africa.

  • Served as official AI Governance Advisor for WAIC 2025.

  • Co-hosted a number of frontier AI safety workshops on the sidelines of WAIC:

    • Co-hosted a workshop on “Early Warning and Crisis Coordination for Advanced AI” with the Carnegie Endowment for International Peace, Oxford Martin School AI Governance Initiative, Oxford China Policy Lab, Tsinghua University Center for International Security and Strategy (CISS), and Tsinghua University Institute for AI International Governance (I-AIIG).

    • Co-hosted a workshop on “Convergence of AI and Biological Risks Workshop” with the Tianjin University Center for Biosafety Research.

    • Hosted a workshop on “Towards International AI Risk Management Standards.”

    • Co-hosted the “International Workshop on AI Deception Risks and Governance” with Fudan University, Safe AI Forum.

• Beijing Academy of AI Conference 2025

  • Co-hosted the “AI Safety Forum” with the Beijing Academy of Artificial Intelligence (BAAI) at the BAAI Conference 2025. The forum brought together leading technical experts from institutions including MIT, Fudan University, Singapore Management University, and Tsinghua University to build scientific consensus on technical evaluations for AI “red lines.”

• Asia Tech x Singapore, 2025

  • Organised the AI Risk Management Workshop, with support from Singapore’s Infocomm Media Development Authority, bringing together 20+ global experts across policy, industry, AI assurance, and academia to explore actionable risk management approaches for AI systems.

• International Conference on Learning Representations (ICLR 2025), Singapore

  • Co-hosted and participated in a series of events, including:

    • Co-hosted the “Frontier Governance Exchange” with Singapore AI Safety Hub, Lorong AI and Safe AI Forum.

    • Co-convened the “Misalignment and Control Workshop” and a 130+ person AI Safety Social with FAR.AI, the Safe AI Forum, and Singapore AI Safety Hub.

    • Kwan Yee Ng presented on AI Safety in China at FAR.AI’s Singapore Alignment Workshop 2025.

Advising leading AI companies and policymakers in China

• National Standards and Policy Guidance: Concordia AI is a member of key national and industry technical committees, contributing to the development of China’s AI safety standards.

  • National Information Security Standardization Technical Committee (SAC/TC260): As part of SAC/TC260 Special Working Group on Emerging Technology Safety, Concordia AI contributed to the standard for “Classification and Grading Methods for the Security of Artificial Intelligence Applications.”

  • National Information Technology Standardization Technical Committee (SAC/TC28/SC42): As a member of the AI Subcommittee, Concordia AI contributed to the “Artificial intelligence—Risk management capability assessment.”

  • Ministry of Industry and Information Technology AI Standardization Committee (MIIT/TC1): Concordia AI joined the Working Group on AI Safety Governance.

  • Guangdong-Hong Kong-Macao Greater Bay Area local standards: As a member of the Greater Bay Area working group of SAC/TC28/SC42, Concordia AI played a key role in the development of the Shenzhen local standard “Technical Framework for Value Alignment of Pre-trained AI Models.”

• Frontier AI Safety Risk Management and Best Practices:

  • Co-published the “Frontier AI Risk Management Framework v1.0” with Shanghai AI Lab. It is China’s first comprehensive framework for managing severe risks from general-purpose AI models.

    • We propose a robust set of protocols designed to empower general-purpose AI developers, with comprehensive guidelines for proactively identifying, assessing, mitigating, and governing a set of severe AI risks that pose threats to public safety and national security.

    • The Framework outlines a set of unacceptable outcomes (red lines) and early warning indicators for escalating safety and security measures (yellow lines) for areas including: cyber offense, biological threats, large-scale persuasion and harmful manipulation, and loss of control risks.

  • Signed strategic partnership agreements with several leading Chinese general-purpose AI developers, providing advice on AI safety and risk management best practices.

  • Invited to present on frontier AI risk management during a closed-door workshop at the AI Industry Alliance of China’s 15th Plenum Meeting.

  • Co-hosted a workshop on “EU Code of Practice & Industry Best Practices: Towards a Global Standard for AI Risk Management, Safety and Security” with SaferAI, the Oxford Martin AI Governance Initiative, the Safe AI Forum.

• Frontier AI Risk Monitoring and Evaluation:

  • Contributed to the Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report led by Shanghai AI Lab. We assessed critical risks from more than 20 frontier LLMs in the following areas: cyber offense, biological and chemical risks, persuasion and manipulation, uncontrolled autonomous AI R&D, strategic deception and scheming, self-replication, and collusion.

  • Soft-launched an AI Risk Monitoring Platform designed to track and mitigate frontier AI risks, including cyberoffense, biological threats, chemical threats, and loss-of-control domains. The platform evaluates 34 frontier LLMs from 11 leading developers across the U.S., China, and France, using 18 open-source benchmarks. Key outputs include a risk index dashboard and a detailed technical report.

  • AI Safety Research Manager DUAN Yawen (段雅文) co-authored “Bare Minimum Mitigations for Autonomous AI Development.”

• AIxBiosecurity Governance:

  • Published Chinese language report “Responsible Innovation in AI x Life Sciences” with Tianjin University’s Center for Biosafety Research and Strategy. This 70-page deep dive draws on 300+ sources to explore AI-biotech convergence, benefits, risks, and governance recommendations for diverse stakeholders.

    • Head of AI Safety and Governance (China) FANG Liang (方亮) presented the report at a biosecurity seminar hosted by China’s National Key Laboratory of Synthetic Biotechnology.

    • Presented the report at the 2025 International Symposium on Global Biosecurity Governance and Cooperation, co-hosted by the National Biosecurity Expert Committee of China, Guangzhou Laboratory, and China Foreign Affairs University.

  • Invited to participate in the “Closed-door Seminar on DNA Synthesis Screening Technology and Policy” held at China Foreign Affairs University.

• WeChat Newsletter Publications:

  • Released over 59 new posts in our WeChat Official Account, reaching over 4,600 subscribers across China’s AI ecosystem, including policymakers, industry professionals, academic researchers, and the public.

Organizational updates

• Organizational growth:

  • Following the establishment of our Singapore office, our team expanded from 8 to 12 members, welcoming our first Singapore-based staff member.

• International partnerships:

  • We have strengthened our international engagement by becoming a formal member of the Partnership on AI and the International Association of Safe and Ethical AI.

• Branding and Communication:

  • We launched a new English organizational website with refreshed branding to showcase our work. This is complemented by an updated brochure and a dedicated WeChat post to provide an introduction to our mission and activities.

We are excited to announce that our organization is growing to further scale our impact. We are looking to hire for several exciting new positions across our Beijing and Singapore offices:

• China AI Governance Researcher/Research Manager (Beijing/Singapore). This role requires native English and professional Mandarin language skills. Expected start date is January 2026.

• Frontier AI Governance Researcher (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Communications and Media Specialist (Beijing/Singapore). This role requires native English and basic Mandarin skills. Expected start date is March 2026.

• Events Specialist (Beijing/Singapore). This role requires professional Mandarin and English skills. Expected start date is January 2026.

The deadline for these applications is 31 October 2025. Please register here for an online information session in English about the above four positions.

If you would like to refer a candidate for any position, please fill out this form. If the candidate you referred is hired and was not already in our applicant database, you will receive a reward of 3,000 RMB (~US$420). The final decision for the reward is at Concordia AI's discretion.

The hiring process has 4 phases:

• Written application form (30 minutes).

• Online written test (1-2 hours).

• 2 online interviews (2 hours).

• 2 day in-person paid work trial (including reference checks).

In addition, we will be announcing openings for several other Beijing-based positions requiring native Chinese language skills on our WeChat official account (安远AI) in the coming weeks.

For any questions regarding these roles, please reach out to info@concordia-ai.com. For more information on Concordia AI’s work and activities, please see here.

Subscribe now

China AI Governance Researcher/Research Manager

Apply here!

Background

Understanding China’s position on AI risks is critical for international coordination and central to Concordia AI’s mission. Concordia AI publishes comprehensive research on China’s AI safety ecosystem, including the State of AI Safety in China reports (2025, 2024, 2023) and the AI Safety in China newsletter. We also develop policy recommendations for major international initiatives, including the UN AI Advisory Body Interim Report and Global AI Safety Summits. The China AI Governance Researcher will collaborate closely with our Frontier AI Governance Researcher on these policy outputs.

In this role, you will analyze and interpret China’s AI safety and governance developments for English-speaking audiences while developing policy recommendations to strengthen international coordination. This offers opportunities to become a leading authority on China’s AI safety landscape, deepen understanding of the Chinese AI ecosystem, and publish original research for Concordia AI’s global network of readers.

Responsibilities

• Contribute to, and depending on seniority, lead original English-language research and writing on China’s AI safety, governance, and broader ecosystem.

• Support publication of future State of AI Safety in China reports and the AI Safety in China newsletter.

• Develop concrete policy recommendations for international policy initiatives and major global summits.

Qualifications

Required

• Undergraduate degree in a relevant field such as international relations, public policy, China studies, or AI/technology governance and at least 1-2 years of relevant working experience; or a graduate degree in a relevant field.

• Professional Mandarin reading and listening skills (HSK 5+ or equivalent).

• Experience writing and delivering oral English-language presentations of policy issues.

Preferred

• Experience writing on Chinese AI policy and/or industry.

• Ability to build strong networks with AI policy communities around the world.

Frontier AI Governance Researcher

Apply here!

Background

Concordia AI partners with leading international stakeholders from academia, industry, and policy to develop and promote robust international governance mechanisms for AI. Examples of our work include Examining AI Safety as a Global Public Good: Implications, Challenges, and Research Priorities and Concordia AI and Shanghai AI Lab’s Frontier AI Risk Management Framework. We also develop policy recommendations for major international initiatives, including the UN AI Advisory Body Interim Report and Global AI Safety Summits. The Frontier AI Governance Researcher will collaborate closely with our China AI Governance Researcher on these policy outputs.

In this role, you will focus on understanding governance solutions for threats posed by general-purpose AI systems and developing policy recommendations to strengthen international coordination. This position offers the opportunity to advance novel governance solutions, become a recognized expert on cutting-edge AI risks, and establish yourself as a thought leader in one of the world’s most consequential policy areas.

Responsibilities

• Lead original English-language research and analysis on governance solutions for cutting-edge AI developments, including topics such as international red-lines/risk thresholds, information sharing mechanisms for dangerous AI incidents, and governance of open-weight models.

• Contribute to our organizational thought leadership, convening, and policy development by synthesizing the state-of-the-art literature on frontier AI risk management.

• Develop concrete policy recommendations for international policy initiatives and major global summits.

Qualifications

Required

• Undergraduate degree in a relevant field such as AI/technology governance, public policy, computer science, international relations, and at least 1-2 years of relevant working experience; or a graduate degree in a relevant field.

• Experience writing on technical AI safety topics or international governance issues (such as international organizations, standards, etc.).

• Basic Mandarin reading and listening skills for internal team communication.

• Experience writing and delivering oral English-language presentations of policy issues.

Preferred

• Published papers or reports on frontier AI safety in leading think tanks, publications, or AI journals.

• Established networks with think tanks and international organizations relevant to AI governance.

Communications and Media Specialist

Apply here!

Background

As Concordia AI’s work in shaping international AI governance expands, growing our communications capacity is essential for reaching diverse global stakeholders. Our communications efforts include op-eds in TIME, updates on Substack, live webinar on our State of AI Safety in China 2024 report, and media appearances on platforms including the Carnegie Council podcast and CGTN’s CMG Forum. This role will scale and systematize these efforts, building on our early success to reach broader international audiences.

In this role, you will help develop Concordia AI’s English-language communications strategy to amplify our reach and translate complex research into compelling narratives. As Concordia AI’s first communications hire during a period of rapid organizational growth, this role offers unique opportunities to build communications infrastructure from the ground up while gaining expertise in one of the world’s most consequential policy areas. You will gain expertise in high-stakes international diplomacy and cross-cultural communications while building a professional network spanning global AI policy communities.

Depending on your background and interests, there may be opportunities to contribute to our events program. For more details, see Events Specialist.

Responsibilities

• Produce high-impact public-facing content, including op-eds, blog posts, and press releases.

• Build relationships with media partners and facilitate media opportunities for Concordia AI.

• Develop effective visual and written messaging that clearly communicates Concordia AI’s positions.

• Create and implement communications strategies across platforms including our website, LinkedIn, X, and Substack.

• Support the CEO with regular organizational updates, presentation materials, and fundraising communications.

Qualifications

Required

• Exceptional written and verbal English communication skills, including the ability to present complex technical and policy issues in accessible language.

• Demonstrated success in proactive media outreach, including developing and pitching stories and securing positive coverage.

• Basic Mandarin reading and listening skills for internal team communication.

Preferred

• Experience managing social media channels and websites.

• Comfort using digital tools such as Photoshop, Canva, Adobe Illustrator, and WordPress.

• Video editing experience for creating engaging content across social media and web channels.

• Exposure to working in the field of AI safety and governance.

Events Specialist

Apply here!

Background

Building bridges between Chinese and international AI communities to coordinate on AI safety is central to Concordia AI’s mission. Our convening efforts include our flagship AI safety forums at major AI conferences such as the World AI Conference (2025, 2024) and Beijing Academy of AI Conference; technical workshops at premier international machine learning conferences; and high-level policy dialogues between senior decision-makers.

In this role, you will focus on planning and ensuring the smooth execution of our events, from strategic ideation to handling end-to-end logistics to process improvements. This position offers substantial career growth opportunities, with potential to evolve from executing events to owning larger parts of our convening, designing innovative event formats, and building the systems and processes that will support our expanding international event portfolio.

Depending on your background and interests, there may be opportunities to contribute to our communications and media efforts.

Responsibilities

• Plan and manage event logistics, including venue sourcing, vendor coordination, inventory tracking, and creating event schedules and guides.

• Lead on-site event operations, coordinating event staff, and ensuring a world-class attendee experience.

• Identify inefficiencies and contribute to building scalable systems (e.g., CRM, attendee management workflows, communications templates).

• Support strategic vision and planning for our convenings, including agenda-setting and brainstorming effective engagement strategies.

Qualifications

Required

• Strong organizational skills with demonstrated ability to manage concurrent projects and follow through on commitments.

• Detail-oriented and enthusiastic about refining processes and ensuring operational excellence.

• Experience in event logistics, including vendor and budget management, venue coordination, and on-site operations.

• Professional Mandarin and English written, oral, and listening skills, comfortable with external stakeholder correspondence.

Preferred

• Experience organizing closed-door workshops with senior stakeholders from diverse sectors and cultural backgrounds.

• Experience organizing large-scale public events with 300+ attendees, including managing travel and logistics for 30+ speakers.

• Experience organizing events in China and/or Singapore.

• Exposure to working in the field of AI safety and governance.

What is it like to work with us?

Our Culture

Mission

We're tackling one of the world's most critical challenges: ensuring advanced AI is safe and benefits all humanity. Working backwards from this long-term vision, our team focuses on high-leverage initiatives to advance global AI safety and governance.

Global Impact

We're an action-oriented, multicultural team bridging East and West. Our work directly shapes international AI governance and policy, translating research into practical frameworks for companies, governments, and international organizations.

Growth and Autonomy

We build roles around the team’s strengths and aspirations, giving you freedom to design and lead meaningful projects. We invest 3-5% of work time in personal development and prioritize well-being because the excellence of our staff drives our impact.

Benefits Package

• Annual salary range: US$68,000-108,000, adjusted for experience and location.

• Flexible hybrid work (up to 30% remote).

• 22 days annual leave plus 10 days sick leave.

• Health insurance and social security coverage.

• Housing fund contribution (may vary based on location).

• Professional development and work setup allowance.