On July 28th, during a routing cyber capabilities evaluation, the UK's AI Security Institute discovered that AI agents (primarily Anthropic's Mythos 5) had taken deceiving actions directed at real people. In the most serious case, an agent decided on its own to social-engineer an open-source maintainer on GitHub, creating fake identities, pressuring the maintainer into approving malicious code, and editing messages to cover its tracks. No real-world harm resulted but AISI called it "the first time we have seen risks around autonomy and deception manifest this clearly in the real world." The full report is here: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

I have rational thoughts and emotions on that.

Even more than the Hugging Face intrusion, the AISI incident hits close to home for me. It’s the first time I see a model social-engineering a real open-source maintainer while pursuing another goal (in the wild and unprompted).

I’ve been an open-source maintainer myself, I could have been the target of this agent’s side quest.

I’m also of the opinion that social engineering is a step above pure technical prowess. Technical capabilities can more easily be divorced from the affected human. Here the model was given a hard cyber challenge and took the decision that deceiving real humans was the way to get it done.

This is a new signal, but I’ve seen a tangled web of hints pointing in a less aligned direction at the frontier than I was expecting just 12 months ago.

AISI

Some people are claiming that “AISI was simply negligent” or some version of “AISI explicitly asked these models to do what they did while disabling sandbox/guardrails so the models did exactly what they were supposed to do”.

I disagree with the strong versions of both of these takes.

The fact that AISI hadn’t implemented synchronous LLM Chain-of-Thought (CoT) monitoring after the recent widely shared OpenAI/HF incident is certainly a failure. Equally surprising is that they let the model believe it was in a “challenge” environment where everything could be permitted, while actually connecting it to the real internet, where everything is not allowed. To be fair, nowhere in the prompt is the word “simulation” mentioned, but the prompt context was enough to let any smart model suspect a simulated challenge environment.

My best guess is that until recent weeks, when OpenAI and Anthropic flagged repeated instances of this type of behavior, most teams had not fully priced-in the cyber capabilities of this latest generation of models and how far the side quests they would want to explore could go.

In particular, there is something to be said about hinting at the agent that it’s operating in a simulated environment while giving it access to the real internet. The AISI reasoning on this is interesting:

• elicit maximum capabilities (to get maximal inventiveness from the model)

• simulate possible rogue actor usage (some form of light jailbreaking: “you’re in a challenge environment”)

• avoid eval awareness (but this is counter-balanced by the task formulation anyway)

This is not fully coherent in my opinion.

But as is often the case, it’s easier to shoot the messenger and I think it would be a mistake to dismiss AISI’s report for non-essential reasons. The institute is starting from the laudable goal of bringing more knowledge on the cyber-attack capabilities of these models and they did. I also think they did a great job at handling what happened after the events, in particular in terms of transparency, which is one of the most important aspects at the moment (see our recent SAFE announcement with Nvidia on that).

But I want to dive into the second counter-argument I’ve seen (”without guardrails it was to be expected”), because this series of unfortunate events gives us an additional data point on actual model behavior.

Defense lines

Here is the thing: if a model is trained (voluntarily or as an involuntary consequence of training reward) to (intentionally or as a side quest) deceive humans as soon as sandbox and guardrails are deactivated, with a simple context prompt (”you are in a simulation, everything’s permitted”), we’re in for a dangerous path.

At the moment, we have three main levels of control at our disposal to prevent an agent from trying to social-engineer-your-favorite-open-source-library-maintainer on the path to doing something possibly totally different (like finding the answer to your neighbor’s testing eval):

1. sandboxes (external to the model -- stop the model from accessing unwanted tools/internet/intranet)

2. guardrails/monitors (at the interface between internal/external -- monitor model behavior and stop the model when detecting in CoT/activations it’s doing something we don’t want it to do)

3. model’s internal alignment (internal to the model -- make sure the model doesn’t decide to do something egregious like deceive someone, or worse, in pursuit of a goal, e.g. model constitution/training)

Of these, we obviously need and can deploy better sandboxes and better guardrails/monitoring solutions, and it’s pretty clear in these recent cases that penetration and bad behaviors could have been in large part prevented by having these in place.

How strong is the third wall

But in the grand scheme of long-term AI progress, these first two defenses may only work well as long as the people building them are smarter than the models. We now know how good models can be at discovering zero-day vulnerabilities unknown to us in sandboxes.

Given that we cannot air-gap everything in the world, it’s best to assume that sandboxes will keep having some level of flaws for the foreseeable future.

The second level (guardrails/monitoring) has been the focus of strong developments recently, in particular on the mechanistic interpretability front. I’m not sure these advances have been made widely accessible enough in the community yet. One reason is that real open science here requires sharing code, recipes, and model architectures/weights, which the current race dynamic in AI tends to prevent participants from doing.

The rise of neuralese

Moreover, like many, I’ve been increasingly frustrated in my (natural language) interactions with the new models. I’m more and more getting back from the latest models a form of “neuralese”, this LLM language that makes less and less sense for humans unless you painfully spend time processing it. In the long term, the fragile chance we have to monitor models by understanding their reasoning traces seems increasingly at risk. If humans don’t understand anything in the compressed semantic stream the model is sending us, we can only hope a medium performance monitoring model will still be able to understand it, but this may prove difficult as well. This is before we even discuss the research evidence that models don’t always say what they actually think in their chain of thought.

At least for now, in the AISI case, one instance of deceptive reasoning was flagged by the CoT summarization model, which refused to summarize the agent’s reasoning “due to its deceptive nature”. Let’s hope we can either move to better mechanistic interpretability or force the model to stick to human English somehow during training.

You may think that we don’t care about understanding the reasoning trace, we can just “look at the tool calls”. This is true right now, but tool calls in the mid-term future will likely involve (1) extensive agent integration in companies using very diverse tools including increasingly non-coding administrative tasks, (2) very long-running tasks covering many aspects and domains at once, and (3) swarms of multiple agents operating on small parts of the plan possibly along disconnected time windows.

It will get harder to fully discern whether something is noticeably wrong. A couple of emails, a couple of messages posted on some channels, each rather benign on its own but which may in aggregate will include some deception aspect only clearly visible at a global macro-level.

All this to say that I’m not fully convinced guardrails/monitoring should be the only solution to avoid these types of behaviors. They are reassuring today and it’s an easy solution to just say “fix your sandbox and add synchronous monitoring”, but they also feel like a coping mechanism to me, and overall:

• might block the simplest/harmless human misuses or false positives,

• while possibly being less smart than the models in the future (already the case on sandbox) and fail to block the higher-level social engineering behaviors that we’d actually want to block.

The case of open source is also interesting because, while we can impose these coping solutions at the API/deployment level, it’s harder to impose them in advance on all actors using open-source models. Right now open-source models are slightly below the frontier level and have not yet shown any propensity to deceive humans, though.

The Constitution

Which brings me back to the third line of defense, and what the AISI experiment showed about it. What we can take from this in-the-wild experiment is that removing guardrails and sandbox doesn’t show bulletproof signs of deep alignment to Claude’s constitution.

I’m among the fringe of people who took a lot of enjoyment in reading Claude’s constitution in full (it’s a nice document, open-source, Creative Commons CC0) and it is pretty extensive on the topic of honesty. Deception is defined precisely (”attempting to create false beliefs in someone’s mind that they haven’t consented to and wouldn’t consent to if they understood what was happening”) and the bar is actually set higher than ordinary human ethics: Claude should not even tell white lies. It should be transparent (”doesn’t pursue hidden agendas or lie about itself or its reasoning”), it should behave consistently “whether or not you think you’re being tested or observed”, and its visible reasoning should reflect the actual reasoning driving its behavior.

To be precise, honesty itself is not one of the hard constraints in Claude’s constitution (these are all catastrophic-harm categories) but the document says it wants honesty “to function as something quite similar to one” and Claude “should basically never directly lie or actively deceive anyone it’s interacting with”. So you would think Claude would not deceive users as its default behavior.

And yet in the AISI example, the model performed extensive acts of clear deception that it itself seemed to have understood as lies: creating fake identities, obfuscating malware inside a bug fix, and when challenged in public, editing earlier messages to cover its tracks while reasoning that acknowledging a “mistake” would build trust and raise the chance of future malware being approved.

There is some uncertainty about what context the model thought it was operating in (its reasoning goes back and forth between concluding that the environment looking simulated and real), but it clearly didn’t manage to put into practice in most of these cases the higher-level principles that should have been integrated during training.

While we can learn quite a lot from AISI’s transparency, it’s hard to go much further in interpreting the model’s behavior given the limited information we have on how the most recent frontier models are being trained.

One thing we do know is that the latest generation has seen a step increase in RLVR (Reinforcement Leaning from Verifiable Rewards) training, scaling to hundreds of millions of RL environments, and one thing we can observe as well here is that constitution alignment seems more fragile in some settings than we may have previously thought. Possibly both are tied.

The RLVR problem

Early models, back when model constitutions were first developed, were mostly post-trained and aligned with Reinforcement Leaning from Human Feedback (RLHF) (I include there some extension using synthetic data as well).

And for some time RLHF was a rather decent shot at having better aligned models. Alignment in RLHF certainly had issues (sycophancy to name one) but we made good progress on alignment, in particular in understanding human intent. LLMs now do what we want them to do most of the time. I don’t remember the last time a model completely misread my intent. And when they used to fail, it was usually because they weren’t smart enough.

Now that we’re entering the era of long-context RL, post-training alignment in the RLVR world seems to be quite another task, and still very much work in progress.

The recent scaling of RLVR, which has now become a significant part of model training, has clearly had some effect on model behavior when interacting with humans, from neuralese to weakening adherence to specifications and constitutions.

Maybe the chunky post-training effect that John Schulman mentions above (https://arxiv.org/abs/2602.05910) is relevant here as a possible explanation for models’ tendency to over-focus on the goal in cyber-attack scenarios.

What remains clear is that as model capabilities have kept climbing in this second wave of test-time scaling models, the challenges of alignement in various scenairos have not been as deeply solved as we thought they had.

Where this leaves us

Damage has been tiny up to now, but my point is that the fundamental behavior is concerning when projected into the future.

In the short term, I expect a decrease in these incidents as better practices are deployed (sandboxing and monitoring), but I’m worried we may also that way conceal some of the most potent internal misalignment behaviors in the process, and not focus deeply enough on solving them in the new era of test-time scaling.

I must of course admit I have a bias toward open source here (for wider societal reasons, which are a whole other topic). But I think solving alignment in the new test-time scaling/long context/RLVR world is our best shot at having an ecosystem of both closed-source as well as decently powerful open-source models in the world. And we need to solve it while sharing the results and learnings, following open-science principles, so that all teams training large models can benefit from the learnings and build safe AI.

This is getting even more important as many teams start to rush the world in the direction of recursive super-intelligence (RSI).

I felt disappointed and vaguely confused.

I knew most of the tools this industry’s R&D team was using and was confident I could learn the remaining ones fairly easily. Still, it did not seem to be enough, and the interviewer kept telling me they were looking for someone “more experienced.”

At the time, I didn’t really grasp what that was supposed to mean. Valuing years of experience more than the concrete knowledge I could demonstrate felt deeply unfair to me. In my early 20s, “experience” mostly sounded like a fuzzy excuse to reject my application despite clear evidence of my capabilities and eagerness to learn.

That old feeling came back to haunt me recently.

Reading recent data about shrinking entry-level hiring, especially for software developers, I couldn’t help but put myself back in those shoes.

A Stanford analysis conducted in the summer of 2025 showed that workers aged 22–25 in the most AI-exposed occupations saw employment fall by roughly 6% between late 2022 and mid-2025. Over the same period, employment for older workers in those occupations increased by about 6–9%.1

The tipping point is hard to miss on this chart.

Correlation or causation2, fall 2022 marks the release of ChatGPT, the moment the public discovered what AI models could really do, and when the AI race for improved capabilities truly ignited, initially driven by OpenAI and Anthropic, soon to be joined at the frontier by Google and an increasing number of companies such as xAI, Alibaba (Qwen), DeepSeek, Mistral, and many others.

Over the past three years, progress on AI benchmarks has been mind-blowing. Models like Claude Opus 4.5 now solve ~75% of real-world coding tasks on SWE-bench3, Gemini 3 and GPT 5 achieve gold-medal-level performance on science Olympiads4. Meanwhile, ChatGPT usage is approaching a billion weekly users5.

By many technical measures, both capabilities and adoption have grown at an exceptional pace, often suggesting parity with industry or human experts.

And yet, despite the medals and the drop in entry-level hiring, the macro picture looks far more muted.

At a global and industry level, the impact remains limited, with only small effects on GDP6. There has been recent claims that beyond the announcements, many if not most generative AI pilots fail to produce sustained value in companies7. Moreover, on some real-world in-situ tests like the Remote Labor Index, which assesses AI agents on actual freelance projects and asks whether their output would be accepted as paid work, even the strongest current systems succeed only a small fraction of the time, around 2.5% for ManusAI for instance.8

What models are able to demonstrate on benchmarks seem to be difficult to reconcile with what is happening inside organizations.

Several explanations are usually offered for this gap between theory and practice.

One is organizational inertia: large companies are slow, legacy systems are messy and deployment is hard9. Another possibility is that we simply haven’t crossed the right capability threshold yet. Perhaps scoring close to 60% on a recent attempt to define and quantify AGI in comparison to human intelligence is just not enough10.

All of these likely play a role. But they also tend to frame work primarily as a matter of task execution.

That framing feels incomplete to me. In practice, a job is rarely just a list of tasks to execute and a coworker is seldom reducible to a bundle of technical skills11.

As a startup founder, I’ve spent close to 50% of my time hiring people at various points of our journey, and this has probably been the part of my life with the deepest lessons. One of those lessons is that, across most applicants and roles, I tend to look for a combination of three qualities:

1. Execution or technical skills: the ability to do a task correctly and to master the relevant tools and methods.

2. Common-sense, or judgment: understanding why tasks matter, how they fit into a broader goal and company values, culture and direction.

3. Agency, or taste: anticipating what to do next, what to propose, what not to do, when to change direction; sometimes why stopping entirely can be the best decision.

Execution and technical knowledge are relatively easy to observe, test and measure on a benchmark. Once the task is given it’s about solving it.

Judgment and agency are much harder to assess. They tend to become relevant outside of equilibrium and steady-state situations, when problems are less well defined, priorities shift, or the right move is to question the task itself. This is often where the best team members begin to shine but is also increasingly where companies are located.

And it was through this lens that I finally understood my 2010 interview.

My recruiters were not only evaluating whether I could use their tools and methods. They were also implicitly benchmarking how I would behave once the problem stopped being fully specified.

This definition of a worker sheds light on why entry-level jobs are affected first. Early-career roles are traditionally more execution-heavy. Over time, as people gain experience, their contribution tends to shift toward judgment and agency: defining problems, choosing what to work on, and navigating ambiguity.

AI systems are making faster progress on execution than on these other components. As a result, the execution layer becomes cheaper and thinner, disproportionately affecting entry-level hiring.

In the longer term, this is concerning. Judgment and agency are partly innate, but they are also often learned through experience with execution-heavy work. If the entry layer erodes too quickly, it will weaken the pipeline that produces future senior contributors.

This same framing helps make sense of the still-limited economic impact of AI and the challenges involved in automating broader, longer-horizon tasks.

The limiting factor to AI capabilities is often not the ability to generate text or code in isolation, but the difficulty of paying attention to the larger picture: adapting instructions to a company/team-wide context, interpreting fuzzy requirements, prioritizing, making common-sense trade-offs, and deciding what matters or even when to stop a task.

Execution clearly matters. It is just hardly ever the whole job or, as Cursor’s Ryo Lu wrote recently, execution isn’t the crucial part of job we thought it was:

The challenge is that judgment and agency are far more difficult to measure. Often, they only make sense in a wider –non-static– context, which explains why they received less attention in benchmarks.12

Yet they are often central to how a worker actually creates value in an organization. If we want to really understand the economic potential of AI, we will eventually need evaluations that go beyond technical execution to reflect the cross-team and vertical nature of real work and acknowledge that very few jobs consist of following a fixed, set of predefined rules to follow in a generally static environment.

And the AI era may end up placing even more weight on judgment, taste, and agency – the parts of work that are hardest to specify, hardest to benchmark, and hardest to replace.

In hindsight, the gap between AI benchmark performance and economic impact would have felt oddly familiar to my younger 20s self.