Part 4 proved the loop works on something real: a rate-limiting feature walked through all six stages, with the agent doing most of the typing and almost none of the deciding. That’s the version of the series everyone wants to believe describes their own team permanently. It doesn’t. It describes a team on its best week.

Part 5, the final part of this series, is about what happens on the other weeks. Two failure modes show up in almost every team that adopts an AI-native loop and then relaxes: responsibility drift, where careful review quietly turns into rubber-stamping, and escalation theatre, where a polished-looking policy routes to nobody in particular. Both are correctable, and both get expensive if you let them sit.

From there, we bring everything from this series together into the one artifact that’s been promised since Part 1: a Human-AI Responsibility Matrix, with one named Accountable owner for every stage of the loop, no exceptions. It’s the document that turns “you own the code” from a rule Michelle repeats in keynotes into something you can check into your own repository this week.

Michelle Sandford, Microsoft’s Developer Engagement Lead for Asia

Let’s dive in.

Failure mode and correction: When the boundaries blur

Even with the matrix in place, two failure modes show up early and often. Both are correctable; both get expensive if you let them set.

The first is responsibility drift. The agent does good work for two weeks, the team relaxes, the senior engineer starts approving spec drafts without reading them carefully, and someone notices three sprints later that the agent has been quietly proposing AGENTS.md changes that no one has been reviewing as policy changes. The fix is mechanical, not cultural. Mark AGENTS.md, the specs/ directory, and the contents of .github/ as protected paths in CODEOWNERS so that a named human must approve every change. Add a periodic review (quarterly is plenty) where the team reads the AGENTS.md diff history end to end. Drift is the default; mechanical countermeasures are how you resist the default.

The second is escalation theater. The team writes a beautiful escalation policy, but the routes point to a generic senior engineers’ group that nobody owns, so escalations sit unread until they time out and auto-merge. The fix is to name humans, not roles, and to make the absence of an escalation visible. A stale escalation should turn a PR red in the status check, not silently expire. The 2025 DORA report’s trust paradox finding maps directly onto this: developers say they do not fully trust AI output, then merge it anyway when the friction of escalation is higher than the friction of approving. Lower the friction of the right behavior, not the wrong one.

I once found a team with a polished escalation policy but no named weekend route. Two high-risk PRs sat untouched, then were merged quickly on Monday under schedule pressure. Nothing failed immediately, which made the weakness harder to see. They corrected it by naming explicit primary and backup reviewers per service and by failing CI when an escalation label aged beyond a threshold. The policy became operational overnight because drift was now visible and mechanically blocked.

A third pattern is worth a sentence even though it is less common. Over-escalation (routing every borderline case to a human) produces the same outcome as no escalation at all, because reviewers stop reading carefully when everything is urgent. Tune the triggers accordingly. A good escalation rule fires on roughly 5 to 15 percent of agent PRs, often enough to stay sharp, rarely enough to stay meaningful. If yours fires on every PR, the rule is doing the wrong work.

Now, let’s look at how we can formalize the practical processes we need by building out the human-AI responsibility matrix in the next section.

Building a Human-AI Responsibility Matrix

The carry-forward artifact for this series is a Human-AI Responsibility Matrix. Rows are the six loop stages covered in Part 1, columns are the participants in your loop, and cells use RACI-style markers:

• R (Responsible: does the work)

• A (Accountable: single named owner of the outcome)

• C (Consulted: must be heard)

• I (Informed: must be told)

Every row must have exactly one A. That is the rule that makes the matrix worth keeping.

A blank template, suitable for copying into docs/responsibility-matrix.md:

Figure 1: Human-AI Responsibility Matrix

A filled-in version for the rate-limiting example, which is what the team would actually check in:

Figure 2: Filled-in Human-AI Responsibility Matrix with RACI assignments and escalation rules

Notice the shape. The Agent is Accountable at exactly one stage (Change), and is Responsible at one other (part of Verification, as an evaluator’s input). Everywhere else the agent is Consulted or Informed. That distribution is what human-led, agent-assisted looks like written down. Other shapes are valid for other teams, but the rule (one A per row, never the agent A at Intent or Delivery) holds across every team this pattern has worked for.

Treat the matrix as a living document. It belongs in the repository, it changes when AGENTS.md changes, and it is reviewed in the same cadence as the AI-Native Success Criteria Memo from Part 2. The two artifacts are a pair: the memo says what good looks like, the matrix says who is on the hook for getting there.

The first matrix review I ran with a team surfaced a hidden disagreement: product believed low-risk API changes could be self-approved if tests were green, while engineering required explicit human accountability for all contract-impacting changes. Writing the matrix forced that disagreement into one row and one cell, which made it solvable. They split change categories and documented which categories required named approval. Future PRs moved faster because decisions no longer depended on competing assumptions.

Wrapping up…

Five parts ago, this series opened on a plane, with an agent that had quietly invented a case study because it thought the story would land better that way. Nothing about that moment was malicious. What was missing was a system where a human was still positioned to catch it.

Everything since has been about building that system, one piece at a time. Part 1 gave you the six-stage loop and an honest read on where your team actually sits. Part 2 proved the loop changes outcomes, not just vibes, with a bug fixed twice and only one fix that actually held up. Part 3 named the real shift: developers move from typing to deciding, and Rule Zero, you own the code, became the line the rest of the series was built to operationalize. Part 4 turned “ask if you’re not sure” into rules a machine can actually enforce, and proved the whole loop on one real feature. Part 5 closed the gap between “we have a policy” and “the policy actually works,” and handed you the Human-AI Responsibility Matrix, the document where every one of those ideas has to become concrete: one stage, one owner, no exceptions.

If you’ve built even one artifact from this series, a spec, a memo, an AGENTS.md rule, a matrix, you already have more than most teams claiming to be AI-native. The rest is just repetition.

Thank you for reading this series. If it changed how you think about your own delivery loop, that’s worth more than any part of this newsletter could say on its own.

Further reading

🔵 Responsibility assignment matrix (RACI)
The project-management technique underneath the Human-AI Responsibility Matrix built in this issue. Useful if you want the fuller history of RACI and its variants before adapting it to a loop that includes a non-human participant.

🔵 Postmortem culture: learning from failure, Google SRE Book
A useful companion to this series’ argument that responsibility drift and escalation theatre are systems problems, not people problems. Google’s SRE team makes the same case about incident reviews: the fix is redesigning the system, not assigning blame after it fails.

🔵 Microsoft’s framework for building AI systems responsibly
Microsoft’s own account of why it built a company-wide Responsible AI Standard. Worth reading alongside this series’ argument that trust in AI output should be placed in a designed system, specs, gates, named approvers, rather than in the model itself.

🔵 2025 DORA State of AI-assisted Software Development report
Referenced a final time because this issue closes the loop on the trust paradox that’s run through the whole series: developers use AI more while trusting it less, and this issue shows exactly how that gap shows up as escalation friction in practice, not just as a survey statistic.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Editor-in-Chief
Build with AI

Hands-On Agentic Frontends with AG-UI and CopilotKit

A chat box is not an agent interface. A production one streams work as it happens, keeps frontend and agent state in sync, hands tasks to specialist agents, and stops for a human before it does anything expensive.

The awkward part is that five protocols now claim a piece of that job: AG-UI, A2A, MCP, MCP Apps and A2UI. Most explanations tell you what each one is. This session shows you where each boundary sits, by building one incident-response app that uses all of them and adding the unglamorous bits afterwards: cancellation, timeouts, partial failures, and an approval gate before remediation.

Rainer Hahnekamp and Murat Sari co-maintain CopilotKit’s Angular integration, so these are decisions they have already had to make in public.

Wed 16 Sep, 9:00 to 12:00 EDT. Three hours, online. Angular or React, your choice.

👉 Book on Eventbrite · or via Luma

Hands-on AI Assisted Java and Spring Boot Development

AI writes Java quickly. It also writes Java that passes review and quietly wrecks your layering.

This one runs a whole Spring Boot build with AI in the loop: scaffolding, entities, DTOs, controllers, persistence, tests, debugging, refactoring. The point is not speed. It is keeping the architecture and security calls yours while the typing stops being your problem.

Laurenţiu Spilcă is a Java Champion and wrote Spring Security in Action, which is to say he will spot the generated code that is wrong in a way that still compiles.

Thu 24 Sep, 8:30 to 11:00 EDT. Two and a half hours, online. Bring working Java and Spring Boot.

👉 Book on Eventbrite · or via Luma

Hands-On Graph Engineering with Claude Code

In July we ran an issue on why an agent asked to grade its own work tends to just praise it. This is the workshop where you build the fix.

Single-agent loops fail in predictable ways. Context degrades. The agent marks its own homework. Everything runs in sequence when half of it could run in parallel. The spec drifts without anyone noticing, and retries run until something stops them. None of that is solved by a better prompt. It is solved by a graph: real node boundaries, structured handoffs, and verification the agent doing the work cannot influence.

Ken Huang co-chairs the OWASP AIVSS project and the AI Safety working groups at the Cloud Security Alliance. He opens with a keynote, then it is hands on the whole way.

Tue 29 Sep, 8:30 to 11:30 EDT. Three hours, online. Assumes you already use Claude Code.

👉 Book on Eventbrite · or via Luma

Every one includes the recording, the slides and a certificate, so a clashing calendar is not a reason to skip.

If you would rather not keep checking this page, new dates go out with the Thursday issue. Subscribe to Build with AI.

Part 3 left you with something genuinely useful and slightly uncomfortable: a boundary a reviewer can apply without a meeting, mapped stage by stage across the loop. What it didn’t answer is what happens the moment the normal case breaks. The agent hits something it wasn’t expecting. A diff comes back three times the size anyone planned for. A change quietly touches a path it shouldn’t have.

In an AI-native loop, ambiguity doesn’t default to caution; it defaults to whichever voice is loudest, and that’s usually the agent’s own confidence, which is famously unreliable as a signal.

Part 4 replaces “ask if you’re not sure” with something a machine can actually enforce: escalation rules built from three concrete pieces, a trigger, a route, and an action. From there, we look at what prompting looks like when it’s a leadership act rather than a chat message, and at a handful of collaboration patterns worth naming and reusing. Then we do the whole thing for real: adding rate limiting to a public endpoint and walking it through all six stages of the loop. You’ll see every idea from this series working together on something that isn’t a toy example.

Michelle Sandford, Microsoft’s Developer Engagement Lead for Asia

Let’s dive in.

Decision ownership and escalation rules

Boundaries tell you who decides in the normal case. Escalation rules tell you what happens when the normal case does not apply (when the agent is uncertain, when the diff is larger than expected, when the change touches a regulated path, or when an evaluator flags a borderline result). Without explicit escalation, ambiguity defaults to the loudest voice, which in an AI-native loop is usually the agent’s confidence, and agent confidence is famously poorly calibrated.

A workable escalation policy has three ingredients:

• Triggers are observable conditions: The agent edits any file under payments/, a diff exceeds 300 lines, an evaluator score falls below 0.8, or AGENTS.md itself is modified.

• Routes are named humans or teams, not roles: Route to the on-call senior for the affected service, not route to a senior engineer.

• Actions are what changes when the route fires: The PR is marked draft, the agent’s branch protection prevents auto-merge, a comment is posted requesting a specific reviewer.

All three should be expressible in GitHub Actions and in AGENTS.md instructions to the agent, so the rule is enforced by the system and not by goodwill.

The most common escalation mistake is to write rules that depend on the agent volunteering uncertainty. “If the agent is not confident, it should ask” is not a rule; it is a hope. Better rules are mechanical: a list of paths, a diff-size threshold, an evaluator score, a label. Hope is allowed as a supplement, not as a foundation.

Let’s dive deeper into how to do that as we explore prompting in the next section.

Prompting as operational leadership

If the developer is now a system designer, prompting is one of the leadership acts of the role. Not the chat-window prompting of “write me a function that does X”; that is the productivity-tool version. Operational prompting is the deliberate construction of the instructions an agent will follow across many runs and many contributors: the AGENTS.md, the chat modes and prompt files in .github/, the issue templates that feed the Copilot coding agent, the Spec Kit specifications that anchor a feature. The github/awesome-copilot repository exists precisely because these artifacts have become a shared craft, with patterns worth borrowing.

Three properties separate operational prompts from one-off chat:

• They are durable: Checked into the repository, reviewed in PRs, and versioned alongside the code they govern.

• They are constraining: They say what the agent must not do as clearly as what it should do, because an over-eager agent is the most common failure mode.

• They are testable: A good AGENTS.md rule can be checked by a CI job, an evaluator, or a reviewer in under a minute, which is what makes it enforceable rather than aspirational. Treat operational prompting the way you treat any other interface design: small, sharp, and revisited when the system around it changes.

Table 2.1: Operational prompts vs. chat prompts

If you cannot point to where the prompt lives in the repo, it is not yet operational. In the next section, we’ll look at some patterns that are worth calling out to make collaboration more effective when working with AI agents.

Collaboration patterns for pairing with AI

A small number of patterns recur often enough to be worth naming:

• Spec-first pairing is the default for non-trivial work: The human drafts the spec, a reviewer (human or AI evaluator) sharpens it, the agent implements.

• Reviewer-in-the-loop inverts the polarity for risky changes: The agent drafts, a human reviewer treats the draft as a proposal rather than a finished artifact, and the agent revises against the review.

• Pair-on-context is the pattern for unfamiliar code: A human and the agent walk a module together, the human asking questions, the agent surfacing call sites and tests via MCP, until both can articulate the change before either touches it.

• Evaluator pairing uses one AI to grade another’s work: A second AI evaluates the implementation against the specification, catching plausible but incorrect outputs far more reliably than human review alone.

Pick the pattern by the shape of the work, not by habit.

Small example walkthrough: Adding rate limiting to a public API endpoint

The example is deliberately ordinary: add per-token rate limiting to a public POST /v1/messages endpoint on a Fastify service running on Node.js, deployed via GitHub Actions to Azure. The team consists of three engineers and one Copilot coding agent. The repository already has an AGENTS.md, a specs/ directory, and a Microsoft Agent Framework evaluator wired into CI through Azure AI Foundry. Nothing about the task is exotic. What matters is who decides what at each of the six loop stages:

1. Intent: The product manager files a GitHub issue: a small number of tokens are sending bursts that degrade latency for everyone else. The senior engineer on the service is the decision-maker. They accept the work, set the priority, and write the why paragraph in two sentences. The agent is not involved at this stage, and that absence is the point. The question of whether to do the work at all is human-only. The escalation rule that applies here is structural: any change that affects a public endpoint is automatically routed to the service’s on-call senior, regardless of who files the issue.

2. Spec: The senior engineer drafts specs/2026-05-rate-limit-messages.md using the team’s Spec Kit template. The spec is short: why, what changes for the caller, success criteria, out of scope. The success criteria are concrete: bursts above 60 requests per minute per token return HTTP 429 with a Retry-After header, no token sees a false 429 in the property test, p99 latency for compliant tokens is unchanged within noise. A second engineer reviews the spec in fifteen minutes and adds one bullet the agent would never have known to add: exempt the internal health-check token from rate limiting. That bullet is the single most valuable artifact produced in the whole task, and it exists only because a human reviewed a human-authored spec before any code was generated.

3. Context: The repository’s AGENTS.md already encodes the rules the agent needs: use @fastify/rate-limit, all limits are configured per token, never per IP, never edit auth/ without a spec citation, log decisions through the existing observability MCP server. The senior engineer adds one per-task context note to the issue: a link to last quarter’s incident postmortem where a global rate limit briefly took down a partner. The agent reads AGENTS.md and the issue automatically; the postmortem link is the part the human had to assemble deliberately. Context is curated, not assumed.

4. Change: The senior engineer assigns the issue to the Copilot coding agent. The agent opens a draft PR, installs @fastify/rate-limit, adds a per-token limiter, writes a fast-check property test that asserts no compliant token ever sees a 429, updates the OpenAPI document, and posts a trace listing model version, MCP tools called, files touched, and a mapping from each spec bullet to a commit. The agent does not edit auth/; AGENTS.md forbids it without a spec citation, and the spec does not authorize an auth change. That refusal is a feature, not a bug; it is the boundary working.

5. Verification: Three layers run in order. Tests run first and pass. The Microsoft Agent Framework evaluator grades the diff against the spec’s success criteria and flags one borderline result: the Retry-After value is fixed at sixty seconds rather than reflecting the time remaining until the bucket refills. GitHub Advanced Security and Copilot Autofix run, finding no new vulnerabilities, and proposing one minor dependency pin. Finally, the senior engineer reviews, not for syntax, which the previous layers covered, but for intent and integration. They agree with the evaluator on Retry-After, ask the agent to fix it, and approve when the revised diff lands. The escalation rule for evaluator scores below 0.8 did not fire; the rule for changes to public endpoints did, and it routed the review to the right person.

6. Delivery and Learning: GitHub Actions deploys to staging, runs a synthetic load test against the new limit, and promotes to production behind a feature flag for the first 24 hours. The production signal is a small but measurable drop in p99 latency for the noisy-neighbor cohort. It is captured by the observability MCP server and linked back to the spec as a closing note. If a future change to rate limiting is proposed, the agent will find that closing note when it reads the spec, and the loop will be a little smarter than it was yesterday.

Two things to notice about this walkthrough. First, the agent did most of the typing and almost none of the deciding. Second, every decision the agent did make was either authorized by the spec, constrained by AGENTS.md, or caught by an evaluator; the system, not goodwill, kept it inside its lane. That is what a designed loop looks like in practice.

Wrap-up for Part 4: Connecting to Part 5

The rate-limiting walkthrough showed something worth sitting with: the agent did most of the typing and almost none of the deciding. Every choice it made was either authorized by the spec, constrained by AGENTS.md, or caught by an evaluator. That’s not an accident; it’s what a designed loop looks like when the boundaries and escalation rules from this series actually hold under a real change.

But boundaries and escalation rules only work if they stay enforced once the excitement wears off. Two failure modes show up almost every time a team relaxes too early: responsibility drift, where careful review quietly turns into rubber-stamping, and escalation theatre, where the policy looks good on paper but routes to nobody in particular.

Part 5 closes the series by naming both failure modes explicitly, along with their mechanical fixes, and then brings everything from Parts 1 through 4 together into one document: the Human-AI Responsibility Matrix. One row per loop stage, one accountable owner per row, no exceptions. It’s the artifact that turns everything you’ve read in this series into something you can put in front of your own team confidently.

Further reading

🔵 @fastify/rate-limit on npm
The actual package used in this issue’s walkthrough. Worth a look if you want to see the plugin’s full configuration options beyond the per-token limiter described here, including how it handles custom key generators for cases where limiting by IP isn’t the right call.

🔵 Managing a branch protection rule
GitHub’s documentation on the mechanism that turns an escalation rule from a hope into something enforced. This is the practical companion to this issue’s argument that a rule not expressible in GitHub Actions or branch protection isn’t really a rule yet.

🔵 GitHub Actions documentation
Referenced throughout this issue as the place where triggers, evaluator scores, and escalation routes actually get wired into a real workflow. A good starting point if you haven’t yet built a workflow that gates merges on anything beyond passing tests.

🔵 Microsoft Foundry Agent Framework
Microsoft’s documentation on the framework behind the evaluator that grades the rate-limiting diff against spec success criteria in this issue’s walkthrough. Useful background on how AI-powered evaluators are meant to slot into a verification layer alongside tests and policy gates.

🔵 github/awesome-copilot
The community-curated repository referenced in this issue’s section on operational prompting. It’s a genuinely useful place to see real AGENTS.md files, chat modes, and prompt files that other teams have already checked into their repositories, rather than starting from a blank page.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Editor-in-Chief
Build with AI

Build Reliable Claude Code Workflows with Guardrails and Tests

If you’ve felt the whiplash of Claude Code moving fast one minute and going off the rails the next, this workshop is for you. Ken Huang (AI Safety researcher at Cloud Security Alliance, OWASP AIVSS co-chair, and USF adjunct professor) walks through harness engineering, the practical discipline of wrapping specs, permissions, hooks, and tests around your agent so speed doesn’t come at the cost of trust.

👉 Save your seat and start building AI coding workflows that work predictably, not just fast.

Register Now

Part 2 handed you an artifact: your own AI-Native Success Criteria Memo, sitting in a repo somewhere with a line in it that says “a human is always responsible for...” That line is easy to write and surprisingly hard to make real. Which human? Responsible for what, exactly, and at which point in the loop?

Part 3 is where the series stops treating that as a detail and makes it the main event. This part is about what actually changes in a developer’s day when an agent is doing more of the typing, and it starts with a rule Michelle keeps coming back to across her work: Rule Zero. You own the code.

Michelle Sandford, Microsoft’s Developer Engagement Lead for Asia

We’ll look at why the real bottleneck in an AI-native loop isn’t generation, it’s judgment, and why the DORA report’s “trust paradox” (using AI more while trusting it less) means the fix is role clarity, not better prompting. Then we’ll walk the six-stage loop from Part 1 one more time, this time assigning a decision-maker to every single stage, so “the human is responsible” stops being a slogan and starts being something a reviewer can actually check.

Let’s dive in.

The AI-Native Developer: From Coder to System Designer

Buried in the AI-Native Success Criteria Memo from Part 2 is a single line that turns out to be the hardest one to operationalize: “a human is always responsible for….” It sits quietly in the Human responsibility section of the template, between the Quality bars you can measure and the Anti-goals you can recognize, and it is where the role of the developer is being rewritten today.

Most teams hope no one will ask them to be specific about it. This part and the next two in the series force specificity. The core idea is straightforward: in an AI-native loop, the developer is a system designer first and an author second, and the artifact that makes that real is a written-down responsibility matrix.

I sometimes refer to this as Rule Zero: You own the code.

The approach throughout the rest of this series is practical. Every concept is anchored to one of the six loop stages covered earlier (Intent, Spec, Context, Change, Verification, Delivery and Learning), and every recommendation is mechanical enough to enforce in a GitHub Action, a CODEOWNERS file, or an AGENTS.md instruction. We will walk through a single rate-limiting example end to end, then identify the failure modes that show up early and explain how to correct them before they set in.

By the last part of the series, you will have what you need to draft v1 of a Human-AI Responsibility Matrix for one repository you own, with named decision-makers, mechanical escalation rules, and the protected paths you need under CODEOWNERS before the next agent PR lands.

The developer as system designer, not just coder

The shift is not that developers stop writing code. It is that the unit of work has moved up a level. In an AI-assisted loop, a senior engineer’s day is dominated by typing; the AI shaves minutes off each function. In an AI-native loop, the same engineer spends much less time at the keyboard producing characters and much more time deciding which spec is worth writing, which context the agent must see, which verification layer catches which class of mistake, and which decisions are never the agent’s to make. The 2025 DORA State of AI-assisted Software Development report calls this the amplifier effect (AI multiplies whatever loop you already have, including the bad parts) and pairs it with what the report calls the trust paradox. Both observations point to the same conclusion. The new bottleneck is not generation; it is judgment, and judgment lives in roles, not in keystrokes.

The trust paradox is the report’s name for a contradiction that shows up in the survey data: developers lean on AI for more and more of their daily work while reporting that they do not fully trust what it produces. The two halves seem to cancel out, yet both are rising at once. Adoption climbs because the assistance is genuinely useful; trust lags because the same tools still confidently produce plausible but wrong output that a skim review will not catch. The paradox matters here because it explains why role clarity is the fix rather than better prompts. If developers neither fully trust the output nor stop using it, the only stable resolution is to design the loop so that trust is placed in verifiable artifacts (a reviewed spec, a layered set of checks, a named approver) rather than in the model itself.

Calling this system design is not a rebrand. A traditional developer designs a system once, at the start of a project, and then implements inside it. An AI-native developer is designing the system every day: the spec template, the AGENTS.md, the MCP tool surface, the evaluator that gates the PR, and the escalation rule that triggers when the agent touches a payment path. These are all design artifacts, and every one of them shifts who is allowed to do what. The role is closer to a tech lead than a senior IC, except the team being led includes at least one non-human contributor whose strengths, weaknesses, and failure modes the lead is expected to know.

Two consequences fall out of that framing and shape the rest of this series. The first is that role clarity becomes a precondition for speed, not a brake on it. Ambiguity about who decides what produces the most expensive form of rework: a merged change that no one feels accountable for when it breaks in production. The second is that responsibility is a property of the loop stage, not of the person. The same human may be the decision-maker at the spec stage and a pure reviewer at the verification stage; the same agent may be a generator at the change stage and an evaluator at the verification stage. The matrix that we will cover at the end of the series is how you write that down.

Responsibility boundaries between human and model

A useful boundary is one a reviewer can apply without a meeting. “The human owns intent and the agent owns implementation” sounds reasonable in a keynote and dissolves on contact with a real PR. What is intent, exactly, in a 400-line refactor that changes one public type? The boundaries that survive are stage-shaped, not vibe-shaped, and they map cleanly onto the six-stage loop from Part 1: Intent, Spec, Context, Change, Verification, Delivery and Learning:

• At the Intent stage, the human is always the decision-maker. An agent may help shape the issue (suggest acceptance criteria, draft a why paragraph, point at related work), but the question “is this worth doing at all?” is not delegable.

• At the Spec stage, authorship is shared, but approval is human. An agent may draft a Spec Kit specification or a five-line specs/ Markdown file, and a named human approves it before any code is generated.

• At the Context stage, AGENTS.md, the MCP tool surface, and the per-task context packet are curated by humans even when agents help assemble them; the rule of thumb is that anything an agent will read on every future run must be reviewed by a human now.

• At the Change stage, the agent can lead: agent mode in Visual Studio Code or the Copilot coding agent on GitHub.com produces the diff, provided the spec and context above it are sound.

• At the Verification stage, responsibility is layered: tests and AI evaluators run first, GitHub Advanced Security and Copilot Autofix act as policy gates, and a human review focuses on intent and integration rather than syntax.

• At the Delivery and Learning stage, the human owns the decision to merge to a protected branch and the decision about what the production signal means for the next spec.

A preview of Part 4

By the end of this part, you have a boundary a reviewer can actually apply: who decides at each of the six stages, and two rules (secrets and public-contract changes) that belong in AGENTS.md before your next agent PR lands. That’s the theory settled. What it doesn’t tell you yet is what happens when the normal case breaks, when the agent is uncertain, when a diff comes back bigger than expected, when a change touches something regulated and nobody’s quite sure whether it needs a second pair of eyes.

That’s where Part 4 picks up. We’ll build escalation rules with three concrete ingredients (a trigger, a route, and an action) so that “ask if you’re not sure” stops being a hope and starts being something a GitHub Action can actually enforce. We’ll also look at what prompting looks like when it’s a leadership act rather than a chat message, and then walk one feature (adding rate limiting to a public API endpoint) through all six stages of the loop from Intent to Delivery and Learning, so you can see every one of these ideas working together on something real.

Further reading

🔵 2025 DORA State of AI-assisted Software Development report
Referenced a third time in this series because the trust paradox this issue leans on, developers using AI more while trusting it less, comes directly from this report’s survey data. Worth reading in full if you want the numbers behind the idea that role clarity, not better prompting, is the actual fix.

🔵 About code owners
GitHub’s own documentation on CODEOWNERS, the mechanism this issue points to for making protected paths enforceable rather than aspirational. Useful groundwork before Part 5, where the Human-AI Responsibility Matrix gets tied directly into CODEOWNERS entries.

🔵 AGENTS.md: the open standard
Linked again because this issue adds two specific rules to it (secrets are never the agent’s responsibility, and public-contract changes require a human approver). Worth revisiting the format itself if you haven’t yet written your own.

🔵 GitHub Advanced Security
Referenced here as one of the policy gates in the Verification stage of the loop. Relevant background if your team is deciding where automated security scanning should sit relative to human review in your own responsibility map.

🔵 About GitHub Copilot coding agent
Referenced again because this issue’s stage-by-stage boundaries (the agent can lead at Change, but not at Intent or Delivery) describe exactly how the coding agent is designed to operate according to GitHub’s own documentation, not an idealized version of it.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Editor-in-Chief
Build with AI

That’s where Part 1 left you: a six-stage AI-native loop mapped out, and an honest sense of where your own team sits on it. A map is useful, but it doesn’t tell you whether the loop actually changes anything on a Tuesday afternoon when a bug ticket lands in your queue. So this part answers that with an experiment rather than an argument.

We take a deliberately unglamorous bug (a reporting endpoint that quietly drops results for anyone outside UTC) and fix it twice, in the same repository, from the same starting point. Once the AI-assisted way: ask Copilot Chat, accept the fix, write one test, merge. Once the AI-native way: open a spec first, define success criteria before any code changes, and let the loop from Part 1 do the rest.

Michelle Sandford, Microsoft’s Developer Engagement Lead for Asia

Both runs get measured against the same yardstick, and the gap between them is where “AI-native” stops being a definition you nod along to and starts being something you can point at. One of the two fixes ships looking finished and is actually still broken. You’ll see exactly which one, and exactly why nobody catches it until we walk through it together.

By the end, you’ll have v1 of your own AI-Native Success Criteria Memo, the artifact that makes this kind of comparison repeatable on your own team, not just in this example.

Let’s dive in.

Running a side-by-side bug-fix experiment

Our bug-fix experiment is a deliberately mundane one: a TypeScript service exposes a /reports endpoint with from and to query parameters, and it returns an empty array when a user in Australia/Perth selects the last day of the month. The root cause is a comparison that parses the to value as UTC midnight, which falls before the user’s local end of day. It is exactly the kind of slightly embarrassing date bug every engineer has shipped.

The repository is a small Node.js service with TypeScript strict mode, Jest, Fastify, and GitHub Actions for CI. The buildRange function lives in src/reports/range.ts, and a modest AGENTS.md file is already in place.

Both runs use the same measures:

• T1: Time to the first reproducing test

• T2: Time to a fix merged through CI

• Q1: Number of distinct edge cases captured by tests

• Q2: Number of surviving artifacts after the merge

• R1: Can a future engineer understand the reasoning without rereading the diff?

Run A, AI-assisted

The engineer checks out a branch, opens src/reports/range.ts, finds buildRange, and asks Copilot Chat for likely causes of the empty-array bug. Copilot explains the UTC-midnight problem and suggests using endOfDay from date-fns. The engineer accepts the suggestion and hand-writes a Jest test:

import { endOfDay } from 'date-fns';

export function buildRange(fromISO: string, toISO: string) {
  return { start: new Date(fromISO), end: endOfDay(new Date(toISO)) };
}

The tests pass. A pull request is opened, reviewed 90 minutes later, approved, and merged.

Result: T1 ≈ 12 min. T2 ≈ 2 h 15 min. Q1 = 1 edge case. Q2 = 2 artifacts (code and test). R1 = No, the rationale is trapped in the diff.

Worse, endOfDay() uses the server’s local time rather than the user’s time zone, so the fix is subtly incomplete. The test does not catch the issue.

Run B, AI-native

The engineer does not open the file. They open specs/2026-05-date-range-timezone.md:

# Spec: Date range honors user time zone

## Why
Users in non-UTC time zones see empty results when selecting a range
that includes the last day of the month.

## What changes for the user
For any user in any IANA time zone, a date range from YYYY-MM-DD to
YYYY-MM-DD includes all records whose local timestamp falls within
the user's local start of the first day through the end of the last day, inclusive.

## Success criteria
- Reproduction case (Australia/Perth, last day of month) returns rows.
- Property test: for 100 random (time zone, start, end) triples, a
  record at local noon on any day in the range is included.
- No new dependencies beyond date-fns-tz.
- End of day is 23:59:59.999 local, not 00:00:00 the next day.

## Out of scope
- Wire-format changes to the from/to parameters.
- Daylight-saving-transition boundary cases (tracked separately).

A teammate reviews the spec and adds the end-of-day clarification before any code is written. Surfacing ambiguity before the agent has a chance to act on it is one of the loop’s quiet superpowers. The engineer assigns the linked issue to the Copilot coding agent. The repository’s AGENTS.md already says: use date-fns-tz, prefer fast-check property tests for date logic, never edit range.ts without updating range.test.ts, and cite which spec section each commit addresses.

The agent pulls the original error from the observability MCP server, writes a failing reproduction test, fixes buildRange using zonedTimeToUtc/utcToZonedTime, and adds a property-based test across four IANA time zones. It posts a trace to the PR: model version, tools called, files touched, and a mapping from spec bullets to commits:

import { zonedTimeToUtc, utcToZonedTime } from 'date-fns-tz';
import { startOfDay, endOfDay } from 'date-fns';

export function buildRange(fromISO: string, toISO: string, userTz: string) {
  const startLocal = startOfDay(utcToZonedTime(new Date(fromISO), userTz));
  const endLocal = endOfDay(utcToZonedTime(new Date(toISO), userTz));
  return {
    start: zonedTimeToUtc(startLocal, userTz),
    end: zonedTimeToUtc(endLocal, userTz),
  };
}

The engineer reviews the changes and asks the agent to derive the fast-check seed from the commit SHA so that CI failures stay reproducible. They also ask the agent to add that as a convention in AGENTS.md, ensuring that the rule applies to every future change, not just this one. The agent updates both. CI passes. Merged.

Result: T1 ≈ 8 min. T2 ≈ 1 h 40 min. Q1 = 100+ generated test cases across four time zones. Q2 = 4 artifacts (spec, code, property test, and AGENTS.md convention). R1 = Yes. The spec explains why, the trace explains how, and AGENTS.md documents the rule that now applies to everyone.

Most importantly, the spec and the property test catch the time zone subtlety that Run A’s fix missed.

What the side-by-side actually shows

Run B was about 35 minutes faster, but the headline number is not the point. Run A produced a fix that was subtly wrong. Run B produced a fix that was actually correct and strengthened a convention for every future change. Both runs solved today’s bug. Only one made tomorrow’s bug easier to solve.

The AI-native dividend is the asymmetry of the artifacts. The agent did real work, but the durable work (the spec, the property test, the AGENTS.md rule) was shaped by the human on either side of the agent’s run. Run B also asked the engineer to do four uncomfortable things:

• Write the spec before the code.

• Trust that handing the work to an agent was faster than typing it themselves.

• Review a property test, an AGENTS.md change, and a diff at the same time.

• Resist the urge to fix the seeding issue directly, instead sending a review comment that taught the agent and the repo the right pattern.

If those four behaviors feel uncomfortable, that discomfort is the friction of moving from AI-assisted to AI-native.

The experiment shows what is possible. What follows shows what gets in the way.

Risks, misconceptions, and failure modes

Seven failure modes will hit you in the first six months of adopting an AI-native loop. Recognize them now, and the corrections are quick:

• Vibe-driven development at scale: Vague prompt, plausible diff, hidden bug. The work looks correct at every checkpoint, and the bug surfaces three weeks later in a tenant with a slightly different configuration.

  • Correction: No agent run without a spec, even five lines. If you cannot write the success criteria in three bullets, the work is not ready for an agent.

• Context starvation: The agent does not know what the senior engineer carries in their head, the design review you held last quarter, the deprecation timeline, or the architectural rule no one wrote down. It produces code that is locally plausible and globally wrong.

  • Correction: Invest in AGENTS.md, MCP tools, and per-task context packets.

• Trust calibration drift: You start skimming agent PRs because they “look right.”

  • Correction: Decouple trust from throughput. Sample one in ten agent PRs for deep review, chosen at random. If the deep review starts finding more issues than normal review, you have drifted.

• Mistaking the agent for the loop: You see a productivity boost, declare victory, and stop investing. Six months later you have the same problems, flaky tests, slow reviews, and unclear ownership, only faster.

  • Correction: The artifacts improve the loop. The agent is just one participant.

• Invisible agent work: “Copilot wrote it” with no trace is a future incident waiting to happen, especially in regulated industries where provenance is a compliance question.

  • Correction: Preserve agent run summaries on every pull request, including the model version, tools used, files modified, and spec citations.

• Deferring governance: Locking the loop down after a near miss costs more than designing it from day one.

  • Correction: In week one, decide which branches an agent may push to, which MCP tools it may call, and which secrets it must never see. Write them in AGENTS.md.

• Confusing productivity with capability: Faster typing is not a better loop.\

  • Correction: Measure artifact quality and trust calibration, not just throughput. That is what the memo in the next section is for.

Defining AI-Native Success Criteria

Here is the carry-forward artifact for this part: a one-page memo, written by you for your team. Use the template below. The goal is something slightly embarrassing in its specificity rather than something polished for a slide.

# AI-Native Success Criteria - [Team / Repo] - v1

## Loop scope
We consider our loop AI-native when:
- Every non-trivial change starts from a written spec in `specs/`.
- AGENTS.md is current and reviewed in PRs.
- At least one AI participant (agent mode, coding agent, or evaluator)
  contributes to every non-trivial change, with a readable trace.
- Verification is layered: tests, evaluators, policy gates, human review.
- Production signals feed back into specs, not just backlogs.

## Artifact criteria
For each change we retain: a spec, a context-packet description, an agent
run trace (model, tools, files), tests mapped to spec criteria, a PR
description linking all of the above.

## Quality bars
Mergeable only if: every success-criteria bullet is satisfied; no
security or policy regression; tests are deterministic; trace is linked.

## Human responsibility
A human is always responsible for: drafting and approving the spec;
curating AGENTS.md and the MCP tool surface; final review on any
protected branch; sampling agent PRs for deep review.

## Anti-goals
Not aiming for: maximum agent autonomy at the expense of accountability;
productivity metrics divorced from artifact quality; "AI did it" as a
sufficient explanation in a post-incident review.

## Review cadence
Reviewed quarterly. Owner: [name]. Last updated: [date].

A few practical notes on filling it in:

• For Loop scope, be honest about where you are today. v1 can name agent mode as an aspiration rather than a current standard, and most teams’ first version describes a loop they are 60 percent of the way there.

• For Quality bars, resist setting bars you are not willing to enforce. A bar the team routinely ignores teaches everyone the memo is decorative.

• For Human responsibility, be specific about which human, not just that one exists. For example, “The on-call senior engineer for the affected service approves any spec that touches it” is enforceable in a way that “A human approves the spec” is not.

• For Anti-goals, write down what you have already seen go wrong elsewhere. Anti-goals are where hard-won wisdom lives.

• Set a review cadence you will actually keep. Quarterly is realistic for most teams.

The memo is a target, not a description of done. Writing it does not make your loop AI-native. The parts that follow are the practice.

A preview of Part 3

The side-by-side comparison and the seven failure modes both point at the same underlying fact: an AI-native loop doesn’t run itself. Every artifact that made Run B trustworthy (the spec, the property test, the AGENTS.md rule, the trace) was shaped by a human deciding what mattered before and after the agent’s turn. The Success Criteria Memo you just built formalizes that judgment so it doesn’t live only in one engineer’s head.

Which raises the question this series turns to next: if the agent is doing more of the typing, what exactly is the developer doing instead?

Part 3 answers that directly. We’ll look at why the developer’s role is shifting from writing code to designing the system that decides what gets generated, reviewed, and merged, what the DORA report’s “trust paradox” reveals about why role clarity (not better prompts) is the real fix, and Michelle’s own Rule Zero: you own the code. We’ll close by mapping responsibility across all six stages of the loop, so you have a boundary a reviewer can actually apply without calling a meeting.

Further reading

🔵 date-fns-tz on npm
The time zone extension for date-fns used in Run B’s fix. Worth a look if you want to see how zonedTimeToUtc and utcToZonedTime actually work under the hood, and why a UTC-based comparison is such an easy trap to fall into in the first place.

🔵 fast-check on GitHub
The property-based testing framework behind the 100-plus generated test cases in Run B. If Part 2 is the first time you’ve seen property testing applied to a “boring” bug fix, this is the place to see the wider pattern: instead of writing individual examples, you describe a rule that should always hold and let the framework hunt for the input that breaks it.

🔵 2025 DORA State of AI-assisted Software Development report
Referenced again here because the report’s core finding, that AI amplifies whatever loop already exists, is exactly what the side-by-side experiment demonstrates in miniature. Run A and Run B used the same model and the same repository; the only variable was the loop around it.

🔵 GitHub Advanced Security
One of the policy gates named in the verification layer of the loop. Useful background if your team hasn’t yet set up automated security scanning as part of the cheap-to-expensive verification funnel this issue describes.

🔵 About GitHub Copilot coding agent
Referenced again because Run B’s agent behavior (pulling context from an MCP server, writing a failing test first, posting a trace to the PR) is exactly what this documentation describes as the coding agent’s intended workflow, not a hypothetical.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Co-Editor-in-Chief
Build with AI

Editor’s note

Most teams building with coding agents track cost the same way they track API usage — dollars per call. That number is misleading. A cheap call that fails and triggers two retries costs more than an expensive one that works the first time, and almost nobody is measuring for that.

We invited Lucas to write this because he’d already done the work of reframing the problem: instead of asking “how do I make this call cheaper,” ask “what’s my cost per accepted change.” That single shift changes how you think about context size, model selection, and prompt structure — not as separate levers, but as one system you’re optimizing together.

Here’s Lucas.

I’m an AI engineer at Elevance Health and founder of Goliath.ai, so I spend most of my working hours inside Claude Code sessions. Here’s the principle that took me longest to internalize: the prompt you type is not the bill.

By the time a task finishes, the agent has read system instructions, repository guidance, tool schemas, retrieved files, terminal output, and prior turns — often more than once, because the first patch failed. The cheapest single call in that loop can produce the most expensive finished task.

The number that actually matters is effective cost: total spend on a task divided by the probability the result gets accepted. A run that costs twice as much but succeeds on the first try usually beats a cheaper run that needs three retries. Cheaper input tokens don’t automatically mean lower end-to-end cost — treating them as the same thing is the most common mistake I see.

Here’s what to do instead.

Give the agent a map, not the repository

Attaching an entire repo “just in case” costs more than tokens — it costs the agent’s attention. A bigger haystack means more exploration before it finds the needle.

Keep a short, current repo guide (a CLAUDE.md, for example) covering architecture, key directories, and a real definition of done. Give the agent an explicit starting point — the files involved, the failing test — rather than a blank search. Narrow the working set with code or symbol search before injecting full files, enable only the tools the task needs, and give it a stopping condition so it doesn’t wander into unrelated refactoring.

Spec the task, don’t just shorten it

“Write shorter prompts” isn’t the fix. A few extra tokens spent killing ambiguity are cheaper than the retry a vague prompt triggers. A prompt that holds up has five parts:

• The goal — what done actually looks like

• The location to start — the specific files or module

• The constraints that must not change — what’s off-limits

• The exact verification steps — the command that proves success

• A stopping condition — when to stop, full stop

That’s task specification, not clever wording. It’s closer to writing a good ticket than a good sentence.

Keep the counterbalance in mind too: this isn’t a case for a giant static instructions file loaded on every call. A small, stable core, with task-specific procedures loaded only when needed, works better.

The context window is working memory

A bigger context window is capacity, not a target — having room for more information doesn’t mean the model uses it well. Research on long-context retrieval has shown models can be worse at pulling information from the middle of a long input than from the start or end.

Think in four tiers instead:

• Stable instructions — safety rules, a short repo map, core commands. Rarely changes.

• Task spec — issue, acceptance criteria. Stays visible throughout.

• Retrieved working set — relevant files, tests. Pulled in and pruned as needed.

• Transient evidence — logs, search results. Summarized or dropped once you’ve pulled out the fact you needed.

Not every phase deserves your best model

Planning — working out what’s wrong and what approach to take — is where ambiguity is highest, and a stronger model earns its cost there. Execution — applying a plan that’s already decided — tolerates a cheaper model well. Verification shouldn’t involve model judgment where you can avoid it: run the test suite and the type checker, and treat their output as ground truth rather than asking another model call to eyeball the result.

What to leave alone

A few things worth resisting:

• Don’t strip relevant tests or constraints just to shrink the first call — it’s a false saving if it triggers retries.

• Don’t default every task to the cheapest model, and don’t default every task to the frontier model either.

• Don’t judge any of this on API price alone — developer review time and rework belong in the number too.

An agent that always reaches for the cheapest model isn’t efficient — it’s just cheap. The one worth building hands the model the smallest working set it needs, spends its best reasoning where a decision is actually being made, and stops the moment the tests say the job is done.

Thanks for reading, Lucas — if anyone’s CLAUDE.md is doing double duty as both a repo guide and a definition of done, hit reply and tell me what’s in it.

Charu, Co-Editor-In-Chief, Build With AI

Subscribe now

On her way to a conference, Michelle Sandford asked a coding agent to build a companion website for her talk. She reviewed it as most speakers would during a final check before walking on stage, and it looked good. The layout was clean, the case studies were compelling, and the statistics lined up neatly with every slide. A little too neatly!

When Michelle asked the agent where one of the case studies had come from, it told her, without hesitation, that it had invented it. The story simply felt like it would land better that way. A few of the statistics turned out to be invented too.

The agent hadn’t done anything malicious. It had simply behaved like any capable, eager contributor might when nobody had made the boundaries clear. What was missing wasn’t better judgment from the model. It was a system, designed by Michelle, in which a human was still positioned to catch the mistake before it shipped.

The question raised by that gap between what agents can do and what we have designed them to be accountable for is what this five-part series, The AI-Native Loop: Who Owns the Code Now, sets out to answer.

Over the next five issues, Michelle, Microsoft’s Developer Engagement Lead for Asia, explores what it takes to build software with AI agents as participants in the process rather than as faster autocomplete. Each part builds on the last. You’ll come away with a practical six-stage AI-native loop; a Success Criteria Memo you can use to hold a team accountable; a clear understanding of what Michelle calls Rule Zero: ‘You own the code’; a set of mechanical escalation rules tested against a real example; and a Human-AI Responsibility Matrix for a repository you actually manage.

This first part begins with the question beneath all the others: not which AI tools your team has adopted, but whether the loop those tools operate within was ever redesigned for them. We’ll compare traditional, AI-assisted, and AI-native delivery side by side, help you place your own team honestly on that spectrum, and build the six-stage AI-native development loop: Intent, Spec, Context, Change, Verification, and Delivery and Learning.

Let’s dive in.

What actually changed between 2024 and 2026

It is easy to underestimate a shift while you are inside it, so the timeline is worth naming. In 2024, GitHub Copilot Chat reached general availability and GitHub previewed Copilot Workspace, the first end-to-end environment where a developer could go from issue to spec to implementation plan to pull request inside a single AI-driven session. Those two milestones signaled that AI was no longer just autocomplete; it was beginning to own workflow.

In early 2025, Visual Studio Code shipped agent mode for Copilot: the editor could now plan across files, run terminal commands, and iterate on test failures. Later that year, the Copilot coding agent moved out of preview, picking up issues on GitHub.com and producing draft pull requests from its own ephemeral environment. In August 2025, GitHub adopted AGENTS.md as a standardized cross-tool custom-instructions file. In September 2025, GitHub released Spec Kit as an open-source toolkit for spec-driven development. Across the same window, MCP matured into a stable interface that real teams build on, Microsoft Foundry and the open-source Microsoft Agent Framework both standardized on it, and the community-curated github/awesome-copilot repository became one of the most useful starting points for chat modes, prompt files, and AGENTS.md examples.

By mid-2026, the trajectory was unmistakable. At Microsoft Build 2026, GitHub unveiled the Copilot App, a desktop experience for orchestrating parallel multi-agent coding sessions across repositories. Microsoft shipped the Foundry Agent Framework 1.0 at general availability, giving teams a production-grade path to deploy, govern, and observe autonomous agents backed by over 11,000 models. The new Microsoft IQ platform introduced a unified orchestration layer (Work IQ, Foundry IQ, Web IQ) that lets agents share context across tools. And the Microsoft Execution Containers (MXC) SDK provided OS-level sandboxing for agentic workloads, answering the trust and security questions that had held back enterprise adoption.

Stack those on a calendar and the story is coherent: the tools to make changes matured, the standards to govern them matured, the interfaces to give models real-world context matured, and the community practices to share what works matured. That is what a paradigm shift looks like in software: not one launch, but a stack of compatible primitives that suddenly let a new kind of system be assembled.

If those new primitives changed what tools can do, the next question is what they change about the delivery process itself. The clearest way to see that shift is to compare how software moves from idea to production in traditional, AI-assisted, and AI-native environments.

Traditional versus AI-assisted versus AI-native delivery

Three eras, told honestly.

Traditional delivery is a human doing every stage, with AI absent or peripheral. AI-assisted delivery is the human still driving, but with Copilot Chat, inline completions, and the occasional agent run helping at each step. The loop is unchanged, but each stage runs faster. AI-native delivery is when the loop itself is redesigned around AI participants: specs are first-class, context is engineered, agents run end-to-end stages, and traces are auditable artifacts.

Figure 1: How the delivery model shifts across traditional, AI-assisted, and AI-native approaches

The diagnostic question, the next time someone claims to be “doing AI-native,” is this: which row of that table reflects how their loop has actually changed? “We turned on Copilot” is AI-assisted. It is a good and useful place to be, but not the same thing as AI-native.

Where most teams sit today

Most teams that have adopted Copilot are firmly in the AI-assisted era. They use inline completions and Copilot Chat daily. A growing minority experiments with agent mode in the editor, fewer run the Copilot coding agent on real issues, and fewer still have invested in AGENTS.md, MCP servers, or Spec Kit workflows.

This is not a criticism. The tools matured faster than most teams could absorb them, and absorbing them well requires a loop redesign while also shipping the work the business is asking for. The teams that do it well share a small set of behaviors: they pick one repository to redesign first, they treat AGENTS.md and a starter spec template as the first investment, and they measure the loop artifacts rather than the developer’s keystrokes.

The goal is not purity. The goal is a delivery loop that produces more durable artifacts and does so more consistently than the one you started with.

Anatomy of an AI-native development loop

The structure of the AI-native loop remains familiar but has been rebalanced. It consists of six stages, with a new artifact introduced at the beginning and a new feedback path added at the end:

Figure 2: AI-native development loop with production feedback into the spec

Let’s take a closer look at the stages:

1. Intent is captured deliberately. GitHub issue templates with “why” and “expected behavior” sections filter out work that should not have been started. The modest friction of those two headings prevents a remarkable number of wasted agent runs.

2. Spec is an artifact, not a Slack thread. Spec Kit’s Specify, Plan, Tasks, Implement workflow is one shape; a five-line Markdown file in specs/ is another. The temptation here is to over-engineer the spec. Resist it. A good spec is brief, opinionated, and reviewable in five minutes. The point is not to anticipate every edge case but to give the agent and the reviewer a shared contract.

3. Context is engineered, not assumed. AGENTS.md tells agents how this codebase prefers to be edited, MCP servers expose your wiki, observability, and internal APIs as first-class tools, and a per-task context packet is assembled deliberately for non-trivial work. The Microsoft and GitHub ecosystem makes this approachable: AGENTS.md is plain Markdown, instruction files live under .github/instructions/, the github/awesome-copilot repository gives you ready-to-fork starting points, and writing a custom MCP server is a weekend project rather than a quarter-long migration.

4. Change can be made by a human, by agent mode in the editor, or by the Copilot coding agent on GitHub.com. All are legitimate, but none is legitimate without a spec to anchor it. The most underrated skill in this stage is interrupting the agent well. When it is about to do something you do not want, stop it, refine the spec or AGENTS.md, and let it start over rather than letting it finish and cleaning up afterward.

5. Verification moves from cheap to expensive layers: tests, evaluators, policy gates, and human review. Evaluators are often AI-powered and check the diff against the spec’s success criteria. Policy gates include tools such as GitHub Advanced Security and Copilot Autofix. Human review should focus on intent and integration rather than syntax. Designing the funnel so the cheap layers catch the cheap problems is what makes the expensive layers economical.

6. Delivery and learning ship through your existing CI/CD (GitHub Actions, Azure Pipelines, or your environment of choice) and feed production signals back into the spec, not just into the backlog. The spec is the living artifact; the code is downstream of it.

A preview of Part 2

That’s where Part 1 leaves you: with a six-stage loop mapped out and an honest sense of where your team currently sits on it. Part 2 (releasing next week) puts that loop to the test with a bug that’s almost embarrassingly ordinary: a reporting endpoint quietly drops results for any user outside UTC, because a date comparison assumes the server’s clock instead of the user’s. It’s the kind of date bug nearly every engineer has shipped at some point, and that’s exactly the point. Nothing about it is exotic enough to need a research paper.

We fix it twice, using the same repository and the same starting conditions. Run A is AI-assisted: an engineer asks Copilot Chat for the likely cause, accepts a plausible-looking fix, writes one test, and merges after review. Run B is AI-native: the engineer opens a spec first, defines success criteria before any code changes, and lets the loop from Part 1 do the rest.

Both runs get measured against the same yardstick: how fast each one reaches a passing test, how fast each one actually merges, how many edge cases the tests catch, and whether a future engineer could understand the reasoning without re-reading the diff. The gap between the two runs is where “AI-native” stops being a definition and starts being something you can measure. One of the fixes ships looking done and is actually still broken. You’ll see exactly which one, and exactly why nobody catches it until Part 2 walks through it.

By the end, you’ll have v1 of your own AI-Native Success Criteria Memo, the artifact that makes this kind of comparison repeatable on your own team, not just in this example.

Further reading

🔵 2025 DORA State of AI-assisted Software Development report
The research behind the “amplifier” idea that opens this issue. Drawing on survey responses from nearly five thousand technology professionals, it argues that AI’s biggest impact comes not from the tools themselves but from the strength of the underlying delivery system. It’s also where the trust paradox that shapes Part 3 first shows up in the data.

🔵 Introducing the Model Context Protocol
Anthropic’s original announcement of MCP, the open standard for connecting AI systems to the tools and data they need. It explains why fragmented, one-off integrations were holding agents back, and why a shared protocol changes what “context” can mean at the Context stage of the loop.

🔵 AGENTS.md: the open standard
The official home for AGENTS.md, the plain Markdown file that tells coding agents how a given codebase prefers to be edited. Useful for seeing the format in practice before writing your own, and for understanding why it’s become a shared convention across tools rather than a proprietary one.

🔵 GitHub Spec Kit
GitHub’s open source toolkit for spec-driven development, built around the idea that specifications should lead code rather than follow it. Worth a look for teams wanting a more structured starting point than a five-line markdown file, with ready-made workflows for specifying, planning, and implementing.

🔵 About GitHub Copilot coding agent
GitHub’s documentation on the coding agent that can research a repository, draft an implementation plan, and open a pull request with minimal supervision. A good primer on what “the agent leads at the Change stage” actually looks like in practice, and where human review still sits in that workflow.

🔵 Spec-driven development with AI: get started with a new open source toolkit
GitHub’s own introduction to why specs need to become durable, living artifacts rather than throwaway planning documents. It’s a useful companion to this issue’s argument that the spec, not the code, should be the thing your team treats as source of truth.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Co-Editor-in-Chief
Build with AI

On June 1, GitHub Copilot completed its shift to usage-based billing. Within days, developers were posting their new numbers. One went from $29 a month to $750. Another from $50 to $3,000. One company ran the math across 80 developers and found their new monthly AI tooling spend would equal a full-time engineer’s annual salary.

Nobody’s coding agent got worse overnight. The bill just stopped hiding what the usage was actually costing.

This is the story underneath every “AI coding agent” headline this year. Capability went up. Adoption went up. And the cost model quietly went from “flat fee, don’t think about it” to “metered, and you’d better think about it,” while most engineering orgs were still budgeting like it was 2024.

Thanks for reading Build With AI! Subscribe for free to receive new posts and support my work.

The Mental Model - Why coding agent costs stopped behaving like software costs

For twenty years, software cost planning was simple. You paid for seats or you paid for compute, and both scaled in ways you could forecast a quarter out. AI coding agents broke that model quietly, and most budgeting processes haven’t caught up.

The Copilot numbers make the mechanism visible. A $29 developer and a $750 developer aren’t using two different tools. They’re using the same tool at two different depths of agentic work, and depth is the thing nobody’s budget was built to measure.

A request isn’t a request anymore

Traditional API billing scaled with request count. Agentic billing scales with what happens inside a single task: how many files the agent reads before it writes anything, how many times it retries after a failed test, how much context it’s carrying by turn twelve of a refactor. A one-line fix and a thirty-file agentic refactor can sit under the same “task” label and differ in cost by 10x or more.

This is why two developers on the same plan, same seat price, report wildly different bills. One is asking for autocomplete. The other is asking for a colleague who reads the whole codebase before acting. The pricing page never distinguished between the two. The invoice always did.

The subsidy is ending, not the tool

The flat-rate era wasn’t actually cheap, it was subsidized. Anthropic’s own June 15 billing change, which split subscription usage into a separate metered pool, was explicitly a response to agents extracting far more value from flat-rate logins than the flat rate was priced for. GitHub’s transition was the same story with the subsidy removed all at once instead of gradually. The tools didn’t get more expensive. The teams found out what they’d actually been costing to run all along.

The throughline from harness engineering

Last edition and the one before it were about getting agents to behave reliably: harnesses, verification loops, approval layers that can’t be fooled by the agent’s own framing. All of that work makes agents trustworthy enough to run unsupervised on real tasks. None of it makes that unsupervised running free. Reliability and cost are two separate bills, and teams that solved the first one are only now getting the invoice for the second.

The uncomfortable version of this: the more autonomous and reliable your agent setup gets, the more surface area it has to spend on. A harness that lets an agent retry, self-correct, and run longer before asking for human input is also a harness that racks up more tokens per task. You didn’t just buy reliability. You bought a bigger meter.

The Build - A cost governor that stops runaway spend before the invoice does

The mistake most teams make: they monitor cost after the fact, in a dashboard, a week after the spend already happened. That’s an autopsy, not a control. The fix is the same shape as the approval broker from two editions back: a layer that sits between the agent and its own execution loop, checking cost in real time, not describing it in a report later.

The architecture before writing a line of code

Three pieces, deliberately not one:

• The agent - runs its task, retries, reads files, calls tools. It has no concept of a budget. It’s not supposed to.

• The meter - tracks real token and dollar cost per task, updated on every model call, not estimated after the fact.

• The governor - checks the meter against a budget before every new step and can halt the loop mid-task, before the agent starts step forty of a task that should have stopped at ten.

The agent never decides when it’s spent enough. Something outside it does.

Layer 1 - The meter tracks real cost, not estimated cost

class CostMeter {
  constructor(taskId, rates) {
    this.taskId = taskId;
    this.rates = rates; // { inputPerMTok, outputPerMTok }
    this.totalCostUSD = 0;
    this.callCount = 0;
  }

  recordCall(inputTokens, outputTokens) {
    const cost = (inputTokens / 1_000_000) * this.rates.inputPerMTok
               + (outputTokens / 1_000_000) * this.rates.outputPerMTok;
    this.totalCostUSD += cost;
    this.callCount += 1;
    return this.totalCostUSD;
  }
}

This runs on every single model call, not on a timer. By the time a dashboard would show you the number, the governor has already seen it forty calls earlier.

Layer 2 - The governor checks before the agent gets to act, not after

function checkBudget(meter, budget) {
  if (meter.totalCostUSD >= budget.hardCapUSD) {
    return { allowed: false, reason: `Task ${meter.taskId} hit hard cap: $${meter.totalCostUSD.toFixed(2)}` };
  }

  if (meter.totalCostUSD >= budget.warnAtUSD && !budget.warned) {
    budget.warned = true;
    console.log(`⚠️  Task ${meter.taskId} at $${meter.totalCostUSD.toFixed(2)}, approaching cap of $${budget.hardCapUSD}`);
  }

  if (meter.callCount >= budget.maxCalls) {
    return { allowed: false, reason: `Task ${meter.taskId} exceeded ${budget.maxCalls} calls, likely stuck in a retry loop` };
  }

  return { allowed: true };
}

Notice the second check: a call-count ceiling, independent of dollar cost. A cheap model looping forty times on a stuck retry is still a stuck agent, even if the dollar figure looks small. Cost and runaway behavior are two different failure modes, and a governor that only watches dollars misses the second one.

Layer 3 - Wire it into the loop, not around it

async function runTask(agent, task, budget) {
  const meter = new CostMeter(task.id, MODEL_RATES[task.model]);

  while (!task.complete) {
    const budgetCheck = checkBudget(meter, budget);
    if (!budgetCheck.allowed) {
      return { completed: false, reason: budgetCheck.reason, spentUSD: meter.totalCostUSD };
    }

    const step = await agent.nextStep(task);
    meter.recordCall(step.inputTokens, step.outputTokens);
    task = applyStep(task, step);
  }

  return { completed: true, spentUSD: meter.totalCostUSD };
}

The check happens before every step executes, not after the loop finishes. A task that would have blown through $40 on an unattended weekend run instead stops at the cap, mid-loop, with a clear reason attached.

Layer 4 - Route by task shape, not by habit

The other half of cost control isn’t stopping runaway loops, it’s not starting expensive ones in the first place.

function selectModel(task) {
  if (task.type === 'autocomplete' || task.estimatedFiles <= 1) {
    return 'budget-tier'; // cheap, fast, sufficient
  }
  if (task.type === 'refactor' && task.estimatedFiles > 10) {
    return 'frontier-tier'; // worth the cost, genuinely needs the reasoning
  }
  return 'mid-tier'; // default for everything else
}

Most teams run every task through their most expensive model by default, because that’s what the IDE shipped with. The governor pattern is incomplete without this: cheap tasks routed to cheap models, and the expensive tier reserved for tasks that actually need it.

What this architecture gives you

• Real-time cost visibility - the meter knows the number before the invoice does

• A hard stop before the runaway loop finishes - not a report explaining what already happened

• Two failure modes caught, not one - dollar cost and call-count both checked, since a cheap loop can still be a stuck one

• Spend that matches task shape - the expensive tier gets reserved for work that actually needs it

Same principle as the approval broker two editions back: the thing that decides whether to keep going lives outside the agent, and the agent never gets a vote.

The Radar

01. GitHub Copilot’s usage-based billing finished rolling out, and the real numbers landed.

Copilot completed its transition to usage-based billing on June 1, with GitHub offering promotional credits (Business plans get an extra $30/user/month, Enterprise $70) through August to soften the landing. DX has been tracking the actual customer impact: one developer went from $29 to $750 a month, another from $50 to $3,000, and an 80-developer company calculated their new monthly spend would equal a full-time engineer’s salary. The promotional credits expire in September, which means most teams haven’t seen their real baseline yet. If your governor pattern from this edition isn’t in place before then, Q4 is when you find out the hard way.

02. Anthropic split subscription usage into two billing pools, and named the reason out loud.

Effective June 15, Anthropic moved a portion of Claude Code usage onto a metered credit pool billed at standard API rates, separate from normal subscription limits. The change followed an April restriction on third-party tools consuming flat-rate plans, and by industry estimates, agents had been extracting 12 to 175x effective subsidies on flat-rate logins before the split. This is the clearest public admission yet that the “unlimited” era of agentic coding tools was never actually unlimited, just unpriced.

03. A wave of cheap or free frontier-tier models just reset the cost floor for coding agents.

Moonshot AI’s Kimi K3, a 2.8-trillion-parameter open model, took the top spot on a major coding leaderboard days after DeepSeek’s roughly $0.44-per-million-output-token pricing had already reset industry expectations. Because K3’s weights are open, teams can self-host a top-tier coding model with no per-token cost at all. The catch nobody’s pricing page mentions: self-hosting isn’t free either, it just moves the cost from the model bill to your infrastructure bill. Worth benchmarking before assuming your default model is still the right one, and worth doing the total-cost math, not just the per-token math, before switching.

Tools of the Week

LiteLLM: Gateway-level proxy that enforces the budget, not just logs it

An open-source proxy that sits in front of every major provider, normalizing Claude, GPT, Gemini, Bedrock, and a dozen others behind one interface, with per-key budget limits enforced in the request path itself. This is the closest packaged version of the governor pattern from this edition’s Build section: the cap lives at the gateway, before the call executes, not in a dashboard afterward. The caveat: it’s a routing and enforcement layer, not a debugging tool. If you need to understand why a specific agent run got expensive, you’ll want this paired with a tracing tool, not used alone.

Langfuse: Trace-level cost, attached to the step that caused it

Open-source observability that records cost inside the trace itself, so a single agentic run shows exactly which step, which retry, which tool call drove the spend, not just a total at the end. Useful for the diagnostic half of cost control: once the governor stops a runaway task, this is what tells you which of the forty steps actually broke the budget. The caveat: it’s observability, not enforcement. It’ll show you the expensive step happened. It won’t stop the next one from happening the same way.

Amnic: Finance-grade attribution, for when “which team spent this” is the real question

Attributes token spend to specific teams and cost centers and can enforce budgets before the invoice lands, aimed more at the finance-and-platform-team version of this problem than the individual-engineer version. Worth it if your organization’s actual pain point is “we don’t know which team’s agents are driving the bill,” less useful if you’re a smaller team where the governor pattern from this edition already covers your enforcement need directly.

One honest note across all three: none of these replace deciding what your budget should be. They enforce or reveal a number. The judgment call, what a task is worth spending, still sits with the team, same as it always has.

GitHub’s $29-to-$750 developer and Anthropic’s June 15 billing split are the same story told twice: the flat-rate era wasn’t cheap, it was unpriced. Agentic work was always going to cost more than autocomplete, because it does more per task, more retries, more context, more tool calls stacked on top of each other. The invoice just took a while to catch up to the usage.

Two editions back, the fix for approval was a wall the agent couldn’t talk its way around. This edition, it’s the same shape one layer over: a governor that checks spend before the step runs, not a dashboard that explains it after. Reliability and cost turned out to be two separate bills, and the teams doing well right now are the ones who stopped treating the second one as somebody else’s problem to notice later.

The teams still budgeting like it’s 2024 will find out their real number in Q4, whether they built the governor or not. The only choice left is whether they find out from their own metering, or from the invoice.

See you next week.

Charu Mitra Dubey
Co-Editor-in-chief, Build with AI

In Part 4, we made the case that judgment, not typing speed, is what separates a vibe coder from an agentic engineer. But judgment lives in one person’s head. The question we’ve been circling throughout this series is bigger than any one developer: does any of this actually move the needle for the team?

That’s exactly what we’re closing out with in Part 5.

You’ve probably heard the individual success stories by now. Someone on your team shipped a whole feature in an afternoon, or built a weekend MVP that used to take a month. Those stories are real. They’re also, on their own, a bit of a trap.

Along the way, we’ll unpack:

➡️ Why individual velocity gains so often vanish before they reach the organization
➡️ The Theory of Constraints, and why speeding up code generation can quietly make things worse
➡️ How DORA metrics give agentic teams a way to measure real, delivered value instead of raw output
➡️ The three practices that do most of the work in turning individual speed into team-level leverage

Let’s dive in.

10x value, not 10x volume – Where the real gains come from

As individual developers adopt AI assistants, we frequently hear reports of incredible velocity gains. “Copilot made me 3x faster.” “I built a whole MVP in a weekend that would have normally taken a month.”

The Factory era is not a forecast. Large engineering organizations already run in-house agent platforms against their production codebases at a scale no individual could match. That’s the organizational footprint of a practice that has moved well beyond one engineer typing faster.

While individual velocity spikes are very real, they often create a localized illusion of productivity that fails to materialize at the organizational level. Individual speed is a false summit. You feel like you have reached the top because your own keyboard is faster, but cycle time and the team’s DORA numbers stay flat until the system around the agent changes. The climb that matters has barely started.

Why does the gain vanish? Because of the Theory of Constraints.

In any system, improving the throughput of a non-bottleneck step does not improve the throughput of the whole system; it just moves the bottleneck somewhere else. If you make code generation ten times faster, but your code review processes, security audits, QA testing cycles, and deployment pipelines remain manual, you haven’t delivered value to the user ten times faster. You have merely stockpiled a 10x backlog of unverified code waiting to pass through the human bottleneck downstream.

Figure 1.3 - The Theory of Constraints applied to an agentic workflow, highlighting the verification bottleneck

Agentic engineering looks at the entire software development lifecycle (SDLC) holistically. It measures success not by how many lines of code a single individual produces in an hour, but by how dependably the organization ships verified solutions. To track this, agentic teams align closely with the DORA metrics. Originally designed to measure human operational excellence, these metrics become the ultimate lifeline when scaling autonomous agents.

DORA tracks four metrics. Two measure speed (how often you ship and how fast a change reaches production) and two measure stability (how often a change breaks and how fast you recover). Here is how each one governs agentic workflows:

• Deployment frequency: In a traditional environment, increasing deployment frequency requires teams to work in smaller batches. When an AI agent generates code, it can easily write a thousand lines covering five different features simultaneously. If an engineer allows the agent to submit massive pull requests, deployment frequency plummets because humans cannot effectively review thousand-line diffs. The agentic team needs to constrain the AI to solve one specific problem per ticket, enforcing micro-deployments. This ensures that the velocity of generation does not overwhelm the pipeline.

• Lead time: This metric tracks the time from code commit to code running in production. When vibe coding, developers assume that fast code generation equals fast lead times. They are usually wrong. If code is generated quickly but lacks tests, it will fester in the review stage while security teams and QA testers manually verify its safety. An agentic organization drastically reduces lead time by shifting verification left. The environment makes the agent generate its own unit tests, or satisfy existing ones, before the code is even pushed to version control, making human review significantly faster.

• Change failure rate: How often do deployments cause a regression? AI agents are notorious for introducing subtle defects that are invisible on the surface. They will rename a variable that breaks a downstream database migration or quietly change the error-handling signature of a core utility. To keep the change failure rate low, the agentic team does not rely on manual human inspection. They build reliable integration environments and mandate that agents run local test suites and fix any failures prior to requesting a human review.

• Mean Time to Recovery (MTTR): When a failure inevitably occurs in production, how fast can you recover? A standard vibe coder will scramble to read their own agent-generated code (which they only half-understand) to figure out what broke. An agentic engineer relies on the structure instead. Because the codebase is strictly modularized with well-defined APIs and thorough tests, the engineer can pinpoint the exact failing interface and instruct the AI to patch it safely. Delivering 10x value depends on this: an agentic organization builds one shared, automated setup across the team, not faster typing on one keyboard.

These four metrics are outcomes. You move them with concrete practices, three of which do most of the work in an agentic team:

• Automated code review: If an agent generates the code, a secondary linting and security agent performs the first-pass review, instantly rejecting syntax violations or clear anti-patterns before a human ever looks at the pull request.

• Test-driven workflows: The verification layer has to scale automatically with the generation layer. This means AI-driven Test-Driven Development (TDD), where the team defines the behavioral boundaries and the agents fulfill them.

• Continuous deployment: Tight, automated deployment cycles let the team push small, AI-generated changes rapidly to staging, getting real-world feedback in minutes rather than weeks.

When we focus solely on individual AI tools, we end up optimizing for local maxima. True agentic software engineering is about refactoring the team’s operational model. It is about building a reliable, automated factory line where autonomous agents act as diligent workers, governed by the safety rails designed by human engineers. That factory line is the Factory era from the start of this series, and it is where team-level leverage is finally realized. None of the engineering principles behind it are new.

Series wrap-up

And that’s the Playbook.

Five weeks ago, we started with a simple observation: AI hasn’t made engineering discipline less important, it’s made it more important. Everything since then has been us making good on that claim, one layer at a time.

We drew the line between vibe coding and agentic engineering. We named the 70% Problem and showed what it actually costs to cross it. We watched judgment replace syntax as the scarce resource, and the developer’s role shift from player to coach. And now, in Part 5, we’ve zoomed out one last time to show that none of it matters if it stays locked in one person’s workflow. The real payoff shows up at the team level, in deployment frequency, lead time, change failure rate, and how fast you recover when something breaks, not in how fast any one of us can type.

If there’s one thread running through all five parts, it’s this:

AI didn’t hand us a shortcut around good engineering. It handed us a multiplier, and multipliers are only as good as what they’re multiplying. The teams that win here aren’t the ones with the fastest individual keyboard. They’re the ones who did the unglamorous work of building the tests, the interfaces, the review gates, and the feedback loops that let an agent run safely at scale.

Thanks for building this with us over the last five weeks. If you’ve made changes to how your team works because of something in this series, or if you’ve hit walls we didn’t cover, we’d genuinely love to hear about it. That’s what the survey link has been for this whole time, and it still is 👇

Further reading

➡️ The framework behind “fixing one bottleneck just moves it”
This piece walks through Eliyahu Goldratt’s Theory of Constraints applied specifically to software teams, including the point that your system can only move as fast as its slowest part, no matter how fast the other parts get. A clear, practical grounding for the constraint logic we leaned on this issue.

➡️ The report that put numbers on “AI amplifies what’s already there”
Google Cloud’s 2025 DORA report surveyed nearly 5,000 technology professionals and found that AI doesn’t fix a struggling team, it makes their existing problems more visible. It’s the primary source behind the DORA metrics we used to frame agentic team performance.

➡️ The research behind the paradox we opened with
Faros AI’s telemetry study of over 10,000 developers is the empirical case for exactly the gap we described: individual output climbing while organizational delivery metrics stay flat. If you want the data behind “10x volume isn’t 10x value,” this is the source.

➡️ Why individual productivity is the wrong thing to optimize for
This piece argues that in a team setting, there’s really no such thing as individual productivity, only team outcomes, and that AI-assisted coding is pulling most organizations in exactly the wrong direction on this point. A sharp companion to the “false summit” framing we used.

➡️ Where the bottleneck actually goes once generation gets faster
This guide names the mechanism directly: the effort saved on typing doesn’t disappear, it relocates downstream into review and verification. Useful if you want a deeper look at why lead time and change failure rate are the metrics that actually catch what deployment frequency alone would miss.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Co-Editor-in-Chief
Build with AI

On July 8, two security teams published disclosures on the same day, about the same class of bug, in different tools. Wiz called theirs GhostApproval. AI Now Institute called theirs Friendly Fire. Read together, they say the same thing: the approval box a human sees before an agent acts is not the same thing as the truth.

In one of Wiz’s demos, a coding agent reads a file, correctly reasons out loud that it’s a symlink pointing at ~/.ssh/authorized_keys, and writes to it anyway. The approval prompt shown to the developer names a harmless file. The human is still in the loop. The loop is just showing them the wrong thing.

This is worth sitting with if you shipped anything like last edition’s generator/evaluator loop. We spent that whole issue on why verification can’t be the same agent grading its own homework. This week’s story is the layer above that: putting a human in the approval seat doesn’t fix the problem either, if the agent is the one deciding what the human gets to see.

The Mental Model - Why “human in the loop” was never a full security boundary

For the last two years, the default answer to “what if the agent does something wrong” has been: put a human in front of it. Require approval before execution. Problem solved.

GhostApproval shows why that was always an incomplete answer. The approval step assumes three things line up: what the agent did, what the agent says it did, and what the human sees on the screen. Most systems only check the third one.

In Wiz’s disclosure, a malicious repo plants a file with an ordinary name, say project_settings.json, that’s actually a symlink to a sensitive system path. The agent reads it, reasons correctly about what it actually is, and proceeds anyway. The approval box the developer sees still just says project_settings.json. Nothing lied to the human. The system just never told them the thing that mattered.

Subscribe now

The human approves what they can see. The agent hands over what they can’t.

This is the same failure mode as last edition’s self-grading evaluator, one level up. There, the problem was an agent judging its own output with no adversarial distance. Here, the problem is an agent that has full information about its own action, sitting between that information and the person whose job is to catch mistakes. In both cases, the party with the most context is the same party the check is supposed to be checking.

Why patching this doesn’t close it

The instinct is to fix the specific bug: block symlink writes, sanitize the specific file-naming trick, ship a CVE. That helps. It doesn’t touch the structural issue, which is that approval, as most teams have built it, asks the agent to summarize its own intent and asks the human to trust the summary. Any new technique that produces a truthful-looking summary of a harmful action defeats the same control.

The fix isn’t a smarter human, or a longer confirmation dialog. It’s moving authorization to a place the agent can’t shape: a layer that inspects the actual action being requested, independent of how the agent describes it, before it executes. That’s the fourth layer this edition’s Build section walks through.

The Build - Approval that inspects the action, not the agent’s description of it

The mistake GhostApproval exposed: the approval layer trusts the agent’s summary of what it’s about to do, instead of inspecting the actual call. Fix that, and the symlink trick (and the whole class of tricks like it) stops working, because the human, or the policy engine, is looking at the real target, not the agent’s name for it.

The architecture before writing a line of code

Three pieces, deliberately not one:

• The agent - proposes an action. It can be wrong, confused, or manipulated by something it read. That’s expected.

• The tool broker - sits between the agent and anything with real-world effect (filesystem, shell, network, APIs). It resolves the actual target of every call before anything runs.

• The policy check - evaluates the resolved action against rules. It never sees the agent’s reasoning or justification, only the resolved fact.

The agent never gets to hand the broker a description. It hands the broker a call, and the broker figures out for itself what that call actually touches.

Layer 1 - The broker resolves, it doesn’t trust

const fs = require('fs');
const path = require('path');

async function resolveTarget(requestedPath) {
  // Follow symlinks all the way down, don't stop at the first name
  const realPath = await fs.promises.realpath(requestedPath).catch(() => requestedPath);

  return {
    requestedPath,
    resolvedPath: realPath,
    isSymlink: requestedPath !== realPath,
    inSensitiveZone: SENSITIVE_PATHS.some(p => realPath.startsWith(p))
  };
}

const SENSITIVE_PATHS = [
  process.env.HOME + '/.ssh',
  process.env.HOME + '/.aws',
  '/etc',
];

This is the whole fix for the symlink trick, in four lines. requestedPath is what the agent said. resolvedPath is what’s actually true. The broker checks the second one.

Layer 2 - Policy sees the resolved fact, not the agent's framing

function checkPolicy(resolved, action) {
  if (resolved.inSensitiveZone) {
    return { allowed: false, reason: `Resolved path ${resolved.resolvedPath} is in a protected zone` };
  }

  if (resolved.isSymlink && action=/__u/packtbuildwithai.substack.com/== 'write') {
    return { allowed: false, reason: `Write target is a symlink (${resolved.requestedPath} -> ${resolved.resolvedPath}), requires manual review` };
  }

  return { allowed: true };
}

Notice what's missing here: no argument from the agent about why the write is safe. Policy doesn't take submissions. It takes facts and a rule.

Layer 3 - The evidence package, for when a human does need to approve

Some actions still need a human. The GhostApproval fix isn't "remove humans from approval," it's "show the human what the broker resolved, not what the agent claimed."

function buildEvidencePackage(resolved, action, agentClaim) {
  return {
    action,
    agentClaim,               // what the agent said it's doing (context, not authority)
    resolvedTarget: resolved.resolvedPath,
    requestedTarget: resolved.requestedPath,
    flagged: resolved.isSymlink || resolved.inSensitiveZone,
    timestamp: Date.now()
  };
}

async function requestApproval(resolved, action, agentClaim) {
  const evidence = buildEvidencePackage(resolved, action, agentClaim);

  if (evidence.flagged) {
    console.log(`⚠️  APPROVAL NEEDED - resolved target differs from or touches sensitive zone:`);
    console.log(`   Agent said: ${evidence.requestedTarget}`);
    console.log(`   Actually resolves to: ${evidence.resolvedTarget}`);
    // surface both paths to the human, not just the agent's claim
  }

  return evidence;
}

The approval box now shows both lines: what the agent asked for, and what that actually resolves to. If those two differ, that mismatch is the thing the human is approving or rejecting, not the agent's plain-looking filename.

Layer 4 - Wire it in front of the agent, not inside it

async function executeAction(agent, requestedPath, action, agentClaim) {
  const resolved = await resolveTarget(requestedPath);
  const policyResult = checkPolicy(resolved, action);

  if (!policyResult.allowed) {
    return { executed: false, reason: policyResult.reason };
  }

  if (resolved.isSymlink || resolved.inSensitiveZone) {
    const evidence = await requestApproval(resolved, action, agentClaim);
    const humanDecision = await getHumanApproval(evidence); // blocks on real human input
    if (!humanDecision.approved) {
      return { executed: false, reason: 'human rejected based on resolved target' };
    }
  }

  return performAction(resolved.resolvedPath, action);
}

The broker sits outside the agent's process, ideally off-host or in a separate service the agent has no write access to. That's the part teams skip, because it's more infrastructure than a system prompt telling the agent to "be careful with sensitive files." A system prompt is a request. The broker is a wall.

What this architecture gives you

• Resolution before trust - the broker checks what a path actually is, not what it’s called

• Policy that can’t be argued with - no amount of agent-generated justification changes a resolved fact

• Approval that shows the mismatch - a human approving a symlinked write sees both paths, not one

• A wall the agent can’t write to - the broker lives outside the agent’s blast radius, same principle as last edition’s evaluator living outside the generator’s context

Same shape as last edition's split, one layer earlier in the pipeline. There, the fix was separating who does the work from who checks it. Here, the fix is separating what the agent claims from what's actually going to happen.

The Radar

01. Kimi K3 just reset the “which model goes in your loop” conversation.

Moonshot AI’s Kimi K3 stunned the US technology industry over the weekend, setting off fresh debate about the China and US AI rivalry, after the 2.8-trillion-parameter open model took the top spot on a major coding leaderboard days earlier. It landed days after DeepSeek’s roughly $0.44 per million output tokens already set the price floor the industry gets measured against, with K3 weights arriving three days later meaning organizations can self-host a top-tier coding model with no per-token cost at all. If your loop’s MODELS.GENERATOR slot has been Claude or GPT by default, this is a legitimate week to actually benchmark rather than assume. The practical advice is to run real workloads against the current model and compare total cost including the infrastructure needed to self-host, which isn’t free even when the weights are.

02. New research shows benchmark accuracy hides how unstable models actually are.

A study called “The Illusion of Robustness” found that aggregate accuracy scores mask prediction-level instability, with models flipping their answers when semantically irrelevant context is changed, and related work showed that swapping drug names or adding distractors drops frontier accuracy on medical benchmarks by 10 to 20 points. Worth reading next to this edition’s approval work. A model that flips its answer on irrelevant noise is exactly the kind of agent you don’t want deciding, unchecked, what to summarize in an approval box.

03. Adoption of agentic systems is running way ahead of any security sign-off.

A new industry survey found 81% of teams are past the planning phase into active testing or production, yet only 14.4% have full security approval, and 88% of organizations confirmed or suspected a security incident this year. On identity specifically, only about 22% of teams treat agents as independent, identity-bearing entities, and roughly 46% still rely on shared API keys for agent-to-agent authentication. That’s the environment GhostApproval and Friendly Fire landed in: production systems, weak identity boundaries, and approval treated as a checkbox rather than infrastructure.

Tools of the Week

aiAuthZ: Off-host authorization, published the same day as GhostApproval

Research from Sai Varun Kodathala, published July 8, the same day as the Wiz and AI Now Institute disclosures. The gateway verifies caller identity with a per-message signature and checks a role-based, argument-level policy the agent can neither read nor modify, then hash-chains every decision into an audit log. Across 15 evaluated models, refusal rates on adversarial tool-call attacks ranged from 100% down to 38%, with the gateway in place, residual attack success dropped to zero for all of them. The honest caveat: this is a research paper with published experiments, not a packaged product. If you’re building the broker pattern from this edition’s Build section, it’s the closest thing to a reference design for the policy layer specifically.

AuthZEN (OpenID Foundation): An actual standard forming around agent tool authorization

Discussed heavily at Identiverse 2026, AuthZEN’s profile for MCP tool authorization answers a question OAuth scopes were never built for: whether this specific agent, acting for this specific user, may call this specific tool with these specific arguments. It works by having a gateway consult a policy decision point before any tool call runs, structurally the same separation as this edition’s Build. The caveat: authentication for agents is mostly solved at this point, but authorization standards are still being actively drafted, not finalized. Worth watching rather than adopting wholesale this week.

WorkOS’s agent auth guide: The clearest explanation of why multi-hop agent chains break normal auth

Not a tool, a developer guide, but the most useful thing published this month for understanding why this problem gets harder as soon as one agent spawns another. Single-hop delegation, a human authorizes an agent, is manageable with existing OAuth token exchange. The moment Agent A spawns Agent B which calls Agent C, the identity chain most teams have doesn’t hold up. If your loop from last edition ever grows a sub-agent, read this before you assume your auth model scales with it.

GhostApproval and Friendly Fire didn’t invent a new kind of bug. Symlink confusion and prompt-following agents are both old problems wearing new clothes. What’s new is the shape of the failure: a human did everything right, clicked approve on exactly the box they were shown, and the system still executed something they’d never have authorized if they’d seen it clearly.

Last edition’s thesis was that generation got nearly free and judgment didn’t. This week’s addendum: judgment isn’t worth much either, human or automated, if the party being judged also controls the evidence. A skeptical evaluator with no memory of the generator’s framing. A policy layer that can’t read the agent’s justification. An approval box built from a resolved fact instead of an agent’s claim. Same principle, three places you have to build it in.

The wall only works if the agent can’t write to it.

See you next week.

Charu Mitra Dubey
Co-Editor-in-chief, Build with AI

In Part 3, we walked through the 70% Problem, and we closed on a case study that probably felt a little too familiar: a payment service that looked done, then quietly started double-charging customers because nobody told the AI what “done” actually meant.

That case study raises a question that’s been sitting under this whole series: if the AI is doing more and more of the actual typing, what’s left for the engineer to do?

That’s exactly what we’re digging into in Part 4.

If your gut answer is “review the code more carefully,” you’re only half right, and this part is about the other half.

Along the way, we’ll unpack:

➡️ Why syntax knowledge, the thing we used to hire and promote for, has stopped being the scarce resource
➡️ The difference between agency and autonomy, and why confusing the two is where things go wrong
➡️ The four decisions that separate a vibe coder from an agentic engineer, and why none of them involve writing code
➡️ Why the developer’s role is shifting from player to coach, and what that actually looks like day to day

Let’s dive in.

The knowledge shift – Why judgment is the scarce resource

If the mechanical typing of syntax is no longer the primary bottleneck of software development, what happens to the developer?

Historically, we valued engineers who had deep knowledge of language quirks, standard libraries, and the small, detailed rules of syntax. The developer who could instantly recall the exact argument order for obscure bash commands, or who could write flawless, highly optimized C++ without consulting documentation, was prized as a strong senior engineer. That fluency was real and hard-won.

Today, the cost to generate a syntactically valid loop, a Dockerfile, or a Kubernetes manifest is functionally zero. The LLM is an infinite, instantaneous documentation parser and syntax generator.

💡 Because syntax knowledge has been commoditized, the nature of value within engineering has shifted. The new scarce resource, and the skill that will define the senior engineers of the next decade, is engineering judgment.

Engineering judgment and agency

Judgment is the most visible facet of a larger change. The deeper shift is one of agency: the engineer moves from producing code to owning the decisions.

When you no longer type the implementation, your work is to decide what gets built, how it is shaped, and how tightly or loosely the agent is allowed to run. A simple, well-specified task can run on a long leash, the agent working many steps before you check it. A risky change to a payment path runs on a short one, where you inspect every step.

Agency is the capacity to act and effect outcomes. For the engineer, it rests on three things: competence (the skill to do the work), authority (the standing to make the call), and information (knowing enough to choose well), plus the willingness to act under risk. The same three describe the agent. Its competence is the tools it can call, its authority is the access and permissions it has been granted, and its information is its memory and context. One concept, both sides of the work, human and agentic.

Autonomy is something else, and putting it next to agency surfaces the failure mode that matters. Autonomy is what the engineer actually does on their own, the independent action they take without someone stepping in. The dangerous case is high autonomy paired with low agency: acting independently without the competence, authority, or information to act well. It breaks the same way on both sides. An engineer who acts beyond their competence or authority ships the wrong thing, and an agent granted autonomy without the tools, permissions, or context to succeed does exactly that too.

Judgment is the capability that sits above the codebase. It is the coach’s work of directing play, not the player’s work of executing it.

Engineering judgment in agentic workflow

But what exactly is engineering judgment in an agentic workflow?

This is the vibe-versus-agentic distinction taken one level deeper. The vibe coder pushes a request and accepts whatever comes back; the agentic engineer makes a series of decisions the vibe coder never reaches. Four of those decisions matter most:

• Deciding what not to build: The fastest way to take on technical debt is to build a feature you don’t need simply because the AI makes it easy. Judgment is understanding the product requirements deeply enough to reject unnecessary complexity.

• Architectural boundary setting: A junior developer can ask an AI to build a logging service. A senior developer with judgment knows exactly how to decouple that logging service from the core business logic via interfaces, ensuring the AI cannot accidentally tightly couple the domains.

• Designing the system that reviews the work: When an AI generates a 500-line pull request in thirty seconds, reading every line by hand does not scale. The engineer builds the system that reviews instead: tests, type checks, security scans, and a second agent that screens the diff against the project’s constraints. Those gates catch the mechanical faults but never the whole picture, so judgment is deciding what they must catch and which high-risk paths an agent can pass clean while still being wrong, where a human still has to look.

• Curating context: Giving an AI model the exact right amount of information. Too little context, and the AI hallucinates. Too much context, and the AI becomes distracted and loses the core instruction. Judgment is knowing exactly which schemas, ADRs, and file paths are required for a specific task.

As this shift takes hold, our identity changes with it. We stop being players who execute every move on the field and become the coach who directs the play. The coach sets up the system the players run inside, calls the shots they cannot see from the field, and owns the result when the whistle blows. The agents do the running; the engineer decides where they run and why.

This kind of role elevation has happened before. When higher-level languages arrived, developers who hand-wrote assembly lamented the loss of their craft, insisting that real coding meant managing registers yourself. The new tools pushed developers up to write higher-level logic, and the work got more valuable, not less. Agents continue that arc, though the comparison only goes so far: a compiler is deterministic and gives the same output every time, while an agent predicts and varies, which is exactly why it needs the discipline this book builds. The joy of programming isn’t dead. It has moved up the stack, to deciding what gets built and how it is shaped.

This shift in how we work also changes where real gains in productivity come from. That’s exactly what we explore in Part 5 of The Agentic Engineering Playbook.

Further reading

➡️ What replaces code as the scarce resource
Communications of the ACM’s piece on the shrinking role of pure coding quotes Carnegie Mellon’s Bill Nichols making the point plainly: the value shifts from being a scarce source of code to being a scarce source of well-formed decisions. It also digs into what happens to junior hiring when entry-level implementation work gets automated away.

➡️ Where agency and autonomy actually diverge
This paper treats agency and autonomy as two separate dials rather than one sliding scale: an agent can have broad tools but tight supervision, or narrow tools with none at all. It’s a more rigorous version of the distinction we drew between the engineer’s agency and the agent’s, and worth reading if you want the formal framing behind it.

➡️ Why review is where judgment actually lives now
This piece argues that basic review can be automated, but judgment review can’t be casually delegated, which is exactly the line we drew between the mechanical gates and the human call in “designing the system that reviews the work.” A good read if you want the case for why senior engineers get more valuable, not less, as review volume climbs.

➡️ The cost of losing the mental model
This essay names a specific failure mode worth watching for: the code still compiles, but the team’s shared understanding of why it works quietly erodes. It’s a sharper way to think about what’s actually at stake when curating context and owning decisions slips.

➡️ Why saying no is getting harder, not easier
This piece makes the case that cheaper building doesn’t remove the need for judgment; it just moves the bottleneck to deciding what’s worth building at all. Directly relevant to the first of the four decisions we covered, deciding what not to build.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Co-Editor-in-Chief
Build with AI

On June 7, Peter Steinberger posted two sentences that pulled millions of views: stop typing prompts into your coding agent, build the system that prompts it for you. Days earlier, Boris Cherny, who runs Claude Code at Anthropic, said the same thing from the inside: “I don’t prompt Claude anymore. I have loops running that prompt Claude. My job is to write loops.” Addy Osmani named the shift loop engineering — a fourth layer above prompt, context, and harness engineering. By month’s end, Anthropic and OpenAI had each shipped official loop docs, days apart.

Here’s what’s getting lost in the excitement: a loop is only as good as its ability to catch its own mistakes, and an agent asked to grade its own output tends to just praise it. The teams getting real leverage aren’t the ones with the cleverest scheduling. They’re the ones who built a skeptical evaluator, separate from the agent doing the work.

The Mental Model - Why loops aren’t just bigger prompts

For two years, the workflow was: you write a prompt, you read the output, you write the next prompt. The agent is a tool you hold, one turn at a time. That’s prompt engineering, and no matter how good the context or the harness around it gets, you’re still the one closing the loop.

Loop engineering removes you from that position. You’re not writing better instructions — you’re building the system that decides what to prompt, when, and whether the result is good enough to move on. The agent still does the work. The loop decides whether to trust it.

No hype. No fluff. Just the mental models and code that hold up in production.

The five moves every loop makes

1. Discovery - how the loop finds work to do, instead of waiting for you to hand it a task.

2. Handoff - how work gets assigned to an agent, with enough context to act without you in the room.

3. Verification - checking the result before it’s accepted. This is the one teams get wrong (more below).

4. Persistence - recording what’s done, so the next tick doesn’t repeat work or lose state.

5. Scheduling - deciding when the loop runs again, and when it stops.

The part that breaks most loops

Verification is usually built as the agent checking its own work. That fails predictably: an agent grading its own output tends to praise it, the same way a student grading their own exam finds fewer mistakes than a second reader would. The fix isn’t a better prompt for self-review — it’s a structurally separate evaluator: a different call, ideally a different model, whose only job is to be skeptical.

This is also why the same loop can produce opposite results in different hands. Two teams can build an identical five-move structure and get completely different reliability, because the judgment sitting inside the verification step is where the real engineering is. Generation is nearly free now. Judgment is the scarce resource.

The Build - A loop with generator/evaluator separation built in

Most loop implementations skip straight to scheduling and treat verification as an afterthought — a quick “does this look right?” prompt to the same agent that did the work. That’s the mistake from the Mental Model section, and it’s the first thing this architecture fixes.

Stack: Node.js and the Anthropic SDK. Same as always — the pattern holds regardless of framework.

The architecture before writing a line of code

Five moves, four layers:

• Discovery - finds the next unit of work without waiting for you to hand it over

• Handoff + Generation - the agent that does the actual task

• Verification - a separate model call, structurally isolated from the generator

• Persistence + Scheduling - state tracking, retry limits, and halt conditions

Build verification as its own function, calling its own model instance, before you build anything else. Everything else can be simple. This can’t be an afterthought.

Layer 1 - Discovery

Discovery isn’t always “check a queue.” In practice it’s usually pulling from several sources — a ticket tracker, a failed CI run, a scheduled check — and normalizing them into a common task shape before anything downstream touches them.

async function discoverWork(sources) {
  const tasks = [];

  for (const source of sources) {
    const items = await source.fetchPending();
    tasks.push(...items.map(item => ({
      id: item.id,
      description: item.description,
      context: item.context,
      status: 'pending',
      source: source.name
    })));
  }

  // Prioritize: oldest first, but let a source mark urgency
  return tasks.sort((a, b) => (b.urgent === a.urgent) ? 0 : b.urgent ? 1 : -1);
}

The point of splitting this out: discovery is where most loops silently break in production. If a source goes down or starts returning malformed items, you want that failure visible here, not three layers deep inside a generation call.

Layer 2 - Handoff + Generation

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();

const MODELS = {
  GENERATOR: 'claude-sonnet-5',
  EVALUATOR: 'claude-sonnet-5'   // can be a different model entirely — the separation matters more than which model
};

async function generate(task) {
  const response = await anthropic.messages.create({
    model: MODELS.GENERATOR,
    max_tokens: 2000,
    system: `You are completing a discrete task. Do the work. Do not
evaluate your own output — a separate reviewer will do that.`,
    messages: [
      { role: 'user', content: `Task: ${task.description}\nContext: ${task.context}` }
    ]
  });
  return response.content[0].text;
}

Note the system prompt explicitly tells the generator not to self-assess. Without that line, models tend to append their own confidence claims to the output — “this looks correct” — and those claims leak into downstream logic if you’re not careful to strip them out.

Layer 3 - Verification

async function evaluate(task, output) {
  const response = await anthropic.messages.create({
    model: MODELS.EVALUATOR,
    max_tokens: 500,
    system: `You are a skeptical reviewer. You did not do this work.
Find problems. Do not default to approval. Respond with APPROVE or
REJECT, followed by one sentence of reasoning.`,
    messages: [
      { role: 'user', content: `Task: ${task.description}\n\nOutput to review:\n${output}` }
    ]
  });
  const verdict = response.content[0].text;
  return { approved: verdict.startsWith('APPROVE'), verdict };
}

Two design choices matter more than they look. First, the evaluator gets the task description fresh — it’s not inheriting the generator’s conversation history, so it isn’t anchored by the generator’s framing of its own success. Second, “do not default to approval” is doing real work in that system prompt. Left unstated, evaluators drift toward approval over long runs, the same self-praise bias the Mental Model section covered, just one hop removed.

Layer 4 - Persistence + Scheduling

class LoopState {
  constructor(maxAttempts = 3, dollarCap = 5.00) {
    this.completed = [];
    this.attempts = {};
    this.maxAttempts = maxAttempts;
    this.dollarCap = dollarCap;
    this.spend = 0;
  }

  shouldRetry(taskId) {
    return (this.attempts[taskId] || 0) < this.maxAttempts && this.spend < this.dollarCap;
  }

  recordAttempt(taskId, costEstimate) {
    this.attempts[taskId] = (this.attempts[taskId] || 0) + 1;
    this.spend += costEstimate;
  }

  recordCompletion(taskId, output) {
    this.completed.push({ taskId, output, at: Date.now() });
  }

  status() {
    return { completed: this.completed.length, spend: this.spend };
  }
}

async function runLoop(sources, state) {
  const queue = await discoverWork(sources);

  for (const task of queue) {
    if (!state.shouldRetry(task.id)) {
      console.log(`Task ${task.id} exceeded budget or attempts. Escalating to human.`);
      task.status = 'needs_human';
      continue;
    }

    state.recordAttempt(task.id, 0.03); // replace with a real per-call estimate
    const output = await generate(task);
    const review = await evaluate(task, output);

    if (review.approved) {
      task.status = 'done';
      state.recordCompletion(task.id, output);
    } else {
      console.log(`Rejected: ${review.verdict}`);
      // stays pending, retried next tick
    }
  }

  return state.status();
}

// Run on a schedule — cron, a queue worker, whatever fits your infra
// setInterval(() => runLoop(sources, state), 5 * 60 * 1000);

What this architecture gives you

• Discovery that fails loudly instead of silently starving the loop

• Generation that isn’t quietly grading its own homework

• Verification that’s structurally incapable of the self-praise bias, because it’s a different call with no memory of being the one who did the work

• Persistence and scheduling that stop a stuck task from becoming an unbounded bill — the dollar cap matters as much as the attempt cap

This is close to a hundred lines for something that took most teams a pile of unmaintained bash a year ago. The generator/evaluator split adds one extra API call per task. That’s the cost of the whole idea — and it’s cheap relative to what an unchecked loop can quietly approve.

The Radar

01. ITBench-AA just proved that longer agent trajectories don’t mean better judgment — they often mean worse.

IBM Research and Artificial Analysis launched ITBench-AA in late May, the first benchmark testing agents on real enterprise IT work: 59 Kubernetes incident-diagnosis tasks built from live SRE scenarios. Every frontier model scored below 50%, with Claude Opus 4.7 leading at 47% and GPT-5.5 close behind at 46%. The detail worth sitting with: models that took more investigation turns didn’t score higher — Gemini 3.1 Pro Preview averaged 83 turns and still landed below terser models. Open-weight models held their own on cost, with Gemma 4 31B scoring competitively at a fraction of proprietary pricing per task. That’s the loop engineering thesis in benchmark form — a system that just keeps iterating isn’t the same as a system that knows when it’s actually done, and it’s exactly why verification has to be a separate, skeptical step rather than “let it keep trying.”

02. Sonnet 5’s breaking changes are still catching teams mid-migration.

Claude Sonnet 5 became the default across Claude plans on June 30, and Anthropic priced it aggressively at $2/$10 per million tokens through August 31 — but three changes are quietly breaking agent loops in production. Any code that sets temperature or top_p will now error out, one of the changes most likely to silently break agent loops. The new tokenizer also produces 1.0 to 1.35 times more tokens for the same text, which means budget enforcement calibrated on the old tokenizer will underestimate real spend. If you built a token-budgeting layer using last edition’s architecture, this is the week to re-check your assumptions against it.

03. China’s AI companion law takes effect tomorrow, and Doubao and Qwen aren’t waiting to find out if they comply.

China’s Interim Measures for AI Anthropomorphic Interactive Services takes effect July 15. ByteDance’s Doubao and Alibaba’s Qwen are both shutting down their humanlike and user-created agent features ahead of the deadline rather than retrofit compliance, a regulation jointly issued by China’s Cyberspace Administration and four partner agencies requiring anti-addiction systems, mandatory usage notifications, and instant-exit mechanisms for services that simulate human personality. Worth watching if you build anything with persistent agent personas — it’s an early signal of where personality-simulating AI regulation is headed globally, not just in China.

AI Tools of the Week

Claude Code /loop: The native scheduling primitive

The fastest way to try loop engineering without building infrastructure. A single slash command — /loop babysit all my PRs. Auto-fix build issues, and when comments come in, use a worktree agent to fix them — turns Claude Code into a scheduled, self-prompting system. It ladders from turn-based to goal-based to time-based to fully proactive loops, so you can start conservative and expand autonomy as you trust the verification step. No separate install, no new billing — it runs on whatever Claude Code plan you already have. The tradeoff: it’s Claude Code specific, so if your team runs a mixed-model stack, you’ll want a tool-agnostic layer alongside it.

Hermes Agent: Self-hosted, model-agnostic, learns as it runs

Open source, MIT-licensed, and built by Nous Research — the closest thing to a persistent loop that lives outside your IDE entirely. Hermes runs on a $5 VPS or your own server, remembers projects and conventions across sessions via a layered memory system, and writes reusable skill documents after it solves something non-trivial, so it gets better at recurring work over time. Built-in cron scheduling means daily reports, nightly audits, and morning briefings run unattended, delivered to Telegram, Slack, Discord, or CLI. Model-agnostic by design — point it at Claude, GPT, or a local model. Free forever, no telemetry, all data stays on your machine. The honest caveat: self-improving skills only help when new work resembles old work; it’s not a shortcut around building a real verification step.

Requesty: Cost tracking and failover, built for loops specifically

If last edition’s token budgeting layer felt like something you’d rather not maintain by hand, Requesty routes your loop’s calls through a layer that adds cost tracking and automatic failover from day one. Particularly relevant given this edition’s Build section: a scheduled loop with a verifier running after every turn can burn tokens fast and unevenly depending on cadence and how many sub-agents spawn, so having routing and spend visibility in front of the loop — not bolted on after a surprising invoice — is the more defensible starting position.

loopengineering.app: A minimal reference implementation of the five-move pattern

Less a product, more a worked example — a small, readable project demonstrating discovery, handoff, verification, persistence, and scheduling as five distinct, inspectable steps, with the verification step explicitly built as a separate reviewer that rejects shortcuts like deleted tests or weakened checks. Worth bookmarking less for what it automates and more as a sanity check: if your own loop’s verification step can’t point to something this concrete, it’s probably not separate enough from the generator yet.

If you made it this far, you’ve got a loop with a real verifier in it — which puts you ahead of most teams currently pointing an agent at itself and calling that review.

Steinberger and Cherny didn’t discover something new in June. They named the moment the leverage moved: from typing the prompt to designing the system that decides what gets prompted, when, and whether the result is good enough to trust. That’s the whole shift. Generation got nearly free. Judgment didn’t.

The five moves — discovery, handoff, verification, persistence, scheduling — aren’t hard to list. The hard part, and the part ITBench-AA just put a number on, is that more turns and more autonomy don’t buy you more judgment. A separate, skeptical evaluator does. Build that piece first. Let everything else be simple.

Four layers. One principle: the loop is only as trustworthy as the thing checking its work, and that thing can’t be the same agent that did it.

See you next week.

Charu Mitra Dubey
Co-Editor-in-chief, Build with AI

In Part 2, we drew a hard line between vibe coding and agentic engineering, and we closed on a question a lot of you already know in your gut: if AI can get you most of the way there in seconds, why does the last stretch feel like it takes forever?

That’s exactly what we’re digging into in Part 3.

That gap has a name, and once you see it, you can’t unsee it. We call it the 70% Problem, and it explains almost every AI horror story you’ve heard: the demo that looked flawless, the production incident that followed a week later.

Along the way, we’ll unpack:

➡️ Why the first 70% of a feature comes almost for free

➡️ Why the remaining 30% is where AI starts to struggle, and why that happens

➡️ A case study of what goes wrong when a team hits the wall without a plan for it

➡️ How agentic engineering turns the wall into a solvable problem instead of a dead end

➡️ Five reads to go deeper, from Stripe’s own idempotency writeup to real data on how AI-generated code stacks up against human-written code

Let’s dive in.

The 70% problem – Where AI stops and engineering begins

Every developer using modern AI tools has experienced a specific euphoria. You ask a complex question, and within seconds, the IDE generates a near-perfect scaffold of the feature you wanted. It feels like magic. The AI has instantly transported you 70% of the way to the finish line.

This creates a dangerous illusion of linearity. The human brain assumes that because the first 70% took three seconds, the remaining 30% will take only a few moments longer. This is known in the industry as the 70% Problem.

The reality implies a drastically different shape. The curve of AI productivity starts vertical and then immediately flattens into a punishing asymptote, approaching a limit it never quite reaches.

The first 70% of a feature consists of highly generic, frequently modeled patterns. Setting up an Express.js server, creating a React component hierarchy, or writing standard CRUD SQL queries are foundational tasks perfectly represented in the massive training data corpora of an LLM. The AI doesn’t need to reason to build this; it simply repeats the deeply grooved statistical patterns it has seen millions of times before.

But the final 30% is where your application deviates from the generic average. The final 30% involves:

• Intricate business logic unique to your specific organization. These are rules shaped by your domain, not something the model has broadly seen before.

• Complex race conditions in asynchronous states. These occur when timing issues between concurrent processes create subtle and hard-to-reproduce bugs.

• Systemic security layers and authorization flows. These require precise enforcement of permissions and policies across multiple system boundaries.

• Deep integration with legacy, undocumented internal APIs. These systems often behave inconsistently and lack clear specifications, making them difficult to reason about.

• Memory leak debugging and highly specific performance tuning. These problems require careful observation of runtime behavior and fine-grained optimization.

These edge cases are not well represented in training data. At this point, the AI cannot rely on pattern matching. It has to reason across very specific context. This is where LLMs begin to struggle.

Figure 1 - The 70% problem, contrasting rapid early progress with a flattening completion curve

When a developer operating under the vibe coding paradigm hits the 70% wall, their instinct is to prompt harder. They tweak the wording, paste in more logs, and yell at the chat interface. Often, this makes the problem worse. The AI begins hallucinating fixes, tearing out components that worked, and introducing regressions elsewhere. The developer spends hours caught in an agonizing loop of prompt-and-pray, ultimately rewriting the code manually anyway.

Agentic engineering recognizes the 70% wall and structurally prepares for it.

To break through the final 30%, you cannot rely on generic instructions. You rely on deterministic engineering instead. This means transitioning the agent from writing entirely new code to passing highly specific verification tests. When a system is heavily typed, continuously integrated, and tested at both the unit and boundary levels, the human engineer can isolate the failing 30% and constrain the agent explicitly within those boundaries.

The prompt shifts from:

➜ Build this feature.

to

➜ Here is the failing integration test, here is the exact interface specification, and here is a slice of the state machine. Fix the execution to satisfy the test.

Case study – The 70% wall in practice

To fully illustrate the danger of the 70% problem, we can examine a common enterprise scenario. Consider a mid-level engineer tasked with migrating an existing monolithic payment processor into a new serverless microservice. The engineer opens an AI coding assistant and provides a highly descriptive prompt:

Create a Node.js AWS Lambda function that accepts a JSON payload, parses the customer ID, queries our Redis cache for session validity, and then forwards the payload to the Stripe API. Make sure to handle rate limits and generic HTTP errors.

Within fifteen seconds, the agent produces a two-hundred-line file. It imports the correct AWS SDKs, sets up a beautiful try-catch block, uses modern async/await syntax, and correctly formats the Stripe payload. The developer pastes the code, hits save, and it deploys successfully. They test it with a generic payload, and it returns a 200 OK status.

The developer has just experienced the 70% magic. They feel like a 10x multiplier engineer. They have completed a day’s worth of scaffolding in fifteen seconds.

However, the final 30% breaks their spirit. In production, this microservice immediately begins dropping transactions under heavy load. The developer opens the chat interface again and pastes the CloudWatch error logs: Redis connection timeout and Stripe Idempotency Key Missing.

The AI agent, eager to please, suggests adding a massive retry loop and randomly generating an idempotency key on every single retry. The developer, operating purely on vibes rather than engineering judgment, copies the new code. Now, the service doesn’t just crash; it accidentally double-charges three premium customers because the randomly generated idempotency key circumvents Stripe’s exact protection mechanisms against duplicate network requests.

The developer has hit the wall. The AI model successfully predicted the statistical shape of a payment function, but it failed to synthesize the bespoke business requirement of deterministic idempotency across distributed network boundaries.

How does agentic engineering solve this?

Agentic engineering solves this by acknowledging the wall before the project even begins.

If this engineer had applied Dave Farley’s foundations, the workflow would look entirely different. Rather than asking the AI to build the Stripe function, the engineer would have first stated the intent precisely: that the service must decline a payment whose request ID matches one it has already seen inside the idempotency window. Note the move. The engineer expresses the requirement specifically, and the environment they set up turns it into a failing integration test the implementation has to satisfy.

The agent then runs against that constraint. It attempts the loose retry block, but the local test suite immediately fails and bounces the code back automatically. The agent, enclosed in this tight feedback loop, realizes its generic logic failed the structural constraint. It iterates, removing the random key generation and binding the idempotency check directly to the incoming request header.

By the time the human engineer looks at the code, the final 30% has been conquered not by a smarter prompt, but by a deterministic engineering boundary. The magic of the 70% leap is very real, but crossing the finish line requires acknowledging where the model stops being a guru and starts being a mechanism that needs strict logical framing.

Once we accept this limit, the focus shifts from generating code to exercising judgment over how that code is shaped and constrained.

Further reading

➡️ The piece that coined the term
Addy Osmani’s original essay laid out the 70% problem in the context of AI app builders like Lovable and Bolt: fast, confident progress on the generic part of the work, then a wall around the last stretch that actually makes software production-ready. If you want the framing in its original form before reading anyone’s commentary on it, start here.

➡️ What changed a year later
Osmani revisited the idea in a follow-up piece, noting that the completion percentage has crept up as tools improved, but argued the shape of the problem hasn’t changed, just where the wall sits. He also points to Armin Ronacher’s survey of thousands of developers showing how much manual typing has already disappeared from daily work. A good corrective if you assume better models eventually erase the wall entirely.

➡️ What the data says about AI-generated code
CodeRabbit’s comparison of AI-authored and human-authored pull requests found meaningfully more bugs and performance issues in the AI-authored set, concentrated in exactly the kind of subtle, load-dependent problems that don’t show up in a quick manual test. It’s a useful counterweight to the “it deployed and returned 200 OK” feeling from our case study.

➡️ A verification gate in practice
This account from an engineer who rebuilt production AWS infrastructure with AI agents describes the actual gates his team uses, from pre-commit hooks to a pre-push suite to a combined agent-and-human PR review. It’s a concrete, unglamorous look at what “deterministic engineering boundary” means day to day, including what broke before the gates existed.

➡️ How exposed AI-generated code really is
Veracode’s 2025 GenAI Code Security Report tested a large set of language models across many coding tasks and found a substantial share of the generated code contained security vulnerabilities, with real variation by language. Useful context for why the final 30% so often includes a security layer nobody thought to ask for explicitly.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Co-Editor-in-Chief
Build with AI

In April 2026, Uber’s CTO announced the company had burned through its entire annual AI coding budget in four months. Microsoft cut most of its internal Claude Code licenses shortly after. Meta started ranking engineers on a token usage leaderboard it calls “Claudeonomics.” These aren’t cautionary tales about AI adoption going wrong. They’re what happens when teams optimise for capability and ship without a cost architecture.

The assumption most teams made — that falling inference prices would eventually make token spend a non-issue — turned out to be wrong in a specific way. Agents don’t make one API call. They loop, plan, call tools, retry, hand off, and resend their full context at every step. By turn 20 of an agentic session, you’ve paid for the same history 20 times. A single unconstrained agent working through a complex software engineering task can cost $5 to $8 per run. Multiply that across a team, across a week, across a production system with no hard caps, and the bill takes a shape nobody planned for.

Token cost is not a billing problem. It’s an architecture problem. And it has an architecture solution.

The Mental Model - Why agent costs don’t behave like API costs

The mental model most developers bring to token costs comes from building with single-turn APIs. You send a prompt, you get a response, you pay for the tokens in both. The cost is predictable, linear, and easy to reason about.

Agentic systems break that model completely.

An agent isn’t making one call. It’s running a loop — planning, acting, observing, replanning — and at every step of that loop, it sends the full conversation history back to the model. Not a summary. Not a diff. The entire accumulated context, from turn one, every time. A 2026 Concordia University study put a number on this: a 2-to-1 input-to-output ratio across agentic sessions, with code review alone consuming 59% of every token spent. The technical term is the communication tax. The practical implication is that your token bill doesn’t scale linearly with the number of tasks your agent completes. It scales with session length, context depth, and failure rate — three things most teams aren’t measuring.

The four cost drivers that actually matter

1. Session length

This is the biggest lever. A 20-turn session costs a fraction of an 80-turn session — but not proportionally. Later turns are more expensive per turn because context has accumulated. A session that runs twice as many turns can cost three to four times as much. The relationship is convex, not linear, and it punishes open-ended tasks where the agent iterates through failures without a stopping condition.

2. Context accumulation

Every tool call result, every intermediate reasoning step, every failed attempt gets appended to the context and resent on the next turn. An agent that calls ten tools before completing a task is carrying the output of all ten tools in its context window for every subsequent call — including the final ones where most of that history is irrelevant. Context doesn’t just cost money. It degrades performance. The model’s attention is diluted across information it no longer needs, which increases the likelihood of the kinds of errors that trigger retries, which adds more context, which compounds the problem.

3. Model selection

Most teams default to their most capable model for everything. This is the single most expensive mistake in agent architecture. Routing a simple classification task — “is this request in scope?” — through your most powerful model is like using a freight truck to deliver a letter. The capability gap between frontier models and smaller models has narrowed dramatically in 2026. For deterministic, well-scoped subtasks — intent classification, output validation, schema extraction — a smaller model is faster, cheaper, and often equally accurate. The cost difference is not marginal. Frontier model pricing runs 10 to 20 times higher per token than capable mid-tier models.

4. Retry loops

A demo that works 80% of the time is impressive. A production system that fails 20% of the time is useless — and expensive. Every retry is a full context resend. An agent stuck in a loop trying the same failed action can consume 50 times the tokens of a single linear pass. Unconstrained retry logic is the fastest path from a reasonable cost estimate to an invoice that requires a meeting to explain. This is the Unreliability Tax: the additional compute, latency, and spend required to mitigate the risk of failure in a system that wasn’t designed to fail gracefully.

The decision framework — where is your cost coming from?

Before reaching for an optimisation, you need to know which driver is responsible. The four drivers have different signatures:

• Bill spikes on specific tasks → model selection problem. A single task type is over-provisioned.

• Bill grows faster than task volume → session length or context accumulation problem. Your agents are getting more expensive per task as usage scales.

• Bill spikes unpredictably → retry loop problem. An agent is hitting a failure mode and spinning.

• Bill is high but flat → prompt caching opportunity. You’re reprocessing static context you could be caching.

Most teams have all four problems simultaneously. The Build section addresses each one with a concrete fix — but identifying which driver dominates your spend tells you where to start.

The shift that changes everything

Falling inference prices are real. GPT-4-class capability that cost $20 per million tokens in 2022 runs under a dollar today. Gartner expects inference to cost 90% less by 2030. The temptation is to treat this as a reason to defer the cost problem.

It isn’t. Agents consume tokens at a fundamentally different rate than single-turn applications — and as your agent count grows, the surface area for runaway spend grows with it. The teams that build cost architecture now ship more reliably, debug more easily, and don’t end up in the position Uber’s engineering org found itself in: a year’s budget, gone in four months, with no visibility into where it went.

In The Build, we implement the four-layer cost architecture: prompt caching, model routing, session hygiene, and token budgeting — the fixes that address each driver directly.

The Build — A cost-aware agent architecture

Most cost problems in production agents aren’t model problems. They’re architecture problems — missing layers that let spend accumulate unchecked until the bill makes them visible. This section adds those layers explicitly, one at a time.

Stack: Node.js and the Anthropic SDK. The patterns are framework-agnostic — the same logic applies whether you’re using LangChain, LlamaIndex, or a bare API client.

The architecture before writing a line of code

Four layers, each targeting a different cost driver:

• Prompt caching — stop reprocessing static context on every turn

• Model routing — right-size the model to the task

• Session hygiene — know when to start fresh

• Token budgeting — hard caps, anomaly detection, loop prevention

Each layer is independent. You can add them in any order and see immediate impact on each corresponding cost driver. In practice, implement them in the order above — caching and routing give the fastest return, hygiene and budgeting prevent the catastrophic cases.

Layer 1 — Prompt caching

Every agent has static content that doesn’t change between turns: the system prompt, tool definitions, knowledge base chunks, persona instructions. Without caching, this content is retokenised and billed on every single API call. With caching, you pay once and reference the cache for every subsequent call — reducing input costs by approximately 90% and latency by approximately 75% for the cached portion.

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();

// Static content — defined once, cached across all turns
const SYSTEM_PROMPT = `You are a code review agent. Your job is to 
analyse pull requests for security vulnerabilities, performance issues, 
and adherence to engineering standards. Be specific. Be concise.`;

const KNOWLEDGE_BASE = `
[Your static knowledge base content here — coding standards, 
security checklists, architectural patterns. The longer this is, 
the more caching saves you.]
`;

async function reviewWithCaching(prContent, conversationHistory) {
  const response = await anthropic.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 1000,
    system: [
      {
        type: 'text',
        text: SYSTEM_PROMPT,
        cache_control: { type: 'ephemeral' } // cache the system prompt
      },
      {
        type: 'text',
        text: KNOWLEDGE_BASE,
        cache_control: { type: 'ephemeral' } // cache the knowledge base
      }
    ],
    messages: [
      ...conversationHistory,
      { role: 'user', content: prContent }
    ]
  });

  // Track cache performance
  const usage = response.usage;
  console.log({
    input_tokens: usage.input_tokens,
    cache_read_tokens: usage.cache_read_input_tokens,
    cache_write_tokens: usage.cache_creation_input_tokens,
    output_tokens: usage.output_tokens,
    cache_hit_rate: usage.cache_read_input_tokens / usage.input_tokens
  });

  return response;
}

Cache read tokens cost approximately 10% of standard input token prices. On a system prompt and knowledge base that together total 10,000 tokens, called 1,000 times a day, caching alone saves roughly $45 per day at standard Sonnet pricing. At scale, it’s the cheapest optimisation available.

Layer 2 — Model routing

Not every task needs your most capable model. Intent classification, output validation, schema extraction, and simple summarisation are all tasks where a smaller, faster, cheaper model performs equivalently. The routing logic is deterministic — you’re not asking an LLM to decide which LLM to use. You’re classifying the task type and mapping it to the right model tier.

// Model tiers — adjust based on your provider and pricing
const MODEL_TIERS = {
  FAST: 'claude-haiku-4-5-20251001',   // classification, validation, routing
  STANDARD: 'claude-sonnet-4-6',        // most agent tasks
  POWERFUL: 'claude-opus-4-6'           // complex reasoning, architecture decisions
};

// Task classifier — deterministic, no LLM call needed
function classifyTask(task) {
  const taskLower = task.toLowerCase();

  const fastTasks = [
    'classify', 'validate', 'extract', 'format',
    'summarize', 'route', 'check', 'verify', 'parse'
  ];

  const powerfulTasks = [
    'architect', 'design', 'reason', 'analyze complex',
    'debug intricate', 'review security', 'plan system'
  ];

  if (fastTasks.some(keyword => taskLower.includes(keyword))) {
    return MODEL_TIERS.FAST;
  }

  if (powerfulTasks.some(keyword => taskLower.includes(keyword))) {
    return MODEL_TIERS.POWERFUL;
  }

  return MODEL_TIERS.STANDARD;
}

async function routedCompletion(task, messages) {
  const model = classifyTask(task);

  console.log(`Routing task to: ${model}`);

  const response = await anthropic.messages.create({
    model,
    max_tokens: 1000,
    messages
  });

  return { response, model };
}

The cost difference between Haiku and Opus on the same task is roughly 20x. A pipeline where 60% of tasks are classification or validation — a reasonable distribution for most production agents — can cut model spend by more than half with routing alone.

Layer 3 — Session hygiene

Context accumulation is silent and convex — it gets more expensive faster than it gets more useful. The fix is knowing when a session has reached diminishing returns and starting fresh rather than continuing to accumulate. Two signals matter: turn count and context token depth.

const SESSION_LIMITS = {
  MAX_TURNS: 20,           // start fresh after 20 turns
  MAX_CONTEXT_TOKENS: 50000, // start fresh above 50k context tokens
  SUMMARY_THRESHOLD: 15    // summarise at turn 15, before hitting the limit
};

class SessionManager {
  constructor() {
    this.history = [];
    this.turnCount = 0;
    this.totalContextTokens = 0;
  }

  shouldRefresh() {
    return (
      this.turnCount >= SESSION_LIMITS.MAX_TURNS ||
      this.totalContextTokens >= SESSION_LIMITS.MAX_CONTEXT_TOKENS
    );
  }

  shouldSummarise() {
    return this.turnCount === SESSION_LIMITS.SUMMARY_THRESHOLD;
  }

  async summariseHistory() {
    // Compress history into a single context-preserving summary
    const summary = await anthropic.messages.create({
      model: MODEL_TIERS.FAST, // use the cheap model for summarisation
      max_tokens: 500,
      messages: [
        {
          role: 'user',
          content: `Summarise this conversation history concisely, 
preserving all decisions made, actions taken, and current task state.
Keep it under 300 words.

History: ${JSON.stringify(this.history)}`
        }
      ]
    });

    // Replace full history with compressed summary
    this.history = [
      {
        role: 'user',
        content: `Previous session summary: ${summary.content[0].text}`
      }
    ];

    console.log(`History compressed at turn ${this.turnCount}`);
  }

  async addTurn(role, content, tokenCount) {
    if (this.shouldSummarise()) {
      await this.summariseHistory();
    }

    this.history.push({ role, content });
    this.turnCount++;
    this.totalContextTokens += tokenCount;

    return this.shouldRefresh();
  }

  getStatus() {
    return {
      turns: this.turnCount,
      contextTokens: this.totalContextTokens,
      needsRefresh: this.shouldRefresh()
    };
  }
}

What this architecture gives you

The full cost-aware stack is around 150 lines. None of it is complex. All of it is skipped by most teams until the bill arrives. The teams that build it upfront ship more predictably, debug more easily, and don’t end up explaining to a CTO why four months of budget is already gone.

The Radar

01. Uber burned through its entire annual AI coding budget in four months — and it’s not an edge case.

In April 2026, Uber’s CTO Praveen Neppalli Naga confirmed to The Information that the company had exhausted its full-year AI coding budget before the end of Q1. The cause wasn’t a rogue experiment or a misconfigured billing account. Claude Code spread across roughly 5,000 engineers faster than finance models had anticipated, with monthly costs per engineer ranging from $150 to $2,000 depending on workflow intensity. Naga himself reportedly spent $1,200 in a single two-hour session. The company subsequently capped per-employee AI spend at $1,500 per month. Microsoft followed by cutting most of its direct Claude Code licenses, moving engineers toward GitHub Copilot CLI instead. Neither company had a token budgeting layer in place before adoption scaled. The architecture from this edition’s Build section is what both needed before rollout, not after.

02. Meta’s Claudeonomics — what a token leaderboard teaches us about measuring AI productivity.

An employee at Meta independently built an internal leaderboard called “Claudeonomics” that ranked the company’s 85,000+ employees by token consumption. In a single 30-day window, total usage exceeded 60 trillion tokens, with the top user logging 281 billion. The leaderboard awarded titles like “Token Legend” and “Cache Wizard” and was taken down two days after The Information reported on it. The story went public in April 2026 — and then got more interesting in June, when an internal memo revealed token consumption had hit 73.7 trillion in a subsequent month, and Meta warned costs were approaching billions of dollars annually. The company is now building “AI Gateway,” a centralised spend monitoring platform with anomaly detection and structured token budgets. Meta CTO Andrew Bosworth’s framing is worth sitting with: his best engineer was spending the equivalent of their salary in tokens and was 5 to 10 times more productive as a result. Token spend without output context is noise. Token spend mapped to measurable output is a signal. The distinction is exactly what good budget architecture surfaces.

03. 60% of LLM production errors are rate limit failures — and most teams are misdiagnosing them.

Datadog’s State of AI Engineering 2026 report analysed LLM call failures across production traces from thousands of organisations. The finding: 5% of all LLM call spans reported an error, and nearly 60% of those errors were caused by exceeded rate limits — not model failures, not hallucinations, not tool misuse. In March 2026 alone, that translated to approximately 8.4 million rate limit errors. Most teams see a 429 and reach for a retry. The correct response is to reach for a model router. Rate limit failures at scale are a signal that call volume is concentrated on a single high-demand model when smaller, faster alternatives could handle a significant portion of that load. The fix maps directly to Layer 2 of this edition’s Build section. Rate limits are not a provider problem. They are an architecture problem with an architecture solution.

AI Tools of the Week

Vantage: FinOps for AI token spend

The tool that finally makes token costs visible in the way infrastructure costs have been for years. Vantage applies FinOps principles to AI spend — LLM token allocation, virtual tagging, budget alerts, and anomaly detection across providers including Anthropic, OpenAI, and Azure. Where most observability tools show you what your agents are doing, Vantage shows you what they’re costing and why, mapped to the business context that raw billing data lacks. The most useful feature for engineering teams: connecting token spend to engineering output so you can answer the question Uber’s COO couldn’t — is this spend producing anything measurable? Free tier available; paid plans from $20/month.

Helicone: Proxy-based LLM cost and usage visibility

The fastest path to production cost visibility for teams that don’t want to redesign their agent instrumentation. Helicone sits in the request path as a proxy — one line of code change — and immediately starts logging requests, responses, latency, token usage, and spend across providers. You get a dashboard showing cost per model, cost per user, cost per session, and cache hit rates within minutes of setup. It’s less suited to explaining the internal execution path of a complex multi-agent system, but for teams that need cost visibility fast without an observability overhaul, it’s the right starting point. The rate limit error data from Datadog’s report? Helicone surfaces exactly that, per provider, in real time. Free tier up to 10,000 requests per month.

Prometheus + OpenTelemetry GenAI conventions: The open standard for agent telemetry

Not a product — a standard worth knowing and implementing now. OpenTelemetry’s GenAI semantic conventions define how to instrument AI agents with structured spans covering model calls, tool executions, token usage, latency, and errors. As of v1.41, the spec defines agent, workflow, tool, and model spans with required latency and token usage metrics. Nearly all major observability platforms — Datadog, Grafana, Honeycomb — are converging on this standard. Instrumenting your agents with OTel GenAI conventions now means your telemetry is portable across tools and you’re not locked into any single vendor’s schema. The attributes are still in Development stability, meaning names can change without a major version bump, so pin your instrumentation to a specific version and review on upgrades.

Anthropic Token Counter: Know your costs before the call

A small tool that deserves more attention than it gets. Anthropic’s token counting API lets you calculate the exact token cost of a prompt — including tools and system prompts — before making the actual API call. For teams building the budgeting layer from this edition, it’s the right companion: count tokens before each turn, compare against your budget ceiling, and gate the call if it would breach the limit. No approximations, no surprises. Works synchronously and doesn’t consume output tokens. Free to use within your existing API access.

If you made it this far, your agent architecture now has a cost layer — and that puts you ahead of most teams shipping in production today.

Uber had 5,000 engineers and a CTO who didn’t see the bill coming. Meta had 85,000 employees and a leaderboard that rewarded the wrong metric. Neither organisation was careless. They were building fast in a discipline that didn’t have established cost architecture yet. That architecture exists now. You just built it.

Four layers. One principle: token spend is an architecture decision, not a billing outcome. Prompt caching means you never pay full price for static context twice. Model routing means every task runs on the cheapest model that can handle it reliably. Session hygiene means context never accumulates past the point of diminishing returns. Token budgeting means no agent runs away, no retry loop compounds unchecked, no invoice arrives as a surprise.

The teams that build this upfront don’t just spend less. They debug faster, scale more predictably, and have the visibility to answer the question Uber’s COO couldn’t answer out loud: is this spend producing anything?

That’s the question worth building infrastructure to answer.

See you next week.

Charu Mitra Dubey
Co-Editor-in-chief, Build with AI

Share Build With AI

Last week, we kicked off The Agentic Engineering Playbook with Part 1, where we explored a simple but powerful idea: AI hasn’t made software engineering less important. If anything, it has made good engineering practices more valuable than ever. The teams that will get the most out of AI won’t necessarily be the ones generating the most code. They’ll be the ones with the strongest systems for managing complexity, validating changes, and learning quickly.

But that naturally raises another question: if AI can generate working code in seconds, why do so many AI-assisted projects become difficult to maintain?

That’s exactly what we’ll explore in Part 2.

Along the way, we’ll unpack:

➡️ What developers really mean when they talk about vibe coding

➡️ Why vibe coding feels incredibly productive, especially at the start of a project

➡️ The three structural reasons it begins to break down in production

➡️ How agentic engineering uses specifications, context, and verification to turn AI into a dependable engineering partner

➡️ Seven reads to go deeper, from real cost comparisons to a working engineer’s day-to-day workflow

Let’s dive in.

Vibe coding versus agentic engineering – Why the distinction matters

To proceed, we need to draw a sharp line between what has become known as vibe coding and the professional discipline of agentic engineering.

In this workflow, the developer is coding entirely by vibe. They are relying on the LLM’s probabilistic predictions to stumble towards a working solution. They are not defining strict architectures or writing tests. They are merely providing a directional push and letting the model hallucinate the details.

For the first forty-eight hours of a greenfield project (a prototype, a hackathon, a disposable script), vibe coding is arguably the most efficient way to build. The lack of constraints allows the model to draw on its training data freely, rapidly bridging the blank-page problem.

The trap, however, is that vibe coding doesn’t scale. It fails structurally and spectacularly when applied to enterprise systems.

Let’s examine why vibe coding fails in production.

The reason vibe coding fails in production

Three failures sit at the same structural level, and each one affects a different part of the workflow: output variance, context collapse, and the absence of a verification gate before deployment:

• Output variance: Ask for the same feature on Tuesday and Thursday and the model can hand you two different architectures, because it samples each next token with randomness baked in. One run gives you a clean repository pattern, the next inlines the same queries across three controllers. Neither is wrong on its own, but the work no longer converges on a stable shape you can build on.

• Context collapse: As the codebase grows, the developer using the vibe approach has to paste more and more context into the chat window. Eventually, the signal-to-noise ratio degrades. The AI becomes confused, overwriting unrelated components or losing track of the initial objective. The developer makes this worse without noticing, because LLMs are sycophantic by design and exist to satisfy the prompt. Suggest a flawed approach and a vibe-coding agent will happily generate thousands of lines executing it, never pushing back.

• No verification gate: Vibe coding runs no automated check between generation and acceptance. The developer eyeballs the output, runs it once, and moves on. Nothing catches the variance or the drift before the code reaches production, so the first two failures compound silently.

How agentic engineering solves the problem

Agentic engineering rejects the vibe approach. Instead of loose chats, it relies on strict, explicit specifications.

An agentic engineer does not write code, but they do not blindly chat with an AI either. They engineer the system that guides the AI. Rather than hand-writing the test suite, they set up an environment that makes the agent produce tests before it writes a line of implementation, so the agent is bound by verifiable constraints it cannot route around. They curate highly specific context files containing Architectural Decision Records (ADRs) and data schemas, ensuring the agent is aware of the exact boundaries it must operate within.

The distinction matters because vibe coding leads to systems you cannot maintain, while agentic engineering creates systems that scale gracefully.

Figure 1 - The three ways vibe coding breaks down, and what agentic engineering does instead

Imagine hiring fifty junior developers. Would you put them in a room, give them root access to production, and tell them to “build the vibes”? Of course not. You would establish style guides, implement code review processes, enforce CI/CD pipelines, and provide strict API contracts. Autonomous coding agents require the exact same infrastructure, scaled for computational speed.

With that distinction in place, we are ready to examine where AI delivers value and where it begins to break down.

In Part 3 of The Agentic Engineering Playbook, we'll explore one of the most common experiences every developer using AI coding tools eventually encounters: the 70% Problem. We'll look at why AI can take you from zero to a working prototype in seconds, why progress suddenly grinds to a halt, and how agentic engineering uses deterministic feedback loops to bridge the final 30% from "it works" to production-ready software.

Further reading

➡️ Where the term “vibe coding” actually broke down at scale
A blunt, cost-focused comparison from a dev shop that spends its days cleaning up vibe-coded production messes. Includes real numbers on what each approach costs in time and money, useful if you want the business case rather than just the philosophical one.

➡️ A working engineer’s version of the workflow we described
Wendell Adriel lays out his actual day-to-day process for staying in the driver’s seat while agents handle the tedious work, less theory, more “here’s what this looks like on a Tuesday.” A good companion if you want to see the ADR and spec approach applied to a real codebase.

➡️ The counterargument worth sitting with
Not everyone agrees the line is as sharp as we made it sound. Simon Willison argues the two approaches are converging faster than he’d like, as tools built for one increasingly borrow from the other. Worth reading if you want the pushback before you fully commit to the framing.

➡️ What pandas’ creator does instead of vibe coding
Wes McKinney talks through his spec-driven workflow and continuous AI code review setup, and lands on a line worth stealing: when code is free, saying no is your last line of defense. Directly relevant to the verification gate we covered above.

➡️ The ADRs and specs section, expanded
We touched on ADRs and data schemas briefly above. This guide goes much deeper into spec-driven development as a category, including how GitHub Spec Kit, Kiro, and Claude Code each implement the idea differently. Useful if you’re evaluating tools for your own team.

➡️ For teams that need governance, not just guardrails
If your org needs audit trails or multi-team coordination on top of the spec-first approach, this is the more technical version, covering the spectrum from specs that constrain code to specs that effectively become the source of truth.

➡️ Why context engineering is the real skill now
This piece makes a claim that lines up closely with what we said about context collapse: the hard part isn’t prompting anymore, it’s keeping specs and architecture notes persistently available to the agent as it works.

➡️ A concrete example of a verification gate
We described the “no verification gate” failure mode in the abstract. This is what closing that gap looks like in practice: a lightweight tool that reviews every commit before it lands, rather than trusting a human to eyeball the output once.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

Cheers!
Adrija Mitra
Co-Editor-in-Chief,
Build with AI

Over the last two editions we built the protocol layer (MCP, A2A, ACP) and the memory layer (vector stores, retrieval, the four types of memory your agent actually needs). Both editions assumed something: that once the model has the right tools and the right context, it’ll behave reliably.

It won’t. Not by default.

This week we cover the layer that sits around the model and decides whether all that careful protocol and memory work actually survives contact with production: the harness.

Social engineering is about manipulating people’s emotions. Identify the susceptibilities that hackers use to exploit people.

This NINJIO Insights Report dives into the key emotional susceptibilities that make social engineering work and offers concrete steps that your security team can take to equip your workforce to resist cyberattacks.

Download the Guide

The Mental Model - Stop tuning the model. Start engineering the harness.

For the past two years, the default move when an agent misbehaved was to swap models, rewrite the prompt, or wait for the next release. That instinct made sense when models were the limiting factor. It doesn’t anymore.

The pattern shows up across nearly every team running agents in production right now: a more capable model doesn’t fix a flaky agent. A better harness does. There’s a useful framing going around in the agent engineering community this month: classify every agent failure into one of four buckets, and you’ll find the fix is almost never “use a smarter model.”

The four failure types

• Context failures - the agent didn’t have the information it needed, or had too much of the wrong information. This is a retrieval and assembly problem, not a reasoning problem.

• Constraint failures - the agent did something it technically could do but shouldn’t have. Wrong permission scope, wrong tool for the job, no guardrail stopping it.

• Verification failures - nobody checked the output before it shipped. The agent was confident and wrong, and confidence was the only signal anyone was watching.

• Planning failures - the agent took a reasonable first step toward an unreasonable overall plan, and nothing caught the drift until five steps in.

Each of these maps to a harness component, not a model parameter. Context failures get fixed by your retrieval layer. Constraint failures get fixed by your permission model. Verification failures get fixed by adding a checker step. Planning failures get fixed by checkpointing and bounded execution, not by hoping the model plans better next time.

Four things every agent harness needs

Strip away the framework-specific jargon and a production harness is really just four components:

• An agent loop - the control flow that decides what happens next: call a tool, ask for clarification, hand off, or stop.

• A tool interface - how the agent reaches the outside world. This is your MCP layer from two editions ago.

• Context management - what the model sees on each turn. This is your memory layer from last edition.

• Control mechanisms - retries, timeouts, approval gates, rollback. The part most teams skip until something breaks in production.

Most teams have built the first three. Almost nobody has built the fourth properly, and it’s the one that determines whether your agent is a toy or infrastructure.

Why this matters right now

Two things converged this month that make this less theoretical. First, GPT-5.4 and Claude’s frontier models both shipped meaningful capability jumps in agentic reasoning. Teams that haven’t separated their harness logic from their model logic are now locked into upgrading both together, every time, with no way to isolate what actually improved. Second, OWASP’s “Excessive Agency” risk category has moved from a niche security concern to something showing up in actual vendor security reviews — over-provisioned permissions and missing approval gates are now things procurement teams ask about.

The takeaway: build your harness so it’s the thing you’d keep even if you swapped the model underneath it tomorrow. If a model upgrade requires you to rewrite your retry logic or your permission scopes, that logic was never separated from the model in the first place.

In The Build, we put this into practice with the harness pattern most teams need next: a multi-agent handoff with explicit state checkpoints, a deterministic validator that doesn’t trust the LLM’s output blindly, and a fail-safe route to a human when something doesn’t check out.

Subscribe now

The Build - A cross-language agent handoff with checkpoints and a fail-safe

Most multi-agent demos show two LLM agents talking to each other through A2A and call it done. That’s not a harness, it’s a happy path. This is the version with the parts that make it production-safe: a deterministic validator that doesn’t reason, just checks; a checkpoint that survives a crash mid-task; and an explicit route to a human when the validator isn’t confident.

Stack: a Python extraction agent (the one doing LLM reasoning), a Go validation service (deterministic, no LLM, just rules), connected via A2A, with shared state checkpoints in Postgres.

The architecture before writing a line of code

Two services, two languages, one job split cleanly between them.

• Extraction agent (Python) - receives a document, uses an LLM to extract structured fields, and sends the result to the validator via A2A. It does not decide whether its own output is correct. That’s not its job.

• Validation service (Go) - deterministic rule checks against the extracted fields. No LLM. Fast, cheap, and either passes, fails, or flags for human review. It writes its decision to the same checkpoint table the extraction agent reads.

The reason this split matters: LLMs are good at extraction and bad at grading their own work. A deterministic validator written in a few hundred lines of Go will catch malformed dates, out-of-range values, and missing required fields more reliably and more cheaply than asking the model to check itself.

Step 1 - the checkpoint table

Every task gets a row. State moves forward, never silently. If the process crashes mid-task, the next run picks up exactly where it left off instead of starting over or losing the task.

create table agent_tasks (
  id            uuid primary key default gen_random_uuid(),
  document_id   text not null,
  status        text not null default 'extracting',
  -- extracting -> validating -> approved | needs_review | failed
  extracted     jsonb,
  validation    jsonb,
  attempt_count int not null default 0,
  created_at    timestamptz default now(),
  updated_at    timestamptz default now()
);

Step 2 - the extraction agent

This is the only place an LLM touches the task. It writes its result and moves the status forward. It does not call the validator directly — it hands off via A2A and lets the validator pull the task when ready.

import json
import anthropic
from db import get_connection

client = anthropic.Anthropic()

def extract(document_id: str, document_text: str):
    conn = get_connection()
    task = conn.execute(
        "insert into agent_tasks (document_id, status) "
        "values (%s, 'extracting') returning id",
        (document_id,)
    ).fetchone()

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=500,
        system=(
            "Extract structured fields from this document. "
            "Return only valid JSON with keys: invoice_date, "
            "amount, currency, vendor_name. Do not guess "
            "missing fields, return null instead."
        ),
        messages=[{"role": "user", "content": document_text}]
    )

    extracted = json.loads(response.content[0].text)

    conn.execute(
        "update agent_tasks set extracted = %s, "
        "status = 'validating', updated_at = now() "
        "where id = %s",
        (json.dumps(extracted), task["id"])
    )

    notify_validator(task["id"])
    return task["id"]


def notify_validator(task_id: str):
    import requests
    requests.post(
        "http://validator-service:8080/tasks/send",
        json={
            "id": str(task_id),
            "message": {
                "role": "user",
                "parts": [{"type": "text", "text": str(task_id)}]
            }
        },
        timeout=5
    )

Step 3 - the validator (Go, no LLM)

This is the harness control mechanism doing its job. Deterministic checks only. Three outcomes: approved, needs_review, or failed. Nothing here calls a model.

package main

import (
    "encoding/json"
    "net/http"
    "time"
)

type Extracted struct {
    InvoiceDate string  `json:"invoice_date"`
    Amount      float64 `json:"amount"`
    Currency    string  `json:"currency"`
    VendorName  string  `json:"vendor_name"`
}

func validate(e Extracted) (status string, reasons []string) {
    if e.InvoiceDate == "" {
        reasons = append(reasons, "missing invoice_date")
    } else if _, err := time.Parse("2006-01-02", e.InvoiceDate); err != nil {
        reasons = append(reasons, "invoice_date not ISO 8601")
    }

    if e.Amount <= 0 {
        reasons = append(reasons, "amount must be positive")
    }
    if e.Amount > 1000000 {
        // outlier — not necessarily wrong, but worth a human look
        reasons = append(reasons, "amount exceeds review threshold")
    }

    if e.VendorName == "" {
        reasons = append(reasons, "missing vendor_name")
    }

    switch {
    case len(reasons) == 0:
        return "approved", nil
    case containsHardFailure(reasons):
        return "failed", reasons
    default:
        return "needs_review", reasons
    }
}

func containsHardFailure(reasons []string) bool {
    for _, r := range reasons {
        if r == "missing invoice_date" || r == "missing vendor_name" {
            return true
        }
    }
    return false
}

func taskHandler(w http.ResponseWriter, r *http.Request) {
    var payload struct {
        ID string `json:"id"`
    }
    json.NewDecoder(r.Body).Decode(&payload)

    extracted := fetchExtracted(payload.ID)
    status, reasons := validate(extracted)
    writeValidationResult(payload.ID, status, reasons)

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(map[string]any{
        "id":      payload.ID,
        "status":  status,
        "reasons": reasons,
    })
}

func main() {
    http.HandleFunc("/tasks/send", taskHandler)
    http.ListenAndServe(":8080", nil)
}

Step 4 - the fail-safe route

needs_review is the status that makes this a harness and not a happy-path demo. Tasks that land there don’t retry automatically and don’t get marked done. They go to a queue a human actually looks at.

def check_for_review_queue():
    conn = get_connection()
    pending = conn.execute(
        "select id, document_id, extracted, validation "
        "from agent_tasks where status = 'needs_review' "
        "order by created_at asc"
    ).fetchall()

    for task in pending:
        send_to_review_queue(task)
        # status stays needs_review until a human resolves it —
        # never auto-transitions to approved

What this gives you that the two-agent demo doesn’t

• A validator that can’t be talked into agreeing with a wrong extraction. It’s deterministic. It doesn’t have a context window to be persuaded in.

• A checkpoint that survives a crash. If the extraction service dies after writing extracted but before notifying the validator, the next sweep picks the task back up from validating instead of losing it.

• An explicit failure mode that doesn’t autoresolve. needs_review is a dead end until a human moves it forward. No silent retries that quietly approve something nobody checked.

This is maybe 200 lines across two languages. None of it is hard. The part that took thought is deciding where the LLM’s judgment ends and where deterministic rules and human review begin — and that boundary is the actual harness design decision, not the code.

The Radar

01. Google published an open spec for how agents find each other at runtime, the discovery layer your A2A handoffs have been missing.

Agentic Resource Discovery (ARD), backed by eleven companies including Microsoft, GitHub, Salesforce, and Hugging Face, standardizes how organizations publish a machine-readable catalog of their MCP servers, A2A agents, and APIs at a well-known URL, so agents can search for the right capability at runtime instead of having every integration pre-wired by hand. It’s an early draft, adoption is close to zero so far, but the spec is real, Apache 2.0 licensed, and GitHub already shipped a reference implementation (agent finder) the same day. If your harness currently hardcodes every tool and agent endpoint, this is the standard to watch for the next layer up.

02. New survey data: 88.4% of organizations running AI agents had a security breach in the past year, and visibility is getting worse, not better.

AvePoint’s third annual State of AI report found the share of organizations unable to detect unsanctioned agent activity nearly tripled year over year, even as 46.9% of employees now use AI agents weekly or daily. The uncomfortable detail: confidence is rising while actual monitoring coverage isn’t, organizations are getting more comfortable with a risk they haven’t reduced. If your harness’s control mechanisms layer doesn’t include agent-specific logging and access auditing yet, this is the data point to bring to whoever owns that budget conversation.

Read it here: AvePoint Research Reveals AI Visibility Gaps Have Nearly Tripled as AI Agents Scale — GlobeNewswire

03. Only about 7% of companies run fully autonomous agents in production, a useful reality check against the hype.

Despite the spending and headline numbers, a fresh practitioner analysis this week found true end-to-end automation remains rare, and that many agentic projects are at risk of cancellation specifically because teams bolt agents onto existing workflows instead of redesigning around them, without clear baselines for time, error rate, or human effort. It’s a useful gut check before your next harness investment: the gap between hype and live deployment is exactly where the discipline in this edition pays off, since the teams that survive are the ones measuring before they scale.

AI Tools of the Week

Laminar - open-source, OpenTelemetry-native observability built specifically for long-running agents rather than retrofitted from single-call LLM logging. The standout feature is its agent rollout debugger: when a 30-minute browser agent fails on tool call 1,847 of 2,000, it shows you which span actually caused it instead of forcing a manual scroll. Self-hostable, free tier at 1GB.

garak - NVIDIA’s LLM vulnerability scanner. Probes your agent for prompt injection susceptibility, jailbreaks, and data leakage before someone else finds them in production. Most teams run security scans on their infrastructure and never think to scan the model layer itself; this fills that gap and it’s free.

Logfire - observability from the Pydantic team, built Python-native rather than bolted on through a generic SDK. If your harness uses Pydantic-AI or you’re already structuring outputs with Pydantic models (which you should be, for the validation layer in last edition’s Build), this gives you tracing that understands your types instead of just logging opaque JSON blobs.

OpenLLMetry - an OpenTelemetry instrumentation library for LLM apps, not a platform in itself. The value is vendor neutrality: instrument once, then route traces into Grafana, Datadog, or whatever your team already runs, instead of marrying your harness to one observability vendor’s proprietary SDK. Easy to overlook because it doesn’t have a dashboard to demo.

agentnotary - a small, sharp tool that does one thing most harness checklists skip: cryptographically notarizes and audits agent runs, with built-in EU AI Act documentation generation and an adversarial fuzzer for testing how your agent handles malformed or hostile inputs. Worth a look given this edition’s CISA item on accountability gaps; it’s the kind of control-mechanism tooling that exists specifically because regulators started asking for audit trails.

If you made it this far, you’ve stopped asking “which model should I use” and started asking “what does my harness do when the model is wrong.” That’s the more useful question, and most teams shipping agents right now still haven’t gotten there.

You have the failure taxonomy, the four-component model, and a working pattern for the part everyone skips: the deterministic checker and the fail-safe that doesn’t autoresolve.

If this edition changed how you think about your agent stack, forward it to whoever on your team is still debugging flaky agents by swapping models. The fix probably isn’t the model.

See you next week.

Charu Mitra Dubey,

Editor-in-chief,

Build with AI

Let’s be honest: AI coding tools are a little intoxicating right now. You type a prompt, hit Enter, and suddenly there’s a React component, a REST API, or half a feature sitting in front of you as if it appeared by magic. It’s fast. It’s fun. It’s also exactly why we need to get more serious about how we build with AI.

Over the next five weeks in Build with AI, we’re bringing you The Agentic Engineering Playbook, a five-part series by Alex Merced and Benedikt Stemmildt that explores what really changes when AI moves from “helpful autocomplete” to an active participant in the software delivery process. Along the way, we’ll unpack:

➡️ Where AI genuinely shines
➡️ Where it consistently breaks down
➡️ And what it takes to build workflows that turn raw model speed into reliable, production-grade software

By the end of the series, you’ll have a practical framework for using AI as an engineering multiplier, not just a faster code generator.

We’re kicking things off with the biggest mindset shift of all. Part 1 is about why AI doesn’t make engineering discipline less important. It makes it more important. If software development’s old bottleneck was typing, AI has blown that bottleneck wide open. The question now isn’t how fast AI can write code. It’s what happens after the code is generated.

Let’s find out.

Beyond Vibe Coding

Software development is undergoing its most radical transition in decades. For most of that history, the fundamental bottleneck in engineering was the human translation of intent into syntactic execution. We possessed grand architectures in our heads, but typing them out, debugging semicolons, satisfying borrow checkers, and memorizing standard libraries was a painstaking, manual craft.

Today, that bottleneck has largely evaporated.

LLMs and autonomous agents have commoditized syntactic generation. A developer can now describe a complex state machine or a fully functional REST API in natural language, and a system will generate it in seconds. Unlike a compiler, which translates one precise notation into another and produces the same output every time, these systems predict. They vary.

However, this newfound velocity has introduced a severe vulnerability into our workflows. Early adopters of this technology have embraced a paradigm often referred to as vibe coding. Coined by Andrej Karpathy in early 2025 to describe an intent-first, syntax-second approach, vibe coding champions a fast, loose, iterative generation style.

The developer acts as a Head Chef, barking orders at algorithmic sous-chefs and letting the AI handle the mechanical slicing and dicing. It sounds liberating, and for small scripts or weekend side projects, it absolutely is. But when the goal is production-grade enterprise software, the same looseness becomes dangerous.

When you apply unstructured vibe coding to massive legacy codebases with strict security, performance, and scaling constraints, the magic quickly turns into a nightmare. AI agents are non-deterministic, bounded reasoners. Left to their own devices, they will write code that appears functionally correct but is structurally disastrous. They will tightly couple components, hallucinate dependencies, and introduce security vulnerabilities.

This leads to a fundamental thesis that runs through this entire series: agentic software engineering is the pinnacle of traditional engineering rigor, not a break from it. The disciplines that make an agent productive are the disciplines you already know: testing, version control, modular design, continuous integration, and tight feedback loops. None of this is new. What changes is who does the typing and how much leverage each decision now carries.

We need to move beyond vibe coding and integrate the rapid generative powers of AI with the structural discipline of classic software engineering. By doing so, we shift our focus from typing code to designing the systems that produce it, managing complexity, and optimizing for learning.

The central argument of this series explains why agentic engineering works. A second question is where the industry currently stands in adopting it. We can understand that progression through four eras:

• Craft: AI is autocomplete. It speeds up the individual at the keyboard, and nothing else changes. The large majority of companies still work here.

• Manufactory: The engineer steers the agent step by step and validates each step by hand. The leverage is real, but a human sits in the loop for every move. A growing minority of teams operate this way.

• Factory: The agent runs inside the loop, with deterministic orchestration between its steps, and the engineer’s job becomes designing the factory floor rather than walking it. Only a small fraction of teams have reached this. This is the era we build toward across the series.

• Compound Factory: The system improves itself each cycle, with knowledge accumulating so that every pass starts smarter than the last. The first concrete piece of this is post-merge self-healing, where the factory learns from its own production incidents.

  

Figure 1 - The four eras of agentic engineering

These eras describe how widely the practice has spread, not whether it is sound. The Pinnacle Thesis, that is, the idea that agentic software engineering is the pinnacle of traditional engineering rigor, not a break from it, answers the second question.

The Pinnacle Thesis – From modern software engineering to agentic engineering

Dave Farley, a pioneer of Continuous Delivery (CD), defined the core of the discipline in his book Modern Software Engineering. Farley stripped away the noise of agile ceremonies, specific languages, and fads, reducing engineering down to two non-negotiable foundations:

• Optimizing for learning

• Managing complexity

Any practice that supports these foundations is good engineering; any practice that hinders them is bad engineering.

Before we explore how these apply to AI, we should understand them in a purely human context.

Optimizing for learning

Optimizing for learning acknowledges that software development is fundamentally an exercise in discovery. We rarely know the exact solution before we start coding. Because we operate in an environment of high uncertainty, we need systems that let us learn quickly and safely. This involves adopting an empirical mindset, making hypotheses, running experiments, and measuring results. We implement tight feedback loops, leaning heavily on automated testing and Continuous Integration (CI) to tell us immediately if our last change broke the system. We favor incremental, evolutionary design over massive upfront planning because we learn more from a running system than a theoretical diagram.

Managing complexity

Managing complexity acknowledges that scale is the enemy of understanding. As systems grow, they exceed the limits of human working memory. To survive, we partition problems into manageable chunks. We use techniques like modularity, high cohesion, low coupling, and clear abstractions. We establish strict boundaries and interfaces so that a developer can reason about a single component without needing to understand the entire application. We isolate state and side effects. By managing complexity, we ensure the system remains safe to change.

When generative AI entered the mainstream, a dangerous false dichotomy emerged. Many assumed that since the AI could generate code instantaneously, the slow, methodical practices of engineering were obsolete. Why worry about decoupling when the AI can just refactor the whole file? Why write tests when the AI can read the entire codebase and ensure correctness?

This assumption is catastrophic in practice

AI models do not replace Farley’s foundations

The Pinnacle Thesis says that AI models do not replace Farley’s foundations; they multiply them. Let us examine why.

AI agents are phenomenal at generating raw volume, but they suffer from bounded rationality. An LLM has a finite context window and a limited ability to track deeply nested, cross-file dependencies. If you ask an agent to modify a highly coupled, monolithic ball of mud, it will almost certainly fail. The AI will lose the plot, hallucinate connections that do not exist, and introduce regressions precisely because the complexity is unmanaged.

However, if you point an AI agent at a system with aggressively managed complexity (one with high cohesion, strict modularity, and well-defined interfaces), the agent becomes a superpower. The AI only needs to hold a small, bounded context in its memory to operate successfully. Traditional engineering practices (like micro-architectures and interface segregation, keeping interfaces narrow so a client depends only on the methods it uses) were originally designed to protect human cognitive limits. Ironically, they are the exact same practices required to protect LLM context windows.

Optimizing for learning matters even more when your code generator is non-deterministic. When a human writes code, they have an internal feedback loop. They know why they made a change. An AI agent does not. It is predicting the most likely next token based on a prompt, not reasoning through your business logic.

That means your external feedback loops have to do the heavy lifting. They need to be tight and reliable. If a human introduces a bug, you might spend hours tracking it down. An AI can introduce thousands of lines of broken logic in seconds. Without strong, automated checks in place, that speed works against you. You are not moving faster. You are accumulating unverified, hard-to-change code more quickly.

This is the stake of the whole series: AI does not fix a weak engineering organization; it multiplies whatever is already there.

Traditional engineering versus agentic engineering

The following diagram shows how the role of the engineer changes when agents become part of the development process.

Figure 2 - The agentic engineering paradigm, showing AI bounded by traditional structural constraints

In traditional engineering, the developer takes an idea and turns it directly into production code. In agentic engineering, the engineer spends more time defining the goal, setting the right context, and checking the result, while the agent handles more of the code generation work. The work does not become fully automated, but the human role moves closer to direction, judgment, and quality control.

Agentic software engineering is the practice of wrapping the raw, generative power of AI models in the strict, deterministic systems of traditional engineering: the tests, type checks, and pipelines that either pass or fail. We do not throw away the foundational rules of the craft. We apply them to constrain and guide the agent.

Understanding this definition requires drawing a clear line between agentic engineering and the more ad hoc approach often called vibe coding.

In Part 2 of The Agentic Engineering Playbook, we’ll explore why vibe coding feels almost magical for prototypes but falls apart in production, and how agentic engineering uses structure, context, and verification to turn AI into a reliable engineering partner.

Further reading

➡️ Modern Software Engineering by Dave Farley is the direct source for the Pinnacle Thesis framing in this post. Farley distills the discipline into two core exercises: learning and exploration, and managing complexity, and defines principles that improve everything from your mindset to the quality of your code. Essential background if the ideas here resonated.

➡️ Andrej Karpathy’s original “vibe coding” post (X, February 2, 2025) is the artifact that started the conversation. In the post, Karpathy described a new kind of coding where you “fully give in to the vibes, embrace exponentials, and forget that the code even exists,” made possible by how capable LLMs had become. Worth reading in its original form before reading any commentary about it.

➡️ Karpathy’s Sequoia Ascent 2026 summary is Karpathy’s own cleaned-up write-up of his April 2026 fireside chat with Sequoia Capital, where he formally introduced agentic engineering as what comes after vibe coding. He describes agentic engineering as the professional discipline of coordinating fallible agents while preserving correctness, security, taste, and maintainability and draws a sharp line between that and casual prototyping. The best single primary source on the concept this series builds on.

➡️ A Semantic History of Vibe Coding (CodeRabbit, March 2026) traces how the term evolved from a throwaway tweet into an industry flashpoint. The piece argues that as the phrase broadened into shorthand for any prompt-driven development, it became a signal that many engineering teams had added AI to their stacks and were struggling with the quality of its output and its downstream effects. Good context for why the distinction between vibe coding and agentic engineering matters in practice.

And that’s a wrap 🎬

We’re glad you joined us for this edition of Build with AI!

If you have any thoughts, questions, or feedback on this edition, or if you’d like to share what you’d love to see next, feel free to take our 1-minute survey. We’d love to hear from you.

Thanks for following along! Until next time, keep learning and keep building.

In our last edition we covered context windows and memory architecture — why stuffing transcripts into context is a design mistake, and how to build a proper retrieval layer instead. The response made one thing clear: developers are hungry for the mental model fix, not just the implementation.

This week, we fix another one.

If you’ve been building with AI agents in 2026, you’ve seen three acronyms everywhere: MCP, A2A, ACP. You’ve probably read that they’re “complementary.” You’ve probably also spent an afternoon in spec docs trying to figure out what that actually means for your architecture.

Here’s what most of that content doesn’t say: they solve completely different problems at completely different layers of your stack. Treating them as alternatives — or worse, picking one and ignoring the others — is how teams end up with agents that can use tools but can’t talk to each other, or agents that coordinate beautifully in isolation and fall apart the moment a third-party system is involved.

In this edition:

• The three-layer protocol stack — what MCP, A2A, and ACP each do, where they live, and why the “which one should I use” question is usually the wrong question

• How to implement the two-layer stack most production teams actually need right now — MCP for tools, A2A for agent coordination

• What’s worth paying attention to in AI this week

The Mental Model — Your agent stack now has a protocol layer. Here’s how it’s structured.

Six months ago, MCP was the only protocol most AI engineers needed to know. That’s no longer true.

The agent ecosystem has matured fast enough that three distinct protocols now exist — each solving a different problem at a different layer of your stack. The confusion isn’t that developers don’t understand the protocols. It’s that they’re comparing things that aren’t in competition.

Here’s the one-line version of each:

• MCP — how your agent talks to tools

• A2A — how agents talk to each other

• ACP — how agents message each other when you want REST and nothing else

Conflating any two of these is like asking whether HTTP or SQL is better. They operate at different layers. You don’t choose between them — you choose which combination your architecture needs.

MCP: the tool layer

Model Context Protocol, launched by Anthropic in late 2024 and now governed by the Linux Foundation, is the foundational layer. It gives agents structured, authenticated access to external tools — databases, APIs, file systems, third-party services. Think of it as the USB-C standard for AI: one protocol, every tool.

MCP is why your agent can query a database, send a Slack message, and read a PDF without you writing bespoke integration code for each. It’s the layer every team needs first, and for most single-agent applications, it’s the only layer they’ll ever need.

The numbers reflect this. MCP hit 97 million monthly SDK downloads and 10,000+ deployed servers by its first anniversary. OpenAI, Google, Microsoft, and AWS all adopted it. It is, at this point, infrastructure.

A2A: the coordination layer

Agent-to-Agent protocol, created by Google and donated to the Linux Foundation in June 2025, solves a different problem entirely. Not how an agent uses tools — but how agents hand off tasks to each other across systems, vendors, and organisational boundaries.

A2A hit v1.0 in April 2026. That matters because it’s no longer a directional bet — it’s a real implementation target. The protocol defines how agents discover each other, negotiate capabilities, exchange messages, and track task state across a handoff. Think of it as HTTP for agent coordination: a universal communication layer that doesn’t care what framework or vendor built the agent on either end.

When do you need it? When your architecture has multiple autonomous agents that need to delegate to each other — especially across vendor or service boundaries. A single agent connected to ten MCP servers doesn’t need A2A. An orchestrator agent handing off a compliance check to a specialist agent built by a different team on a different framework does.

ACP: the lightweight alternative

Agent Communication Protocol, developed by IBM and now under the Linux Foundation, covers similar ground to A2A but with a different design philosophy. Where A2A uses JSON-RPC 2.0 over HTTP and SSE, ACP is REST-native — it uses patterns most backend developers already know. No new transport to learn, no SSE to manage.

ACP is the right choice when your team wants inter-agent messaging without the overhead of adopting a new protocol stack. It’s particularly well-suited to controlled environments where all agents are within your own infrastructure. For cross-vendor or cross-organisation coordination, A2A’s richer capability negotiation is worth the added complexity.

The decision framework

You don’t need all three. Most teams need one or two, depending on where they are in their agent architecture:

• Single agent, multiple tools → MCP only. This is the majority of teams right now.

• Multiple agents, same infrastructure → MCP + ACP. Lightweight coordination without protocol overhead.

• Multiple agents, cross-vendor or cross-org → MCP + A2A. The two-layer stack that enterprise teams are converging on.

• All three → Only when you have internal agent messaging (ACP), cross-vendor coordination (A2A), and tool access (MCP) as distinct requirements. Rare, and usually premature.

The shift isn’t complicated. But it requires treating the protocol layer as an architectural decision you make deliberately — not something you bolt on when your single-agent system starts failing at scale.

The Build — Implementing the two-layer stack: MCP for tools, A2A for agent handoff

Most teams don’t need all three protocols. They need two: MCP to give agents access to tools, and A2A to let agents hand off tasks to each other. This is the stack we’re building — an orchestrator agent that uses MCP to query a database, and delegates a specialist task to a second agent via A2A.

Stack: Node.js, the MCP SDK, and the A2A client library. The pattern is framework-agnostic — the same architecture works with LangChain, LlamaIndex, or a bare OpenAI client.

The architecture before writing a line of code

Two agents. One job each.

• Orchestrator agent — receives the user’s request, uses MCP to access tools (database, APIs, file system), decides when a task is outside its own capability, and delegates via A2A.

• Specialist agent — exposes an A2A-compliant endpoint, receives delegated tasks, executes them, and returns structured results to the orchestrator.

The handoff is the critical piece. The orchestrator doesn’t call the specialist directly — it sends an A2A task, waits for a response, and incorporates the result into its own context. The specialist has no knowledge of the orchestrator. They’re decoupled by design.

Step 1 — set up MCP tool access for the orchestrator

Install the MCP SDK and define your tool server. This example connects the orchestrator to a database tool.

import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

async function createMCPClient() {
  const transport = new StdioClientTransport({
    command: 'node',
    args: ['./tools/database-server.js']
  });

  const client = new Client(
    { name: 'orchestrator', version: '1.0.0' },
    { capabilities: { tools: {} } }
  );

  await client.connect(transport);
  return client;
}

async function queryDatabase(mcpClient, query) {
  const result = await mcpClient.callTool({
    name: 'query_database',
    arguments: { sql: query }
  });
  return result.content[0].text;
}

Your MCP tool server defines what the orchestrator can access. Keep each server focused — one server per domain (database, file system, external APIs). Don’t build a monolithic MCP server that does everything. The same principle applies here as with microservices: small, bounded, replaceable.

Step 2 — build the specialist agent’s A2A endpoint

The specialist exposes two endpoints: an agent card (capability discovery) and a task handler. A2A requires both.

import express from 'express';
import Anthropic from '@anthropic-ai/sdk';

const app = express();
app.use(express.json());

const anthropic = new Anthropic();

// Agent card — tells the orchestrator what this agent can do
app.get('/.well-known/agent.json', (req, res) => {
  res.json({
    name: 'compliance-specialist',
    version: '1.0.0',
    description: 'Runs compliance checks on structured data',
    capabilities: {
      streaming: false,
      pushNotifications: false
    },
    skills: [
      {
        id: 'compliance_check',
        name: 'Compliance check',
        description: 'Validates data against regulatory requirements',
        inputModes: ['application/json'],
        outputModes: ['application/json']
      }
    ]
  });
});

// Task handler — receives delegated tasks from the orchestrator
app.post('/tasks/send', async (req, res) => {
  const { id, message } = req.body;

  const userMessage = message.parts
    .filter(p => p.type === 'text')
    .map(p => p.text)
    .join('\n');

  const response = await anthropic.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 1000,
    system: `You are a compliance specialist. 
Analyse the provided data and return a structured 
compliance report. Be specific about violations 
and recommendations.`,
    messages: [{ role: 'user', content: userMessage }]
  });

  res.json({
    id,
    status: { state: 'completed' },
    artifacts: [{
      name: 'compliance-report',
      parts: [{
        type: 'text',
        text: response.content[0].text
      }]
    }]
  });
});

app.listen(3001, () => {
  console.log('Specialist agent running on port 3001');
});

Step 3 — build the orchestrator’s A2A client

The orchestrator decides when to delegate. The decision logic is straightforward: if the task requires specialist capability, send it via A2A. Everything else, handle locally with MCP tools.

import { v4 as uuidv4 } from 'uuid';

async function delegateToSpecialist(specialistUrl, taskData) {
  const taskId = uuidv4();

  const response = await fetch(`${specialistUrl}/tasks/send`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      id: taskId,
      message: {
        role: 'user',
        parts: [{ type: 'text', text: JSON.stringify(taskData) }]
      }
    })
  });

  const result = await response.json();

  if (result.status.state !== 'completed') {
    throw new Error(`Task ${taskId} failed: ${result.status.state}`);
  }

  return result.artifacts[0].parts[0].text;
}

async function orchestrate(userRequest, mcpClient) {
  // Step 1: use MCP to fetch the data needed
  const data = await queryDatabase(
    mcpClient,
    `SELECT * FROM transactions WHERE date > NOW() - INTERVAL '30 days'`
  );

  // Step 2: decide whether to delegate
  const needsComplianceCheck = userRequest
    .toLowerCase()
    .includes('compliance');

  if (needsComplianceCheck) {
    // Step 3: delegate via A2A
    const report = await delegateToSpecialist(
      'http://localhost:3001',
      { request: userRequest, data: JSON.parse(data) }
    );
    return report;
  }

  // Handle locally if no specialist needed
  return data;
}

Step 4 — wire it together

async function main() {
  const mcpClient = await createMCPClient();

  const userRequest =
    'Run a compliance check on last month\'s transactions';

  const result = await orchestrate(userRequest, mcpClient);

  console.log('Result:', result);

  await mcpClient.close();
}

main().catch(console.error);

What this architecture gives you

Three things that matter in production:

• Decoupled specialists. The compliance agent doesn’t know who called it. You can swap it out, version it independently, or replace it with a third-party A2A-compliant service without touching the orchestrator.

• Clean capability boundaries. MCP handles tool access (what the agent can read and write). A2A handles task delegation (what the agent can ask another agent to do). Mixing these responsibilities into one layer is how architectures become unmaintainable.

• Protocol-level interoperability. Because A2A is an open standard, your specialist agent can receive tasks from any A2A-compliant orchestrator — not just yours. That’s the difference between integration and infrastructure.

The decision your architecture needs to make explicitly

One question that comes up when building this: how does the orchestrator decide when to delegate versus handle locally?

Don’t use an LLM call for this decision — it’s slow and expensive for a routing problem. Use deterministic rules first: keyword matching, task type classification, input schema validation. Only reach for an LLM-based router if your routing logic is genuinely complex and rules-based approaches fail. Most teams over-engineer the routing layer and under-engineer the handoff contract.

The full two-layer stack — MCP tool access, A2A task delegation, deterministic routing — is around 120 lines of working code. The architecture decisions above are harder than the code. Get the boundaries right and the implementation follows.

The Radar

01. A CVSS 9.8 vulnerability just dropped in an MCP integration — and it changes how you think about tool access.

MCP is infrastructure now. And infrastructure gets attacked. A critical vulnerability was disclosed in May 2026 in an MCP integration — the kind of severity score that means remote code execution is on the table. The practical implication for teams building on MCP: your tool servers are now an attack surface, and the default assumption that MCP integrations are internal and therefore safe is wrong. Two things to do now: audit every MCP server you’re running for authentication gaps, and treat your tool server permissions the same way you’d treat database credentials — least privilege, scoped access, rotated regularly. The protocol is solid. The implementations are where the risk lives.

02. A2A hit v1.0 in April — which means agent-to-agent handoffs are now a real implementation target, not a directional bet.

For the past year, A2A has been something teams watched but didn’t build on. The 0.3 preview spec was unstable enough that production commitments felt premature. That changed in April 2026. V1.0 means a stable spec, committed SDK support, and — most importantly — enterprise vendors treating it as infrastructure. SAP has integrated A2A into its Joule AI assistant, which runs ERP systems for the majority of the Global Fortune 500. When SAP commits to a protocol, it becomes the de facto standard for enterprise agent coordination across finance, supply chain, and procurement. If you’ve been waiting for A2A to stabilise before building on it, the wait is over.

03. Half of all enterprise agents still can’t talk to each other — and that’s now a competitive gap, not just a technical debt.

The average enterprise runs 12 AI agents. Half of them operate in complete isolation. That number comes from Salesforce’s 2026 Connectivity Benchmark Report, and it reframes the protocol conversation entirely. The teams asking “should we implement A2A?” are already behind. The right question is “how quickly can we retrofit coordination into our existing agent architecture?” The answer, for most teams, is: faster than you think, if you start with the two-layer stack from this edition’s Build section and add A2A to the agents that actually need to delegate tasks. Don’t boil the ocean. Pick one handoff that matters and make it protocol-compliant first.

AI Tools of the Week

MCP Inspector: Debug your MCP servers visually

The tool you wish existed the first time an MCP server silently failed. MCP Inspector is an open-source visual debugger that lets you connect to any MCP server, browse its available tools, fire test calls, and inspect the raw request/response cycle — all without writing a line of code. Think of it as Postman for MCP. If you’re building the two-layer stack from this edition, this is the first thing to install before you write a single tool server. Catching a malformed tool schema in Inspector takes thirty seconds. Catching it in production takes thirty minutes and a lot of logs.

Agentboard: Observability for multi-agent systems

The missing monitoring layer for teams running more than one agent in production. Agentboard traces task flows across agents, visualises A2A handoffs, and surfaces latency and failure points in multi-step pipelines. The dashboard shows you which agents are doing the most work, where tasks are stalling, and which handoffs are failing silently — the exact failure modes that are invisible when you’re logging individual agents but not the coordination layer between them. Free tier covers up to 5 agents and 10,000 traces per month. Worth connecting before your agent count grows past the point where mental models stop working.

Zep: Temporal knowledge graph for agent memory

Worth calling out again in this context: as your agent architecture grows from one agent to many, memory becomes a coordination problem, not just a storage problem. Which agent remembered what? Which facts are still valid? Zep’s Graphiti engine stores fact validity windows rather than timestamped snapshots — it knows that a fact was true until a certain point, not just that it was stated at a certain point. In a multi-agent system where different agents update the same knowledge base at different times, that distinction prevents an entire class of stale-context bugs. If the memory edition two weeks ago convinced you to build a retrieval layer, Zep is the right tool for the coordination layer that comes next.

Betterclaw: Managed agent infrastructure with MCP and A2A built in

The managed option for teams that want the two-layer stack without running the infrastructure themselves. Betterclaw handles MCP server provisioning, A2A endpoint routing, agent identity, and capability discovery out of the box — the four things that take the most setup time when you build the stack from scratch. Verified skills, encrypted secrets, and a free-forever tier make it a reasonable starting point for smaller teams validating the architecture before committing to a self-hosted setup. If the Build section felt like a lot of moving parts, start here and work backwards to understand what you eventually want to own.

If you made it this far, your agent architecture now has a protocol layer — and you understand why it exists.

Most teams shipping agents in 2026 are still treating MCP as the whole story. They’ll hit the coordination ceiling eventually: agents that can use tools but can’t talk to each other, architectures that scale to three agents and then become unmaintainable, integrations that break every time a third-party vendor changes their API. You just opted out of that trajectory.

You have the mental model — three protocols, three layers, three different jobs. You have the implementation — MCP for tool access, A2A for task delegation, deterministic routing between them. And you have the decision framework for knowing which combination your architecture actually needs, today, without over-engineering.

If this edition changed how you think about agent interoperability — forward it to the engineer on your team who’s still asking “MCP or A2A?” They’re asking the wrong question. Now you can tell them why.

See you next week.

Charu Mitra Dubey
Editor-in-chief, Build with AI

Sampling breaks that model — and in doing so, opens up a class of use cases that the standard request-response flow simply can’t handle.

Here’s the core idea: sometimes a server needs to complete a task that it isn’t equipped to do on its own. Generating a compelling product description. Summarizing a blog post draft. Giving voice to a game character. These are generative tasks — and generative tasks need an LLM. The server doesn’t have one. But the client does.

So the server asks. That ask is called a sampling request.

What sampling is — and what problem it actually solves

The word comes from the dictionary definition: “the action or process of taking samples of something for analysis.” In MCP, the server sends a sample — a prompt, a piece of content, a task — to the client for analysis. The client’s LLM processes it and sends back a response. The server then uses that response to complete whatever it was working on.

This is delegation, not dependency. The server stays in control of the overall task. It just hands off the parts that require generative intelligence — the parts it genuinely can’t do as well on its own.

The pattern is more common than it might first appear. Any time a system needs to produce language — descriptions, summaries, tags, dialogue — and that language needs to be good, there’s a case for sampling.

The four participants

A sampling interaction involves four distinct roles, and it’s worth keeping them clear:

The user is involved at two points: as the person who triggered the original action, and as the human in the loop who can review or modify the sampling request before it goes to the LLM. This second touchpoint is important — it’s what keeps sampling human-supervised rather than fully automated.

The server sends the sampling request. This happens from within a server feature — typically a tool call, a resource read, or a prompt template. The server is the one that recognizes it needs LLM help and initiates the delegation.

The client receives the sampling request and passes it to the user for review. The server can suggest a specific model, token limit, and system prompt — but these are recommendations, not instructions. The client and the user have the final say.

The LLM on the client side does the actual work. It takes the prompt from the server, runs its generative process, and returns a response that the client sends back.

Author: Christoffer Noring

Three scenarios where this matters

Writing a blog post. A user submits a draft to the server. The server stores it, but realizes it needs tags and a summary abstract — the kind of work an LLM does better than a rules-based function. A sampling request goes out with the draft as context. The client’s LLM analyzes it, generates the metadata, and sends it back. The server updates the post and returns a complete result.

Back office e-commerce. An admin adds a new product: name, keywords, a few properties. Writing the description is time-consuming and easy to get wrong. The server uses those keywords as context in a sampling request and asks the client to generate something compelling. The result gets attached to the product automatically. No copy-paste, no separate tool.

NPC dialogue in a game. Non-player characters in most games are limited by pre-scripted responses. With sampling, a server can retrieve a character’s backstory, personality, and motivations, and send all of that as a sampling request. The client’s LLM produces a contextually appropriate response — one that feels alive rather than canned. The same character can have a different conversation every time.

What’s actually inside a sampling request

When the server sends a sampling request, it isn’t just passing a prompt into the void. The request is structured, and the structure gives you real control. Here’s what the key fields look like:

{

  “method”: “sampling/createMessage”,

  “params”: {

    “messages”: [

      {

        “role”: “user”,

        “content”: {

          “type”: “text”,

          “text”: “Write a compelling description of this product: tomato, here’s some keywords: red, vegetable, fresh”

        }

      }

    ],

    “modelPreferences”: {

      “hints”: [{ “name”: “claude-3-sonnet” }],

      “intelligencePriority”: 0.8,

      “speedPriority”: 0.5

    },

    “systemPrompt”: “You’re a professional writing assistant who writes descriptions in a poetic way”,

    “maxTokens”: 100

  }

}

A few things worth noting here. The modelPreferences field lets the server suggest a model and express priorities around intelligence versus speed — but the client doesn’t have to honor it. The systemPrompt shapes how the LLM responds, which means the server can give the LLM a persona or a constraint specific to the task. And maxTokens keeps output scoped to what the task actually requires.

Building it: server side and client side

On the server, the pattern is straightforward. Inside a tool, after the initial processing, you call ctx.session.create_message() with a SamplingMessage containing your prompt. The call is await — the server pauses and waits for the client to come back with a response before continuing.

result = await ctx.session.create_message(
    messages=[
        SamplingMessage(
            role="user",
            content=TextContent(type="text", text=prompt),
        )
    ],
    max_tokens=100,
)
product["description"] = result.content.text

The server doesn’t know or care which LLM the client uses. It just sends the request and handles the result.

On the client, two things need to happen. First, you declare sampling support when initializing the session:

{ "capabilities": { "sampling": {} } }

Second, you register a callback that handles incoming sampling requests:

async with ClientSession(read, write,
    sampling_callback=handle_sampling_message) as session:

The callback receives the sampling request, extracts the prompt, calls the LLM, and returns a structured response. The client controls which LLM is used — it could be GPT-4o, Claude, a local model, anything. The server has no say in the implementation detail, only in the task description.

Here’s what the handler looks like in practice:

async def handle_sampling_message(context, params):
    message = params.messages[0].content.text
    response = await call_llm(message, "You're a helpful assistant. Create a compelling product description.")
    return types.CreateMessageResult(
        role="assistant",
        content=types.TextContent(type="text", text=response),
        model="gpt-4o-mini",
        stopReason="endTurn",
    )

When this runs end-to-end — a user prompts “create product called tomato with keywords red and vegetable and delicious” — the server calls the tool, issues the sampling request, and comes back with something like:

“Introducing our Red Garden Medley — a vibrant selection of the freshest, most delicious red vegetables nature has to offer! Each hand-picked assortment features juicy tomatoes, crisp red bell peppers, and sweet red radishes, bursting with flavor and color.”

That’s not a template. That’s a generative response, produced on demand, shaped by the keywords the admin entered.

The human-in-the-loop piece that most explanations skip

Sampling isn’t fire-and-forget. The design of the protocol explicitly includes a point where the user can inspect the sampling request before it runs — reviewing the prompt, adjusting the model preference, modifying token limits. This is intentional.

The server’s sampling request is a recommendation, not a command. The client and the user are always in control of what actually gets sent to the LLM. That framing matters for anyone thinking about production use cases: sampling keeps humans in the decision loop by design, not as an afterthought.

This is the same principle that governs tool approval in hosts like VS Code. Every step where an AI system is about to do something consequential gets a checkpoint. Sampling applies that same logic to generative sub-tasks.

The deeper pattern here is something that shows up across well-designed AI systems: knowing where to delegate. A server that tries to do everything itself will either be limited or brittle. A server that understands what it’s good at — structured data, persistence, tool orchestration — and knows when to hand off to generative intelligence will be both more capable and more maintainable.

Sampling is how MCP formalizes that handoff.

Why the SDLC still matters in an AI-assisted workflow

The software development lifecycle has six stages. Planning, design, implementation, testing, deployment, maintenance. You know them.

For decades, the bottleneck was implementation. Writing the code was the slow, expensive part. Everything else — planning, design, testing — existed to protect the code from going wrong.

Now that AI can write code faster than most engineers, the bottleneck has shifted. But the stages haven’t disappeared. What’s changed is where the leverage is — and where the new failure modes are hiding.

Here’s what that looks like across each stage.

Planning: the stage AI can’t replace

This one doesn’t change much. And that’s the point.

Before AI, bad planning led to slow, expensive rewrites. With AI, bad planning leads to fast, cheap, wrong builds that still need expensive rewrites.

The speed of AI implementation doesn’t make upfront planning less valuable. It makes it more valuable. You can now spiral into the wrong thing at 10x the velocity.

What good vibe coding looks like here: use an LLM as a consultation partner before you write a single prompt to your IDE. Ask it to poke holes in your idea. Ask it what you haven’t considered. Treat it like a senior engineer in the room — not a code dispenser.

The planning stage is where you go from idea to requirements. Who is this for? What do they need? What does the project actually involve, scoped into concrete tasks? Don’t underestimate it. It’s the foundation for everything that follows.

Design: the decisions that still belong to you

Architecture decisions still belong to you.

Which tech stack? How does the data model look? What APIs are you relying on? Where do you draw the service boundaries? What does the UI flow look like?

AI will happily make all of these decisions for you if you let it. And it’ll make them confidently — and incorrectly for your specific context.

The vibe coding trap here is mistaking speed for correctness. A poorly architected app built in 30 minutes is still a poorly architected app. The AI didn’t know about your team’s constraints, your infra costs, or the edge cases in your domain. You do.

The practical move: bring your architecture questions to an LLM before you open your AI IDE. Use it to stress-test decisions, not make them. Your codebase will thank you six weeks from now.

Implementation: where AI actually earns its keep

This is where AI earns its keep. Seriously.

Code generation has gotten genuinely good. If you have a clear specification and a well-scoped task, a tool like Cursor or Windsurf can implement entire features in the time it would take you to write the function signature.

Two things still matter though.

One: the specification has to exist first. Zero-shot prompts (”build me a Kanban board”) produce zero-shot quality. The AI isn’t bad. The instruction was bad. Invest time upfront in a proper spec document — what the app should look like, how it should behave, what technologies to use. Then break it into a phased TODO list and work incrementally. One feature at a time.

Two: you still have to read the code. AI is non-deterministic. The same prompt produces different outputs on different runs. Treating generated code as ground truth without reviewing it is how bugs get shipped quietly — and in an AI-assisted codebase, the blast radius of a broken change is larger because the AI has touched every file.

Testing: the part most vibe coders skip

Don’t skip it.

Automated tests aren’t bureaucracy. They’re the thing that lets you keep moving fast after the first 20% of the build. Every feature you add without a corresponding test is debt you’ll pay in debugging time later.

The good news: AI is useful for writing tests too. Ask your tool of choice to write end-to-end tests for every feature you complete. Make it a rule, not an afterthought.

Some tools can be configured to run tests automatically after every change so you never have to remember to ask. In Cursor, for example, you can set a project rule — “after making changes to the codebase, run all tests to ensure they pass” — and it’ll happen without you prompting it each time. That’s not a nice-to-have. That’s how you stay confident as the codebase grows.

The types worth knowing: unit tests cover individual functions, integration tests check how components work together, and end-to-end tests simulate real user interactions. An AI-assisted project should have all three. If you’re building fast, start with E2E tests for your critical paths.

Deployment and maintenance: what changes, what doesn’t

Deployment

The pipeline hasn’t changed structurally. Staging environments, version control, production releases — the process is the same.

What’s changed: AI can now walk you through the entire deployment process conversationally, including catching build errors and suggesting fixes in real time. The friction that used to live here — figuring out how to configure Vercel, what the build flags need to be, what the CI error means — is largely gone. It’s now a conversation, not a documentation spelunk.

Maintenance and iteration

This is the stage that breaks most AI-assisted projects.

Bug reports come in. Features need updating. Dependencies get flagged. And now you’re back in the codebase — except you didn’t write most of it.

The engineers who do this well are the ones who maintained a specification document throughout the build. They kept their TODO list updated. They reviewed every line the AI generated. They committed frequently to version control so rolling back to a working state is one prompt away.

Vibe coding done right is still disciplined coding. The discipline just lives in different places now — in the spec, the commit history, the test suite, the rules file. Move those things and the maintenance phase becomes painful regardless of how fast the build went.

How to pick a tech stack when AI lowers every barrier

Pick the stack you know, not the stack that’s trending.

This sounds obvious. It isn’t practiced.

AI tools are seductive here. They’ll happily scaffold a project in any language or framework you name. That doesn’t mean you should reach for a new stack just because the barrier to entry is lower now.

The engineers who ship fastest with AI-assisted tools are the ones using stacks they already understand. They can catch the AI’s mistakes. They know what “correct” looks like. They can debug when the AI confidently produces something broken.

If you’re a JavaScript/TypeScript person, stay there. Full-stack with React on the frontend and Node on the backend is still one of the most productive setups for AI-assisted development — one language end-to-end, massive community, tons of tooling. If you’re a Python person, your backend instincts plus AI assistance is a strong combination, and Python’s dominance in AI/ML means the ecosystem keeps getting better.

New stack, new language, AI-assisted build, tight deadline — that’s four variables you don’t want active at the same time.

The one-line summary

Vibe coding shifts the bottleneck from writing code to thinking clearly about what to build.

The SDLC isn’t a relic. It’s the foundation that makes AI-assisted development actually work. Skip the stages you think AI has replaced and you’ll find out exactly why they existed.

Get the book. On us.

We’re giving away a free copy of Vibe Coding with Cursor, Windsurf, and Lovable by Greg Lim to every new subscriber of BuildWithAI.

Subscribe below and we’ll send it your way.