AI assistance note: Hate reading AI slop? This article was rated 94% human written by Substack and Pangram. We used some AI assistance to structure our thoughts, but the ideas and opinions are entirely ours. Rest assured, you are not wasting your time reading slop.

In a previous article, Loop Engineering: New Idea or Old Automation, we talked about the origins of loop engineering and the key components that define a loop. We also looked at the LangChain stack for agent loops and discussed the risks that come with autonomous loops.

Inspired by Nate Herk’s tutorial, in this article, we look at building an agent loop for a website that teaches people about agent loops. We define the goal, final outcome, review criteria and guardrails and see how the first version compares with the final version produced by the loop.

Step 1: Choose a task that can survive a loop

Before we get into defining the loop, let’s talk about how you can understand if it makes sense for your use-case to have a loop.

You can start by asking yourself two questions:

1. Is the work such that it will require multiple attempts to get right?

2. Can I define clear criteria that the agent can use to check its work and determine how close each attempt is to ‘done’?

In our example of building a website, the answer to both these questions is a yes because building a good guide for beginners often takes multiple attempts. Once you create an initial version of a website, you often need to review for things like the content to make sure it is easy to understand and evaluate whether something should be presented differently, you have to make sure the navigation works and the overall structure and information architecture is easy to follow. The agent could render the page, inspect it, compare it with the brief, and revise the files.

Step 2: Write the goal and define the outcome

The goal we define here is building a self-contained website that can explain agent loops to a non-technical reader.

You also define what ‘done’ looks like.

Prompt

The more clear the brief, the less the agent will assume. Here “good design” is still subjective and a stronger version could provide additional details like a site to reference, layout requirements, accessibility checks, some visual rubrics etc.

Step 3: Define separate agents for building and reviewing

Since judgment is needed for both building and reviewing the work, we assign two separate agent roles:

1. A builder that is responsible for creating the website by following the guidelines as mentioned in step 2.

2. A reviewer that compares the output with the original guidelines and determines where the criteria was not met. It does not correct anything by itself and only provides input to the builder for its next run of the loop.

prompt continued

Step 4: Give the loop memory

Since each run builds on the previous one, the loop needs a record so it can learn and not repeat the same mistakes again. In order to do this we maintain two logs:

1. A research log records the sources, the claims taken from them, and where those claims appeared in the site.

research.md log file produced during the loop

2. A change log records what changed between versions and why.

changelog.md log file produced during the loop

Step 5: Define the guardrails

It is important to set a maximum on the number of times the loop can run so that it does not keep running and use up all your tokens. In our example, we give it a maximum of five passes.

prompt continued

The maximum does not mean that it will always do that many passes, if it meets the criteria earlier it will stop as in our case it stopped at 4 runs. Similarly tool access can be another guardrail you can add.

Outcomes of the loop

If we compare the very first version with the last version that we got, we see some key differences such as information being presented more visually, better section breaks, more examples, improved navigation and addition of animations.

The differences that happened after each run can also be seen in the logs so you can understand better what was iterated on.

Build your own agent loop

You can adapt the template we used for your own use case and experiment with building your own agent loop.

This article grew out of a live Chai & AI session conducted by Akshat Kharbanda where we got into loop engineering and saw a live demo of an agent loop. All our Maven cohort members get access to our Chai and AI community, check out our cohort here.

Part of this is because most content out there treats these as disconnected and un-related concepts but in reality they all build on top of each other.

In this video, I discuss all these shifts and breakdown:

1. What each term means and why it matters

2. How each practice can be used in real examples like a system that monitors a product price and one that writes an article

A quick visual of everything we discuss:

I hope the next time a new AI engineering term gets introduced, this gives you a better way to assess what that means and think about what problems it helps address that the previous one didn’t.

If you do enjoy this style of short but knowledge-dense videos, do check out the previous videos on my channel as well:

• AI Engineering Explained in 17 mins | The Ultimate Beginner’s Guide

• Is RAG Really Dead in 2026?

• Are Agents Running My Entire Company Stories True?

In Part 1: The Model is Not the Product, we argued that the model is only one part of the agent and the harness is the real product. The harness is the software which surrounds the model and decides what information reaches the model, which tools it can call, how those actions run, what gets saved and what the user sees.

That argument is easy to make but hard to prove because most agent architectures are proprietary and therefore invisible. In this article we look at Tau, which is an open-source project from Hugging Face, intentionally designed as a readable Python coding agent. Since it’s meant to educate, its main components can each be seen separately and studied.

We will go through the code in this order:

• The reusable core: How Tau separates its small reusable agent core from the larger coding product built around it.

• The provider event stream: How adapters convert the different request and response formats used by OpenAI, Anthropic, and Gemini into one format the agent loop can use.

• The agent loop and its rules: How the loop manages a turn and where the surrounding product can inspect or block a tool call.

• Tools: How the model is connected to external actions such as reading files, editing code and running shell commands.

• The session log and model context: How Tau preserves a durable record while sending only the relevant working context back to the model.

• Project instructions: How instructions retain their source and are loaded according to the agent’s capabilities.

• Product assembly and extensions: How the reusable core is connected to the coding environment, and how trusted extensions add custom behavior without changing the core loop.

• The user interface: How agent events are converted into the state the user sees.

By the end of this, you will understand that:

A coding agent is not just a model with bash access. It is a set of explicit boundaries between provider, loop, harness, tools, state, project context, and user interface.

In Tau, these boundaries are visible as code and we start by looking at the number which makes this evident.

The reusable core is 5% of Tau

Tau contains about 40K lines of Python across three packages. The portable agent brain which contains the pieces you could reuse to build a different agent is only 5% of that at 1,764 lines.

The other 95% handles two jobs: translating between Tau and different model providers, and turning the reusable agent loop into a coding product with files, shell access, saved sessions, project instructions, extensions, trust checks, and a terminal interface.

As we saw in part 1, this ratio reinforces that the hard part is not the agent loop but everything that decides what the loop sees and what it is allowed to touch. An important note to make is that, while Tau uses AgentHarness for a reusable stateful core inside tau_agent, this article uses harness more broadly for the complete control structure around the model.

If you search the tau_agent package for imports from tau_coding, you will not find any. That is by design.

The brain cannot reach the face.

One event stream for every model provider

Anthropic, Gemini, and OpenAI use different request formats, tool-call formats, errors, and streaming protocols. Tau keeps those differences inside provider adapters.

The contract lives in tau_agent/provider.py:

tau_agent/provider.py

Each adapter converts its provider’s raw response into Tau’s typed event stream which means that every event is labeled and structured, so the rest of the system knows what kind of information it contains.

A simplified sequence might look like this:

Other events carry model thinking or errors. Because the events are labeled separately, an error is not mistaken for ordinary text and a tool request can be executed instead of merely displayed. The loop and UI can therefore handle each event type without knowing whether it came from Anthropic, Gemini, or OpenAI.

Because provider-specific logic stays inside tau_ai, Tau can add new model providers or adapt to API changes without adding provider-specific code to the reusable tau_agent core.

The agent loop and the rules around it

run_agent_loop is the central agent loop which lives in tau_agent/loop.py. It receives the model provider, system prompt, current conversation history and available tools. Its job is to manage one agent run.

A run begins with a user request and may contain several turns. In Tau, each turn consists of one model response and any tool results produced by that response. If the model requests a tool, the loop executes it, adds the result to the transcript, and gives the model another turn.

Suppose the user asks Tau:

| Find the failing test and fix it.

During that turn:

1. The model begins responding. The loop emits live events so the interface can show progress immediately.

2. The model may request a tool such as read to inspect a file or bash to run the tests. The loop executes the requested tool and captures its result.

3. The loop adds both the model’s message and the tool result to the working transcript. The cycle repeats until the model returns a final response without requesting another tool.

The transcript is caller-owned which means that the loop can update the conversation history, but it does not decide where that history is stored or how long it survives. The surrounding AgentHarness supplies the transcript and carries it across turns.

The loop handles the sequence of a turn. It does not need to know how the tools are implemented, how messages are saved to JSONL, or how the terminal interface displays progress.

The ability to call a tool does not mean every requested action should run. The surrounding product still needs execution rules, sometimes called policy, that decide what is allowed.

For example:

Tau provides two places to enforce these rules: before_tool_call and after_tool_call. Before a tool runs, the product can inspect or block the request. After it runs, the product can check or modify the result before the model continues.

The complete permission system stays outside the loop. A coding product might require approval for a destructive command. A support product can use the same loop with different rules, such as requiring approval above a refund limit.

AgentHarness is the stateful wrapper around the loop. It owns the working transcript, pending work, cancellation, and the listeners that receive events. Its docstring describes it as a “reusable stateful agent brain independent of coding/UI policy.”

Tools define what the agent can do

Execution rules decide whether a requested tool may run and the tool implementation determines what happens once it does. Tau represents each tool with an AgentTool object.

tau_agent/tools.py

The description and parameters tell the model what it can request. execute_fn performs the action, while the optional renderers control how the request and result appear to the user.

The coding-tool implementations live in tau_coding/tools.py, outside the reusable tau_agent core. At the start of a coding session, Tau creates the read, write, edit, and bash tools and configures them to work from the selected project directory.

tau_coding/tools.py

This carries a lot of product meaning because it is what gives the reusable core its coding abilities. The same portable loop could run a different agent with different tools.

These built-in tools are designed carefully:

• read limits how much text is returned at once, gives instructions for continuing, and handles supported images.

• write creates or replaces files and prevents simultaneous changes from interfering with each other.

• edit checks that each replacement matches exactly once, prevents overlapping edits, and returns a diff.

• bash runs commands from the project directory, captures output, truncates noisy responses, saves the full output when needed, and returns structured execution details. It supports an optional timeout but has no default.

When reading a tool implementation, don’t just ask what the model can call but also ask what the tool preserves when things go wrong.

• Did it distinguish timeout from cancellation?

• Did it preserve the full output somewhere?

• Did it validate before mutation?

That is harness engineering in its infancy!

The session log preserves more than the model sees

tau_agent/session/entries.py

Tau stores an append-only JSONL log rather than treating the current transcript as the complete session. Messages, model changes, thinking-level changes, compactions, branch summaries, labels, leaf pointers, session metadata, and extension data all become entries.

When Tau needs the current state, SessionState.from_entries replays the relevant path through that log. The replay derives the messages the model should see, the selected model, thinking level, session information, and any compactions.

Take a 120-turn coding session. To stay within the context window, Tau compacts turns 1 through 90 into a summary. The model’s next request contains that summary plus the recent turns so it no longer reads the original 90 turns.

The JSONL log still retains the original entries. Tau can use them to reconstruct session state, export the history, or create a different branch. Resuming rebuilds the latest saved state, while branching rebuilds the session from an earlier point and continues along a new path. Compaction works differently by replacing older messages in the model’s working context with a summary, without deleting the original entries from the log.

Project instructions keep their source

Tau does not insert project instructions as an anonymous block of prompt text. ProjectContextFile carries both the content and its source path:

When Tau builds the system prompt, format_project_context carries both the path and content into the prompt by placing each file inside a labeled block:

That path attribute is important because the model not only sees what the instructions say, but also where they came from. Before adding the path, Tau converts special characters into a safe format. This prevents an unusual or malicious filename from breaking the prompt’s structure or being mistaken for an instruction. That might appear like a small detail but it is the kind of detail a real harness needs.

The full contents of each discovered project instruction file are added to the system prompt at the start of the session.

Skills are loaded differently. At first, the model sees only an index of the available skills. When a skill becomes relevant, it uses the read tool to open the full instructions. If the session does not have a read tool, Tau does not include the skill index. This, again, is a smart product decision because advertising a skill file the model cannot open would create fake capability.

Project instructions therefore remain connected to their source, while full skill instructions enter the context only when needed.

Product assembly and extensions

Most of Tau’s code lives in tau_coding, where provider setup, project trust, resources, tools, prompts, extensions, durable sessions, and the UI come together.

CodingSession is the assembly point. It loads session entries, resolves project trust, creates tools bound to the working directory, builds the system prompt, creates the AgentHarness, attaches listeners, and connects the extension runtime. AgentHarness owns the reusable in-memory agent brain and CodingSession owns the coding environment assembled around it.

tau_coding/session.py

Extensions are Tau’s plugin system. A developer can write an extension for their own project or install a trusted extension created by someone else. Extensions can add tools, slash commands, prompt guidance, message renderers, UI components, and input or lifecycle hooks without changing the core loop. Extensions run as Python code with the user’s permissions, so they must be treated as trusted software. Project extensions require both project approval and a separate opt-in before Tau loads them.

tau_coding/extensions/runtime.py

ExtensionRuntime merges extension tools with Tau’s built-in tools before creating the harness. An extension can replace a built-in tool with the same name, while new tools are added in registration order.

tau_coding/extensions/runtime.py

When Tau reloads extensions or switches projects, it prepares a new runtime before retiring the previous one. This prevents extensions from one working directory from remaining active in another.

The extension system is one example of Tau’s larger product boundary. Provider setup, tool rules, sessions, extensions, and interface behavior remain separate parts assembled around the reusable core. Each part can be inspected, tested, and changed without rewriting the agent loop. This is a useful thing to study!

The interface renders the event stream

Tau’s Textual TUI is large, but it is one frontend around the coding agent. The reusable core does not depend on Textual or terminal-specific behavior.

TuiEventAdapter connects the coding session to the interface. It receives session events and updates the display state that Textual renders:

Agent events, queued messages, tool execution, compaction, and automatic retries all pass through this adapter before appearing in the interface.

tau_coding/tui/adapter.py

Because the interface consumes events instead of being built into the agent loop, the same core can support Tau’s Textual TUI, plain terminal output, JSON output, or another frontend. Keeping the event-to-display conversion in a separate adapter also makes the interface easier to test. Tau can check how an event changes the display state without launching the terminal application.

What Tau is actually teaching

After tracing Tau from the provider layer to the interface, its feature list reveals the architecture underneath:

1. Session features: Resume, branching, compaction, and export are not standalone agent features. They come from append-only session entries and replay.

2. Provider support: Provider selection and model switching are not just settings. They come from a small provider contract and adapters outside the core.

3. Tool behavior: Read, write, edit, bash, image reads, and extension tools are not just capabilities, they define what the model can request, what the system will execute, and what the user will see.

4. Project context: Project instructions are not just prompt text. They are resources loaded with their source paths, escaping rules, and capability checks.

5. The interface: Tau’s terminal interface is not the harness and just a consumer of the event stream. The same underlying events could be presented through a different interface.

All this brings us back to our real lesson from Tau:

A coding agent is not a model with bash access.

It is software that decides what the model can see, what it can call, what gets persisted, what gets compacted, what the user sees, and where product policy can intervene.

Tau still leaves important production work unresolved. Its bash tool needs stronger controls around approvals, sandboxing, command classification, default timeouts, and verification.

But Tau shows where these decisions belong. Not inside a bigger prompt. Not inside one magic agent loop. But, inside explicit boundaries: provider adapters, tool contracts, session logs, context loaders, event streams, extension hooks, and UI adapters.

That is why Tau is useful. It makes the harness visible.

Harness Engineering is a three-part series. In Part 3, we will talk about what has to get stronger before a harness like this can safely run against something that matters.

Reading a harness and designing one are different skills. This series is about reading. Designing is what we teach in Harness Engineering: Designing the Operating Layer for Reliable Agents.

People often make claims at the extreme ends, but as with any technology, its that RAG and what is needed of retrieval has simply evolved. The early version was usually one question, one vector search and one answer. It worked, but it soon became a very limiting way to think about retrieval. Context windows got larger, search became more iterative and coding tools showed that vector databases were not the right fit for every kind of information.

In this video, I talk about the three different camps claiming RAG is dead, break down the elements underpinning each of their arguments, and discuss how RAG at many mature enterprises is evolving beyond vanilla naive RAG into full knowledge runtime systems. Definitely worth checking out if you're thinking about the state of RAG in 2026 and what some of the top teams are actually doing with it.

A quick visual of everything we discuss:

If you have been wondering whether RAG is still useful to learn, I hope this video helps settle that debate for you. If you do enjoy this style of short but knowledge-dense videos, do check out the previous videos on my channel as well:

• Learning to Code in 2026 and Beyond

• How to Use AI Agents Better

• My Codex Operating System

AI assistance note: Hate reading AI slop? This article was rated 70% human written by Substack and Pangram. We used some AI assistance to structure our thoughts, but the ideas and opinions are entirely ours. Rest assured, you are not wasting your time reading slop.

If you have recently been on X, you might have come across posts on how instead of spending their time prompting an agent directly, developers should be designing systems that prompt agents for them. In June, Addy Osmani published an article that cemented this as a practice called ‘loop engineering’ and soon after LangChain published its own framework for thinking about loops.

Take away all the hype though and nobody really knows how to implement it in a useful way. So much so that in early July, the community at AI Engineer World’s Fair was debating whether the hype had outrun the actual practice.

To understand what loop engineering is, let’s look at where it sits on the ladder of all the ‘AI Engineering’ practices;

• You start with Prompt engineering which is about what you tell a model.

• Context engineering deals with what the model has available to it at a given moment.

• Harness engineering is about designing one run which includes things like the tools, permissions, retries and actions.

• Loop engineering sits on top of all this and asks how that harnessed agent can run itself repeatedly.

While the hierarchy is a good reference, it is not universally agreed upon, with some people like Lilian Weng treating loops as part of the harness rather than a separate layer above it.

The origins of the loop

When people think of “loop engineering,” they often use it to describe several related ideas, some of which have existed for years under other names.

Last year, Geoffrey Huntley created the “Ralph loop,” which was simply a bash while-loop. It worked by feeding the exact same prompt to a coding agent over and over again in a fresh context window. It was noted then, that this technique is bad in a non-deterministic world.

Earlier this year, a tutorial called “ship working code while you sleep” that used Ralph Loops went viral, and things escalated. Codex shipped a /goal command, and soon after, Hermes and Claude Code added similar built-in commands to every major coding harness.

At a workflow level, it feels very similar to ReAct agents, where the agent is acting, observing, and revisiting steps whenever necessary. Traditional software systems have also used cron jobs, hooks, workflow engines and DAGs to automate repeated work for a long time.

How to design a loop

At its core, the steps of running a loop are straightforward.

You send context to the model. If the response has tool calls, you run the tools and append the results. If not, you break out of the loop.

There are four things you have to think about when designing loops:

1. When it stops: You have to define maximum iterations, budget caps, how you detect progress in workflows and what your final completion or success looks like so that the loop does not keep running forever.

2. What stays in context: You need to think about methods like compaction, handing off messy subtasks to sub agents and offloading some part of the output to different files or to the disk. This is important so that old tool outputs, dead ends, and stale reasoning don’t pile up and make your model feel dumber as the runs get longer.

3. Which tools it can reach: Fewer tools improve the success rate. Whenever you are writing to a file, you have to make sure it is safe to repeat or go over. If multiple agents are in the same file, you must ensure it is a safe write. You also need a subagent to check the errors you get and determine what to do with them e.g. setting escalation conditions.

4. How you check the results: You must pick a model that is different than the one you used to generate the output. This is important because if the same model is judging its own output, it will mostly agree with itself and lead to bias.

The LangChain stack

Now that we have looked at how to define a loop, lets look at the kinds of systems that can benefit from them. LangChain has mapped this out into a four-loop stack which can be thought of as 4 architectural layers you can deploy.

The agent loop

This is how most teams are currently using loops. It is the simplest layer where the model calls tools until it believes the task is complete.

Let’s take the example of a code documentation agent. It might receive a request, inspect a repository, plan the changes, edit the relevant files and open a pull request without anybody manually telling it which tool to call at each step.

The verification loop

The verification loop adds a grader around the agent loop which keeps checking the output against a rubric and sends it back with feedback if it fails. It can be anything, deterministic like a simple type check, a linter, or even the regular tests that you do in software development. It can also be another model acting as a judge.

As we saw in loop design, the model for the judge has to be different from the grader. Simply asking the generator to be more self-critical doesn’t work because the same system would still be reasoning from many of the same assumptions.

An open challenge Lilian Weng talks about here is how these verification and self-improvement loops are more prone to reward hacking because it is essentially optimizing for the metric you provide, which can cause overfitting.

The event-driven loop

This is the layer where the agent stops waiting for you to kick it off and has something in the environment trigger it:

• A heartbeat can continuously watch logs or system health.

• A cron job can perform a scheduled code review or dependency audit.

• A hook might fire when a pull request is pushed or a CI job fails.

• A goal-based loop can keep iterating through a refactor or migration until a condition is met.

This is also where designing the stopping condition becomes important because if a goal-based loop has no clear limit, “keep going until this is fixed” can easily turn into a model burning tokens for hours.

The hill climbing loop

The is an interesting one because it isn’t just about automating the work but going beyond and improving the system doing that work.

Every time an agent executes, it creates a trace that records its reasoning, tool calls, and actions showing how it got to the results. An analysis agent scans these traces to figure out what went wrong or where friction occurred, then tweaks the system’s prompt or code harness accordingly. Rather than feeding outputs back into the start of the user workflow, it rewrites the internal logic so future runs perform better.

Andrej Karpathy’s AutoResearch project shows this idea in action. The system runs an experiment, checks if the result improved, keeps what worked, and moves on to the next attempt. A caveat here is that a recursive loop does not automatically guarantee self-improvement and the base model still needs to be capable enough to recognize and produce genuine improvements.

This pattern clearly separates the inner and outer loops:

• The inner loop (capability): The core engine doing the actual work. It writes code, uses tools, checks results, and iterates.

• The outer loop (agency): The decision layer that chooses what to build and determines what gets deployed or blocked. It handles oversight through evaluations, feedback, and human input.

An example workflow

An engineer’s morning triage routine that uses a cron job, agent skills, sub-agents and a connector as shown below is a good example of an automated workflow.

So why are people skeptical?

Skeptics ask the basic question of whether Loop Engineering is just traditional software automation rebranded under a fancy new name. As we saw in the origin of loops, many practices have existed from long before.

Loops are built to keep trying, which means they can burn through tokens quickly when a task takes longer than expected or gets stuck. This can create a very real cost problem.

Lastly, even if Loop Engineering proves to be a reliable pattern, there is a good chance much of it will simply get absorbed into the tooling. We’ve already seen that happen with techniques that started as manual workarounds and later became built-in features, such as slash commands.

The costs of autonomous loops

Some problems become more evident after a loop has been running for a while.

A loop can open twenty pull requests overnight and still miss problems that the tests were not designed to catch. That is how verification debt starts to build. If the team starts to rely more and more on AI and those changes get merged without anyone really going through them, the codebase can quickly outgrow the team’s understanding of it, leading to comprehension rot.

Once the loop becomes reliable, it also gets easier to stop checking and manual code reviews become less rigorous leading to cognitive surrender. Add token costs on top and a loop that might have looked efficient, becomes expensive to trust.

This is also why not every workflow needs a loop. If a task rarely happens, or if success is difficult to define, the time spent designing and maintaining the loop may never pay off.

Loops are easier to justify when the work repeats and there is a clear signal for whether the next iteration improved anything. Tests, migrations, version upgrades and model tuning are obvious examples. They become much harder when “correct” is subjective or difficult to verify.

This article grew out of a live Chai & AI session conducted by Prahitha Movva where we got into loop engineering and whether it is a genuine shift or mostly a rebrand. All our Maven cohort members get access to our Chai and AI community, check out our cohort here.

The first three session recordings are already available to check out here:

• Building Agentic AI Systems in H2 2026, with Kiriti and Aish

• Building Agentic Workflows with Cursor, with Zoë Barnswell and the Cursor team

• Building AI Native Products in 2026, with Erran Berger, CTO of Engineering at LinkedIn

And here’s what we have coming up:

• Aug 17: Codex for Everyone: Building Agentic Workflows, with Kiriti Badam, Member of Technical Staff, OpenAI, Codex

• Aug 21: Open Source AI in Production, with Chris Alexiuk, Senior Product Research Engineer, NVIDIA

• Aug 24: Building a Production Voice Agent End to End, with Ed Charbeneau, Developer Relations Lead, Deepgram

• Aug 31: Why Agents Die Between Demo and Production, with Siva Surendira, Founder and CEO, Lyzr AI

• Sep 2: From Meeting to Content: Live MCP + Skills Build, with Ben DeMordaunt, COO, Grain

• Sep 9: The Anatomy of a Self Improving Agent, with Aparna Dhinakaran, Co Founder and Chief Product Officer, Arize AI

• Sep 11: Accelerating Voice Agents from Demo to Production, with Nikhil Gupta, Cofounder and CTO, Vapi, and Dhruva Reddy, Forward Deployed Engineer, Vapi

We’re also starting to announce in person meetups soon so keep an eye!

We’re having such a blast putting this together, and we’d love to have you join us.

Sign up for the AI Builders Summit here to get invites to all the upcoming sessions and future recordings.

Link again JIC: summit.levelup-labs.ai

Some of that is true. But I think a lot of people end up with the wrong explanation of what’s actually happening, which leads to completely unrealistic expectations around what AI agents can do. And because of that, I see people making the same mistakes over and over again when they try to automate their own work.

In this video, I break down:

• Can AI Agents Run Your Business Alone?

• How to Know What to Automate With AI

• Why Your First AI Automation Won’t Work

• Why AI Automation Still Needs Human Oversight

• Why You Shouldn’t Automate Everything at Once

A lot of this video is actually about what you shouldn’t be doing with AI. Because once you understand that, it becomes much easier to figure out what you should automate, how to get started, and how to gradually build processes that can run more independently and improve over time.

Most importantly, you’ll have much more realistic expectations around what AI agents can and can’t do today.

A lot of this comes from real experience automating processes for companies, as well as building these systems inside our own AI native startup, LevelUp Labs.

Hope this is useful.

-Aish

AI assistance note: Hate reading AI slop? This article was rated 100% human written by Substack and Pangram. We used some AI assistance to structure our thoughts, but the ideas and opinions are entirely ours. Rest assured, you are not wasting your time reading slop.

Vendor lock-in is often viewed as something to avoid at all costs. But locking into a platform sometimes offers a good trade-off and is not necessarily a bad thing. Subscribing to a managed service gives you speed, reduces what you have to manage, and leaves the maintenance to someone else. So it's not about having zero lock-in, but rather being deliberate about it and knowing exactly what you are trading away.

The problem often occurs when you sign up for a platform without knowing what is exportable or portable if you ever want to leave and get accidentally locked-in. It is crucial to understand that “owning” your architecture does not always mean self-hosting. You can rent or subscribe to a managed service and still own your intellectual property, provided you can export it in the formats you need.

You also cannot solve the lock-in problem by simply selecting a vendor that advertises itself as “open”. While open tools are great, whether a platform is truly open depends entirely on what you walk away with when you decide to switch.

We look at the different layers of the architecture, where it is better to rent vs own and the questions to ask when making vendor decisions.

The lock-in trap in practice

Consider a mid-size insurance company building a small claims triage agent. This agent reads incoming claims, summarizes them, flags suspicious ones, and drafts an initial response. Imagine they build this entirely on a single vendor’s platform, and the agent goes live and works well.

Two years later, a competing architecture hits the market that appears to do the same job just as accurately for roughly 40-50% less money. Because the company has a month-to-month contract, switching looks like a simple decision on paper. In reality, the switch is incredibly difficult. If their prompts were tuned to the original model, they have to figure out how to move them. If their data is stored in the vendor’s vector store, they need to know if they can export the embeddings or just the raw text. Without a solid set of evaluations, or “evals,” they also cannot easily prove that the new model will produce safe and accurate outputs.

Proving the new setup is safe would require re-tuning prompts, re-embedding two years of data, and building evals from scratch. This amounts to months of work before anyone can sign off on a system that is already live. Consequently, the company is trapped the moment a switch becomes too expensive to prove it is safe.

Renting, owning and the seams of your architecture

To avoid the trap of vendor lock-in, you have to check the different parts of your architecture and decide what to keep on your side. For example, if we look at an agent built with LangGraph, the framework handles the orchestration, branching, and tool calls. However, everything truly valuable sits outside of that framework.

You need to ask specific questions about each layer to determine what you should rent and what you should own.

• The Model (Rent it): You do not need to build your own large language model. Models are highly swappable, but only if you own the surrounding layers. Even a simple swap usually requires re-tuning prompts to the new model. This is less of an issue when your prompts are versioned locally and your evals can prove the quality holds.

• The Interface (Own it): Instead of calling a model directly from your code, put a gateway layer like LiteLLM in between. This turns a model switch into a simple configuration change. It also provides resilience, access control, and observability.

• Prompts and Context (Own it): Your prompts are your intellectual property, and it takes months of tuning to get them right. They need to be versioned and kept in a repository or database you control so they remain portable. Additionally, while a model has general knowledge like say accounting principles, it does not know your specific standard operating procedures (SOPs) like the specific way your company closes its books or prices a deal, the small shortcuts that make the process work etc. You must carefully decide how much of your specific business context you are willing to hand over.

• Data and Embeddings (Own it): This is the heaviest area of lock-in and the hardest thing to move. If you decide to leave a platform, you need to know if you can export the actual vector embeddings or just the raw text. If you only get the text, you will have to re-process and re-embed everything from scratch. You should keep your data in a store you can easily export from.

• Evaluations (Own it): Without a reliable eval suite, you cannot tell if switching models is safe. Your evals allow you to ascertain whether quality is retained during a switch. You must ensure you can download your eval set rather than being forced to manually copy and paste it if you leave an observability provider.

• Orchestration (Rent it): Frameworks like LangGraph or CrewAI save you a lot of time. Switching between them is not easy, but you will not do it very often. The key is to keep your prompts, data, and evals outside the framework so they remain in your control.

Enterprise scale: governance vs. innovation

The approach companies take to these architectural choices often depends on the scale of the company. Larger companies tend to focus heavily on governance and security when making platform decisions. Major vendors like Amazon, Google, or Microsoft provide the compliance and security backing that enterprise decision-makers require, such as adherence to the EU AI Act. This reliance on governance creates its own kind of lock-in, as it forces companies to choose between a secure, governed architecture and the raw innovation of a smaller startup platform.

Conversely, smaller companies are often more pragmatic. They are highly focused on the final output and frequently use tools like n8n to integrate AI into specific workflows. While they still care about compliance, they prioritize getting results quickly.

Taking Control of Your Architecture

You cannot own every part of your architecture from day one. You have to think about which parts are the most expensive to switch. Usually, data, evaluations, and prompts hold the most intellectual property and carry the highest switching costs.

To maintain control, you should ask three questions of any vendor:

• Can I export my vectors, or do I only get text?

• Can I take my prompts out as files?

• Do my evals run against another model, or only the one this framework uses?

If the vendor supports open standards, make sure that does not just mean data is portable only in their specific format. Do not over-engineer a massively portable system, but do deliberately accept the lock-in you choose. When you present your architecture and business case, you want to show that you made informed decisions, rather than discovering a painful lock-in months down the line.

This article grew out of a live Chai & AI session led by Jacob Chacko where practitioners dug into vendor lock-in in agentic systems, and where people shared exactly how it plays out inside large enterprises. These sessions happen regularly and they get intense. It’s invite-only for course alumni, but we’ll keep sharing highlights here every week.

AI assistance note: Hate reading AI slop? This article was rated 79% human written by Substack and Pangram. We used some AI assistance to structure our thoughts, but the ideas and opinions are entirely ours. Rest assured, you are not wasting your time reading slop.

Everyone has a lot of ideas when it comes to building AI Agents. The challenging part is figuring out what the ROI is going to be and if it is worth building at all. McKinsey’s 2025 State of AI survey shows that while 62% of organizations are at least experimenting with agents, only 39% report any EBIT (Earnings Before Interest and Taxes) impact at the enterprise level.

When building, most teams go through their ever growing backlog to rank the ideas and pick the top ones. But ranking is not the same as costing and when leadership asks about the impact, you don’t quite know how to answer.

Being able to understand the true value provided by AI takes four steps that need to happen before you commit and together form The ROI Gate:

1. Baseline: Measure today’s reality instead of assuming it.

2. Attribute: Determine the value that comes solely from the agent.

3. Book: Use terminology and labels that finance can understand instead of vague statements like “hours saved.”

4. Gate: Provide a defensive go/no-go decision with a clear payback period.

The baseline and attribution you define upfront become the exact instruments you measure against once the agent is live.

The gap in existing frameworks

Most analyst firms and big consultancies have their own frameworks for prioritizing agent ideas but these are often paywalled or require a consulting engagement. Google’s and OpenAI’s frameworks are open and free, which makes them easy for teams to leverage.

Google scores a use case on Value, Actionability, and Feasibility, while OpenAI plots Impact against Effort. If you work in product, you might recognize that these are not really new frameworks built for AI Agents but a variation to the frameworks that are already used to triage and prioritize ideas.

Triage, however, is only the first half of a longer process. Conventionally, a downstream product manager or analyst would take the shortlist and cost it properly. In the agent era, things move so fast and there is a bias to ‘just build it,’ that the costing step is easily missed. What you are left with is a pair of tools that are great at picking which idea to chase, but completely unable to tell you what it will cost.

To see how this gap plays out and how to overcome it, let’s take a common enterprise use case: an Accounts Payable (AP) Agent. We will run it through the standard frameworks first, and then through the ROI Gate.

The use-case: an Accounts Payable Agent

Accounts payable (AP) is the team that pays your company’s bills. A supplier ships something, sends an invoice, and AP has to decide whether that invoice is legitimate before paying it. At a mid-size company that’s roughly 120,000 invoices a year, which translates to around 500 every working day.

To verify an invoice, AP teams run a three-way match comparing three documents:

• The Invoice: What the supplier says you owe.

• The Purchase Order (PO): What you agreed to buy and at what price.

• The Goods Receipt (GRN): Proof the items actually arrived or the service was delivered.

If all three agree, the invoice is determined to be “clean” and gets paid. If any of them disagree, which is roughly 1/5th of the times, it becomes an exception. Someone then has to chase down the ‘why,’ which could be anything from the quantity being off to the price being different. That chasing takes time and costs money.

The money hiding in the calendar

Suppliers often offer early payment terms, typically written as 2/10 net 30. This means if you pay within 10 days, you take 2% off the invoice. Otherwise, the full amount is due in 30 days. This means that you get a discount without having to negotiate a better price and simply handing over cash 20 days early.

But to pay in 10 days, you have to approve in 10 days. If an invoice sits in a manual queue while an exception is investigated, the window closes and the discount is gone.

An AP Exception Agent attacks this by doing four things:

1. Ingesting the incoming invoice.

2. Running the three-way match automatically.

3. Auto-approving anything where all three files reconcile perfectly.

4. Routing the remaining exceptions to a human and recommending a fix.

It is an ideal test case because the process is mature, the data is structured, and the savings are reflected in a place finance is already tracking.

Applying the Traditional Frameworks

Framework 1: Google’s matrix

If we score this AP Agent use case from 1 to 5 on Google’s three axes we get the following:

• Expected Value (4/5): The savings are large and well-documented, but since it is cost reduction rather than top-line growth, it doesn’t get a perfect 5.

• Actionability (5/5): There is a clear owner and an existing budget line.

• Feasibility (5/5): It is invoice matching with structured data and versions of mature OCR systems have done this for years.

Multiplying these scores yields 100 out of a possible 125. Re-scaled, that reads as a 4/5 and a resounding green light to “build it.”

Google’s template also prompts you to measure the cost per invoice, touchless rate and cycle time post-deployment, but none of these metrics track actual dollars.

Framework 2: OpenAI’s impact and effort

OpenAI’s version comes from its Identifying and Scaling AI Use Cases guide, which is used by its customer success teams to help enterprise customers prioritize. It sorts every candidate into one of four quadrants.

For the Account Payable Agent, since the saving is strong and well documented, the impact is high. The scope is bounded and the tooling is mature, so effort is low. It lands in quick wins, which means do it first. You start with bounded wins so you can prove the thing works and earn the buy-in for harder agents later.

The Verdict of the Frameworks

While both tools are helpful for prioritization, they stop short of the actual business case. You walk away with a quadrant, a self-assigned score, and a feeling of rigor but still lack a baseline, an attribution model, and hard numbers.

The ROI Gate Framework

We look at this for a fictional company and reference industry benchmarks from Ardent Partners’ State of ePayables for what best in class looks like.

Step 1: Establish the Baseline

Write down the current reality before you start building.

Now, let’s calculate the cash leaking from our early payment terms:

• Total supplier spend: $200 million a year

• Addressable Spend: 20% of suppliers offer early terms so $40 million of spend is eligible

• Maximum Available Discount (at 2%): $800,000 per year.

• Our Current Capture Rate (40%): We only capture $320,000.

This means $480,000 a year is left on the table simply because we approve too slowly. This is what the agent is trying to help with. You must document this baseline before you go live, because you cannot reconstruct this data afterward.

Step 2: Define Attribution

If the agent works that we assume we can hit best-in-class performance. The touchless rate becomes 35%, discount capture goes to 70% and late payment penalties drop. This gives $395,820 in savings:

• Processing Labor: 12,000 invoices stop costing $10.89 to process manually and drop to $2.78 giving savings of $97,320.

• Discount Capture: Raising our discount capture rate from 40% to 70% gives an extra $240,000.

• Penalties Avoided: 600 fewer late payments at $97.50 each gives savings of $58,500.

The agent is not solely responsible for these changes though and the $395,820. For instance, some of the discount gains could have come from procurement renegotiating terms.

You have to go line by line and ask what the agent is genuinely responsible for. The process already ran OCR before you arrived, so what did you add on top? Are you lifting the cases OCR was failing? Is it a confidence score across more parameters, so fewer invoices route to a human at all? You have to break the workflow into pieces, find the areas the agent actually improves, and work out its honest share of each:

• AP Labor Attribution (95%): This is almost entirely driven by the agent and $97,320 x 0.95 = $92,454.

• Discount Capture Attribution (60%): This is shared with procurement’s negotiation efforts so $240,000 x 0.60 = $144,000

• Penalties Avoided Attribution (80%): This is driven primarily by the agent’s approval speed so $58,500 x 0.80 = $46,800

The gated and attributed value then becomes $283,254 instead of the modeled $395,820.

Step 3: Book it

Things like “Hours saved” are ghost metrics. A controller needs something that maps to a named accounting line and can be audited:

• Labor Savings books to SG&A.

• Discount Capture books to Income / Cost of Goods Sold (COGS).

• Penalties Avoided books to Cost Avoidance.

Step 4: Gate it

Now, we net our gated annual value against the actual costs of running the software:

• Gated Annual Value: $283,254.

• Annual Run Cost (Tokens, licenses, IT maintenance): $120,000.

• One-Time Build Cost: $100,000.

• Net Gated Annual Value: $283,254 - $120,000 = $163,254.

At a 7.4-month payback, the verdict becomes a clear GO.

By gating this honestly, you get a $283,000 figure instead of an inflated $396,000. You get a number you can actually defend, rather than having to explain a $113,000 shortfall once the agent is live.

When the same agent has the wrong scale

What happens if you run this exact same agent in a smaller operation processing only 20,000 invoices a year where there is already 50% automation?

Applying the same benchmarks proportionally gives a far less margin to save. The gated annual value decreases to just $31,000 but software vendors have minimums so your costs barely budge and building the integration still costs $100,000.

With a run cost of $120,000 and savings of $31,000, you net a negative $89,000 per year. This makes the verdict a hard NO-GO.

The agent pays off when there is repetitive and deterministic work to be done but if you shrink the volume, the exact same setup returns a financial loss.

When the case is wrong

The AP use case is an easier one because it is internal. If you were to look at a customer-facing agent, the same standard cost-per-contact calculations could severely mislead you.

It might look like a human customer service interaction costs several dollars while an AI agent costs pennies. But while the savings model looks great on paper, it fails to capture the depth of which contacts the agent is handling well.

Say 75% of your customer queries are routine and the agent resolves 90% of them. The remaining 25% are high-complexity issues like disputes, fraud and hardship where here the agent’s resolution rate drops to 55% compared to a trained human’s 85%.

Your dashboard shows this as an 81% resolution rate but misses to highlight that the agent is failing on the most sensitive, high-value queries.

Those unresolved complex cases lead directly to customer churn. Because customer lifetime value (LTV) never shows up on a cost-per-contact spreadsheet, your baseline must explicitly account for customer retention and not just the operational speed.

When one agent becomes several

Say that the first Accounts Payable agent goes live and other teams want their own. It would be natural for leadership to ask should we build a shared platform architecture for all of them?

This is asking a fundamentally different question. For a single agent you ask if it pays for itself quickly but for a platform you have to consider whether a shared foundation is worth building at all. A platform makes sense once several agents live on it, so you can only answer the platform question across a multi-year portfolio.

To keep these two decisions distinct from each other, you can refer to these guidelines:

1. Ensure each agent still pays for itself. A platform makes the next agent cheaper to build, because the integrations, evals, and guardrails already exist. But ‘cheaper to build’ is not the same as ‘worth building,’ and an agent has to deliver business value.

2. Only build the platform once you have a real queue. A from-scratch agent might cost $100,000, whereas building on the platform costs only $31,000. That is $69,000 saved per agent against a $250,000 platform, so it takes roughly four credible agents before the platform breaks even.

3. Don’t trust the later years. Year two and three are guesses where AI costs and benefits drift. Instead of modeling a flat line, model a curve. Capture about 60% of an agent’s projected value in its launch year, 100% in the second, and decay it to 85% in the third.

The Amortization Trap

The way you allocate the platform cost on paper decides whether it looks good or bad.

If you load the entire $250,000 platform cost onto the very first agent, that single agent suddenly has to justify a $290,000 bill. Its payback period will stretch past 21 months and a great platform will get rejected.

In order to avoid this, gate each agent on its own marginal build cost and amortize the platform across the portfolio. Six agents over three years pay back that $250,000 platform in about 17 months, generating a cumulative net value of over $800,000. A low-value agent that would have failed as “Build #1” makes sense as “Build #5,” costs just $31,000 to build and pays itself back in 6 months.

The Reality of Agent Business Cases

Attribution can be genuinely hard. In a workflow with a dozen moving parts, you can rarely claim the exact precision your model implies, so be conservative. The ROI Gate also biases you toward countable wins over ambitious ones and it is important to acknowledge that when it happens.

The frameworks are not wrong. They are good at triage which is an important first step. The ROI gate is not a replacement for them but rather a lens you apply after triage. You still need a way to narrow a backlog of 50+ ideas down to the 2-3 worth doing costing for.

Focus on areas where you can get the data in weeks and not quarters. Finance data is ideal because it is usually structured and readily available. Compare getting duplicate vendor records to pulling raw sensor data off 200 factory machines and you will spend six months on preparing data before an agent can even run a pilot. That data-engineering cost belongs in your business case and is never “somebody else’s” problem.

Design the guardrail before you ship. An agent that wrongly auto-approves 200 invoices gets dismantled in its first week of go-live and nobody reads your eval suite at that point. Route anything above a threshold to a human, let the agent recommend rather than act where the stakes are real and keep false positives near zero.

Before you say yes

Run the gate on your next pilot before the first sprint, rather than after go-live.

Sources and Frameworks

This article grew out of a live Chai & AI session run by Jacob Chacko where he shared the frameworks for measuring the value of use-cases and practitioners worked through the ROI of enterprise agents in real time. These sessions happen regularly and they get into the details most tutorials skip. It’s invite-only for course alumni, but we’ll keep sharing highlights here every week.

The ideas in the this article are original and it was written with some AI Assistance.

We’ve reached a point where writing code is often the easiest part of the workflow. The harder work is deciding what to build, specifying it clearly, reviewing what the agent produces, catching flawed assumptions, and knowing whether the system is actually ready to ship.

In this video, I break down 5 trends changing how coding works and what you should focus on learning in 2026 and beyond.

In essence, I think the old way of learning to code, then specializing deeply in frontend or backend development etc., is fading. The newer skill is to pick up foundations and then learn how to judge whether the code is good, understand the architecture and logic behind it, write the right specs, and stay closely involved in the building process without manually handling every line of code.

I share a streamlined framework in the video to do so.

A quick visual summary of everything we discuss:

Hope you enjoy the video. If you do enjoy this style of short but knowledge-dense videos, do check out the previous videos on my channel as well:

• My Codex Operating System

• AI Engineering Explained in 17 mins | The Ultimate Beginner’s Guide

• How to Use AI Agents Better

Most of the content out there is still a rundown of features. It tells you what Codex can do, but not how to combine those features into a useful system, which use cases it’s actually good for, or how to set it up so it becomes part of the way you work.

That’s pretty much the intention of this video: to give you an overview of what Codex can do and the operating system you need to build in order to make it work well. It mirrors how we use Codex inside our company and how we’ve built an operating system underneath it, with Codex features sitting on top.

The important part is that the operating system isn’t tied to Codex. If we decide to experiment with another harness or move to a different system later, the underlying structure stays intact. We can simply plug in a new tool without rebuilding everything from scratch.

It’s extremely beginner friendly, but it still covers nearly every major Codex feature and shows how to use each one in a real system. We’ll cover how to build a strong knowledge layer, work with plugins and skills, create personal evals, and use Codex Threads, slides, and other features. Here’s everything we cover in a 17 min video (phew, that’s a lot now that I see it).

Hope you enjoy the video. If you do enjoy this style of short but knowledge-dense videos, do check out the previous two videos on my channel as well:

• AI Engineering Explained in 17 mins | The Ultimate Beginner’s Guide

• How to Use AI Agents Better

Prefix caching is one of the most measurable ways to cut agent cost, but it is just as easy to misconfigure, where it saves nothing, adds latency, or quietly changes what your agent does. When PwC ran more than five hundred agentic sessions across OpenAI, Anthropic, and Google, they found that the best caching strategies cut cost by around 80% and time to first token by 31%. But naively caching the whole context, which is often a common setup, actually made latency worse.

The good news is that the failure modes are understood, and most of them come down to a handful of habits you can fix.

Part 1: What caching is

The model only reasons over what it sees

Everything an agent does comes down to what is in front of the model at inference time. The model has no access to anything outside that window, so success depends less on how you word the prompt and more on what actually fills that window. This includes the system prompt, tool definitions, memory, history, retrieved documents, all of what constitutes context engineering.

Anthropic’s Effective Context Engineering for AI Agents post, states that the guiding principle is to find the smallest possible set of high-signal tokens that get you the outcome you want. Even with enormous context available through code, markdown files, and external references, most of the value comes from a few high-signal instructions, while the rest adds noise, maintenance burden, or token overhead.

While there are wider layers to engineer like the environment the agent runs in (harness engineering) and the loop that drives it (loop engineering), the cost still concentrates in the context (context engineering). Every token in the context window gets processed on every turn, and caching is what lets you reuse that curated window without paying the full price each time. If you get it right, everything built on top of it gets cheaper too.

Not all caching is the same

Prefix caching is one of a family of caching techniques that sit at different layers of the stack, and it helps to know where it falls before going further. From the top of the stack down:

• Prefix caching sits at the provider API level. As a model generates, it stores the intermediate state for tokens it has already processed so it doesn’t recompute them. This is KV caching. Prefix caching extends this idea of KV caching across requests, so when calls share the same leading tokens, such as a stable system prompt and tool definitions, the state for that prefix is stored and reused. It is also referred to as prompt or context caching and is the layer most builders actually control.

• RadixAttention and PagedAttention sit at the serving-engine level. SGLang’s RadixAttention builds a tree of token sequences and automatically detects when requests share a prefix. vLLM’s PagedAttention does a block-level version that mainly reduces memory fragmentation. These only matter if you run your own inference stack.

• Semantic caching sits at the application level. It returns a stored answer for a query that is similar but not identical. Since an approximate match can return an answer that doesn’t quite fit the question, it is lossy. This makes it a fit for simple FAQ-style questions rather than precise ones.

• Session caching sits at the application level. It keeps track of a conversation’s growing history within a set time-to-live for multi-turn use.

Most builders only ever touch the top row. The serving-engine layer is something we will come back to when we get to eviction.

Why just turning it on fails

So why would caching ever make things slower? It comes down to how the match works. Rather than looking for a prompt similar to one it has seen, the provider matches exact tokens, running left to right from the very first one and stopping only until it reaches a token that differs from last time. Everything up to that first difference is the stable prefix (the leading tokens that are identical on every call) and it gets read straight from the cache. Everything after it has to be recomputed which is what makes a single dynamic value so expensive.

Say you put a timestamp near the top of the prompt. This becomes the first point of difference, so the cache holds only up to that line. The model then recomputes everything below from scratch including the tool definitions, skill bodies and examples, even though none of that content changed.

This is also why reordered JSON keys in your tool definitions can wreck a hit rate. If your app builds tool schemas at runtime, filters them by permissions, or serialises them with a library, you have to account for the key order that comes back. Otherwise you end up with tools that are semantically identical but hash to different bytes, which means zero cache hits and a lot of needless recomputation. The model is matching exact tokens from the very start, not reading for meaning.

Part 2 : Doing it right

The anatomy of a cache-stable prompt

Cache-stable prompts follow one layout rule: put everything stable first so it can be cached, and everything that changes last so it can’t break the prefix. A recent blog post on cache-aware skill design breaks this down.

Keep your system instructions, tool definitions, skill bodies, output contracts, and few-shot examples, in that exact order, in the stable prefix. These are the things that stay the same on every request. A single whitespace character in the stable section is enough to break the prefix so it is best treated like a build artifact. You version it, keep it frozen, and change it only when you genuinely need to or when you’ve missed something important. Everything that changes belongs in a dynamic suffix that never gets cached. Things like the task description, retrieved documents, tool results, timestamps, session IDs, and user context go there.

Providers differ on the exact order: Anthropic’s is tools, then system, then messages, while OpenAI generally wants the system prompt first, then tools and definitions. Beyond the ordering, two other rules are important to follow:

1. Never timestamp the system prompt. Per-request metadata belongs in the user turn, not the top of the prompt. One timestamp at the top of a 20k token prompt means zero cache hits, every single time.

2. Freeze and sort your tool definitions. If you assemble schemas at runtime, you can end up with structurally identical tools that serialise to different bytes. Sort the keys and pin the serialisation order before you send them.

How you actually mark something as cacheable depends on the provider. Some enable prefix caching automatically and simply reward you for keeping static content at the front. Others make it explicit. With Claude’s Messages API, for example, you send an ordered set of blocks, tools, then the system message, then the user message, and you attach a cache_control parameter set to ephemeral on the block you want cached up to. That’s the signal that tells Claude to cache everything to that point. When you are sending ordinary chat messages through a consumer interface, this is handled for you internally and most of the manual work applies only when you are building against the API directly.

Measuring the hit rate

There are plenty of situations where everything looks set up correctly and you still aren’t getting the benefit so it is important to measure whether caching is actually working.

The core metric is hit rate, the fraction of your input served from the cache, which you can think of as cache_read / (cache_read + input). Log it per call type and per agent type, set a threshold, and treat it the way you would treat latency spikes or error rates. If the number drops, treat it as a regression and find out what changed.

One team found that moving a 300 token user-context block out of the system prompt and into the start of the user turn, increased the hit rate from 23% to 71%. The block had held a user-specific ID that was breaking the cache for everyone. Once it moved to the user turn, the system prompt’s token usage became identical for every user in the organization and the cache could finally do its job.

Two more signals are worth instrumenting.

• cache_read_input_tokens tells you what you are getting back per call, and

• cache_creation_input_tokens tracks your write cost so you can sanity-check the TTL math.

On stable workloads, a target of 70% or higher is reasonable. As a related aside, a rising output-token ratio is a sign your agent is getting too verbose, and something as small as adding “be concise” to the cached system prompt has been shown to cut output costs by roughly 15 to 20%.

What a clean prefix is worth

A LinkedIn post illustrates this well with a scenario where a CLAUDE.md file is injected into every conversation with a coding agent. A 2K token file sent on every turn across a large team for a year, can cost around a quarter of a million dollars just to process. However, if that same file sits in the most cacheable spot there is, right at the front of the prompt, caching reads it back roughly ten times cheaper and drops the real cost to about 25K. For a large company, that is probably close to a rounding error, so why does it matter?

It is important because cost is only one part of it. Every line in that file also competes with the actual task for the model’s finite attention budget, so a weak or redundant line costs you in wrong turns and re-prompts, not just in tokens.

Part 3 : Where caching still doesn’t save you

The PwC study shows two sides: what caching is worth, and how it backfires when misapplied. The study set out to measure the ROI of caching for multi-turn agents rather than simple chatbots, using a 10K token system prompt. The researchers simulated a real production environment and asked the agent to autonomously run web searches and produce PhD-level research reports across different fields. It wasn’t run at a massive scale, so treat the exact figures as directional rather than gospel.

They compared three strategies.

• Full-context caching stores everything, including tool results and conversation history.

• System-prompt-only caching stores just the system prompt.

• A third strategy excludes tool results while keeping everything else.

On most strategies the results are strong on both cost and speed, with one exception: naively caching the full context on GPT-4o. The same GPT-4o that saw a healthy time-to-first-token improvement on system-prompt-only caching performs badly when the whole context is cached. The cost saving is still there, but the speed gain reverses.

The reason ties straight back to the byte-level matching from Part 1. The system spends time writing dynamic, non-reusable tool results into the cache, and that write overhead never gets paid back because those specific results aren’t needed in future sessions. You pay the cost of caching and never collect the benefit. It is the clearest illustration of a rule that recurs across the literature: caching everything is an anti-pattern.

That failure is the first of several. Even with a clean structure and a healthy hit rate, there are places where caching does nothing for you or actively works against you, each with a different cause and a different fix.

The cases where there is nothing to reuse

Some setups can’t cache well no matter how you structure them:

• One-shot or high-variance traffic. When every request carries a different prefix, there is nothing to match and you get zero hits. You pay the write cost and never earn it back.

• Sparse traffic. A cache entry only pays off if it is read back before it expires. On low or occasional traffic, it often expires unused, so you keep paying to write a cache nobody reads. Anthropic, for instance, needs one read within the five-minute window to break even, or two within the one-hour one. Exact numbers vary by provider, but as rough anchors, caching usually kicks in somewhere around 1K to 4K tokens and entries live for anywhere from 5 minutes to 24 hours.

• Dynamic content in the prefix. A timestamp or a session ID near the top drops your hit rate to zero on its own. If you can’t move it out, caching isn’t worth turning on.

The next few are trickier because they can come up even when your prompt is structured perfectly.

When the cache fills up: eviction

The serving engine has finite GPU memory and when the cache fills, something has to go. The default rule almost everywhere is LRU (least recently used), the same policy you see in databases.

For agents, LRU is a surprisingly bad fit. Picture a multi-agent workflow where one agent finishes its turn and hands off. While it waits to be called again, its cached prefix just sits there and because the other agents in the workflow are active, it becomes the least recently used thing in memory. LRU evicts it. When it’s called again, it recomputes everything from scratch. You can structure the prefix perfectly and still lose, because the eviction policy doesn’t understand the workflow.

Pan et al. tackle this with KVFlow, a workflow-aware alternative to LRU. Because it knows which agent runs next, it can hold on to the prefix that agent will need, drop the ones that are actually finished, and fetch the next prefix back before the agent is called.

On a single workflow with a large prompt, KVFlow ran about 1.83x faster. Under high concurrency, where memory is tightest, that rose to 2.19x. Naive caching went the other way, dropping to 0.57x of the plain baseline, so under load it was actually slower than doing no clever caching at all. That is the same problem of paying to cache what you can’t reuse, this time at the eviction layer.

The honest caveat is that you can’t control eviction through a provider API. That is the provider’s job. Treat this as a know-it-exists topic unless you self-host on something like vLLM or SGLang, in which case it becomes a real lever.

RAG breaks the prefix by design

RAG is another place where a stable prefix simply doesn’t hold. In a RAG pipeline the retrieved chunks change on every query and get reordered by relevance score, so no two requests share a prefix. That leaves you with a near-zero hit rate on exactly the content prefix caching was meant to help with.

Naively reusing the KV cache from old requests for the new chunks makes it worse, not better, because you lose the cross-attention between chunks and quality drops. The CacheBlend paper, a EuroSys 2025 best paper, offers a middle path. You reuse the cached KV for each chunk, then selectively recompute only a small fraction of the tokens (5 to 18%) carrying the cross-attention, so the model still understands how the chunks relate. The reported results are near-zero quality loss, 2.2 to 3.3x faster time to first token, and a near-100% effective hit rate.

Caching only handles the static half

Caching does well by the static half of your context like the system prompts and tools. The dynamic half, like the agent’s memory and history, is another matter. It degrades in ways caching can’t touch. Your Agents Are Aging Too lays out four of them, all of which happen while the model weights sit perfectly still:

• Compression aging: Summarising memories at write time drops a detail you turn out to need later.

• Interference aging: Similar memories pile up and bury the right one. Ask about John Smith and you get John Smyth, because the more common name wins.

• Revision aging: State changes, but the derived copy in memory never gets updated.

• Maintenance aging: A flush or a re-compaction quietly breaks things, for example by sumarising a whole chat and destroying the exact-match content prefix caching relies on.

Caching keeps the stable context cheap and reliable, but none of that touches the moving parts. Memory has to be engineered on its own, right alongside the caching work.

Restructuring for caching can break behavior

There is a subtler risk running through all of this. When you restructure a prompt for cache stability, moving blocks around and consolidating instructions, you are changing the prompt, and that can change behavior in ways you didn’t intend.

In the paper What Prompts Don’t Say, researchers found that LLMs infer underspecified requirements by default about 40% of the time. They are good at guessing formatting and terrible at guessing conditional rules. They also confirmed the intuitive result that piling too many instructions into the prefix makes instruction-following worse, whether from conflicting evidence, sheer volume, or both. They define underspecification as leaving essential requirements out of the prompt on the assumption that the model will fill the gap with common sense. Those implicit behaviors, the ones you never wrote down, are exactly the ones most likely to regress when you touch the prompt. On their data, roughly 41% of unspecified requirements were met at 98% accuracy or better, but those same behaviors were about twice as likely to regress when the prompt changed.

So before you restructure anything for caching, audit what your current prompt actually produces, add regression tests for those behaviors, and write down the things you truly care about. That is how you catch the underspecified behaviors and either specify them or protect them. The paper takes this further with a technique it calls Bayesian Prompt Optimization, which models each requirement as an on-off switch and works out which rules the model already follows on its own, then strips those from the prompt. That requirements-aware approach improved accuracy by about 4.8% over naively specifying everything.

Shared caches: the governance risk most teams miss

Caching carries a privacy risk that has nothing to do with cost and is easy to miss.

When one study audited 17 API providers, 8 showed detectable caching and 7 of those shared their caches across all users. That sharing is the problem. If two people send the same prefix and the second response comes back faster because the first person’s computation was reused, the timing itself leaks information. An attacker can submit a sensitive string, a proprietary code snippet or a specific medical term, and read the time to the first token to learn whether someone else recently sent the same thing. The risk is small for most workloads, but it is worth a second look if your prefix holds a sensitive system prompt, proprietary tool definitions, or internal business rules.

This has already exposed real secrets. It’s how researchers worked out that OpenAI’s text-embedding-3-small is a decoder-only transformer, a detail OpenAI never published, since only decoder models produce these prefix cache hits. If you are running a high-security deployment, per-user caching shuts this door. It is also worth asking your provider a few direct questions: is the cache isolated per organization, are timings normalized, and what is the residency guarantee?

Zooming out: the text layer is a real optimization axis

To see why any of this matters, it helps to zoom out. A recent blog post argues that the mutable text layer, the prompts, harness, caching, and retrieval, is an optimization axis in its own right. It isn’t just scaffolding around the “real” model work of pre- and post-training. It is an actual mechanism for improving the system, with its own scaling behavior.

Think of three layers.

1. The weight layer is training, fine-tuning, and RLHF which is high cost, high leverage and slow to move.

2. The inference layer is sampling, temperature, and decoding which is runtime, per-call and cheap.

3. In between sits the text layer, the prompts, context, memory, retrieval, harness, and caching, which is lower cost, fast to iterate on and sample-efficient. Caching is one discipline within that middle layer.

What keeps showing up is that the teams shipping the most cost-effective agents in 2026 aren’t only picking better models. They treat the text layer with the same rigor they give to model selection, and in quite a few cases that’s enough to get a small model performing close to a large one, purely by optimising this layer well.

Where to start

None of this needs a big project. Pick one prompt in production, work down the checklist below, and treat the cache-hit rate you end up with as a number you now watch.

This article grew out of a live Chai & AI session conducted by Prahitha Movva where we got into prefix caching, and how a setting that looks like a cost-only checkbox can quietly make your agent slower. All our Maven cohort members get access to our Chai and AI community, check out our cohort here.

The vocabulary around what wraps a model keeps shifting. What used to be called an orchestrator became an agent, then a harness, and in some flavours it now goes by agentic SDLC. These terms overlap heavily, and the field has not landed on clear distinctions.

A simple way to think of a harness is as extremely good software engineering practices applied to Applied AI problems. The model itself can be thought of as the unit of compute and the harness as the design philosophy that wraps around it, scaled via code.

The framework that follows, walks through how to think about a Harness:

• The three layers of a well defined harness

• Whether to design from the model or from the job to be done

• Whether to build from scratch or compose from existing harnesses

• How to treat evals so they do not become a bottleneck

• Where harnesses sit as models get more capable

1. Map the three layers

Every harness has three layers worth distinguishing.

The model layer is how intelligence is consumed and how it flows through code. This covers the model itself, how it is prompted, and the inference patterns used to call it. How you use the model can matter as much as which model you use. A recent MIT paper on Recursive Language Models is a good example. The technique wraps the LLM so it treats long context as an external environment and recursively sub-calls itself over snippets. An RLM using GPT-5 scored 91% on BrowseComp-Plus, a multi-million token benchmark where standard base models scored zero. The gain came entirely from how the model was wrapped.

The external data and API layer is what the harness consumes to do useful work. Existing data sources, third-party APIs, internal services, and MCPs all sit here. This is the deterministic part of the harness, and it is entirely software engineering.

The verification layer is what stops bad output from cascading. It covers evals, structured handoffs between agents, and rules about when not to pass unverified data downstream. The classic example is a team of two agents. If agent A hallucinates and the handoff to agent B keeps the hallucinated data in the context, agent B is burning tokens for nothing. The verification layer catches that before it propagates further.

A natural question that comes up is how agents and harnesses differ. An agent is the intersection of model and harness, given a mandate to execute end-to-end. What that intersection needs to look like depends on the task. Hard-verifiable tasks like writing code have a built-in feedback loop, since tests either pass or fail. The harness can stay relatively light because the model has a clear signal to iterate against. Soft-verifiable tasks like filing taxes have no such loop, since you cannot really tell if the outcome was correct until much later. The harness has to do more of the work here to make the result trustworthy. Either way, the agent is the tool calls plus the harness engineering.

2. Design from the job, not the model

Most teams fall into one of two camps when they start building.

The first is what you might call model-first. The premise is that we trust this model and we are going to build everything around it. It is a small subset of teams, but they exist. A year ago, that often meant building entirely around Claude.

The better camp starts from jobs to be done. Before designing the harness, picking tools and choosing context strategies, define what the thing actually needs to do. For a deep research agent, if the job is researching the people on a call, the starting point is realising the actual work is finding LinkedIns, GitHubs, Twitters, and bios. Model choice, context strategy, and tool design all come second to that.

The idea is to design backward from the outcome you need, with the job to be done as the anchor. Everything else, from context management to prompt structure and tool selection, sits downstream of it.

3. Choose to build or compose

Once the jobs are clear, the next decision is whether to build or compose.

Building means writing a harness from scratch for five to seven specific use cases you need to support. Because the scope is bounded, the context windows stay tighter and evals stay clearer. It is the right approach when you already know what you need.

Composing means inheriting from existing harnesses. You pull in an orchestration library, use its preset context management tooling, and assemble what you need. Since you are not reinventing the wheel for problems that already have solutions, speed is an advantage here.

Both approaches have trade-offs. Building takes longer up front, since you are writing more of the machinery yourself. Composing is faster to start, but you sit downstream of someone else’s release cadence, which bites in production. A cleanly designed library can still be difficult to maintain when the package keeps updating fast enough to create dependency issues. Heavier packages also affect build times and runtime, and the flexibility you get is bounded by what the abstractions allow.

Since speed is everything in this space, a practical tactic, regardless of which approach you take, is to lean heavily on open source so long as the license permits it. Shipping a harness in a couple of weeks beats shipping in a couple of months, by which point the trend has usually moved on. Inherit components where they save time, and reinvent only where you have to.

4. Treat evals as a first-class citizen

Only two parts of a harness are actually non-deterministic. One is context management and the other is evaluation. Everything else is basically software engineering.

Context management matters for latency, cost, and the quality of downstream output. If a sub-agent hallucinates and the handoff to the next sub-agent preserves the hallucination, you are burning tokens to amplify bad output. Putting a verification step in place before the handoff prevents this, and so does designing sub-agents so that each one handles a small task rather than making any single one carry a long workflow.

Without strong evals, you end up running in circles. Prompt improvements that should help do not translate into any measurable change and regressions creep in without warning. Tool calls time out, APIs return 429s and 500s, and there is no clean way to isolate whether the problem sits in your harness or somewhere else.

A mix of Behavior-Driven Development (BDD) and Eval-Driven Development (EDD) is helpful here. BDD helps you anticipate user behaviour and encode it into the harness before users find the edges. EDD catches behaviour and quality regressions before users do.

For the deterministic parts of the system, the external services, MCPs, and APIs, the unfashionable answer is to write a lot of tests. Go and look at the test repository of any reasonably designed harness and the volume of tests is significant. Harnesses are, when you strip it down, lines of code telling a model what to do, and you can manipulate that behaviour deterministically using classic software QA. Doing this well makes about 70% of the harness work easier, which frees up your time to spend on the parts that actually need experimentation.

5. Know where a harness will still matter

As models get more capable, two trends run in parallel.

For vertical-specific use cases, harnesses become more valuable, not less. Ramp built Ramp Inspect, their in-house coding agent, despite having access to Claude Code and Codex. It runs in sandboxed VMs, integrates with Ramp’s CI/CD, monitoring, feature flags, and internal services, and at last public report wrote 30% of merged pull requests across their frontend and backend repositories. Ramp built rather than bought because the specific problem they were solving needed integration that off-the-shelf coding agents could not provide. As the bar for vertical fit goes up, the in-house harness becomes the actual differentiator.

For more general use cases, the opposite trend is playing out. The Titans paper from Google Research introduced a neural long-term memory module that scales context beyond two million tokens by learning to compress and recall information at inference time. If approaches like that mature, much of what current harnesses handle through external context management gets absorbed into the model itself. Things like tool calling, memory, and routing logic could move from harness code down into the model layer.

The dividing line seems to be between stable use cases and nuanced ones. Stable use cases will get absorbed by better models and richer application layers over time. Nuanced ones, where context saturation or vertical-specific behaviour matters, will keep needing harnesses around them. A lawyer reviewing case documents still hits context saturation no matter how good the model gets, and the harness around that workflow is what makes the difference between something usable and something that frustrates the user.

How to build the intuition

The best way to learn is by getting hands on.

Pick 3-5 different harnesses with different design philosophies and spend time in each of them. You will start to notice the differences yourself. CrewAI, for instance, is built around role-based agents, while Claude is built for long-running ones. OpenCode is model-agnostic and centered on tool calls and LangChain leans on inheriting components from a large standard library. Sitting with these design philosophies side by side teaches more than any single resource.

A useful tactic when you find a harness you like is to hand the source code to Claude or Codex, have it walk you through the patterns, and then replicate the workflow yourself in code. This helps internalize the design philosophy by writing it out yourself, since harness engineering is mostly philosophy scaled via code.

The other recommendation is to get your hands dirty. Try things, break things, and read the test suites of harnesses you like. The practitioners with strong views on harness engineering have almost all spent time in the source code of several of them.

The vocabulary is going to keep moving. Whatever the field calls this layer six months from now, the underlying question stays the same. Can you give a model data, tools, and an environment, and can you trust it to get a task done autonomously? Whatever you call that thing is your harness.

These sessions happen regularly, and they get into the details most tutorials skip. It’s invite-only for course alumni, but we’ll keep sharing highlights here every week. If you want in, subscribe to The Nuanced Perspective to get notified.

I’ve always been a little averse to roadmaps. They look clean, but they often make learning feel more linear than it really is. Learn this, then this, then this, and somehow you’re supposed to know how to build.

But humans don’t really learn that way. We learn by connecting ideas. We remember through stories, patterns, examples, and relationships between concepts. That’s why mind maps often work better, especially for messy fields like AI engineering.

What I wanted to create was more of a mind map, basically a connected set of ideas you can keep coming back to, even as new concepts come in. Something that builds your foundation instead of confusing you further.

Here’s my attempt to explain what it takes to become a good AI engineer in the most beginner friendly way possible.

And doing all of this through a connected mind-map, over scattered ideas. The goal is to help you build a strong foundation that can take you anywhere, without memorizing endless roadmaps and disconnected concepts.

Here’s a snippet of the topics covered, do watch the video to go deeper (it’s only 17 mins long and super value packed!)

Do check it out, and send it to anyone who’s feeling overwhelmed and struggling to connect the ideas together.

Every major AI lab has shipped voice in the last year. Claude, ChatGPT, Gemini, and most of the consumer apps that wrap around them are all building it out. A demo that wows in a meeting and a system that holds up across millions of real customer calls are two very different builds though, and most teams underestimate this difference.

Vapi is the infrastructure behind some of the most visible enterprise voice deployments today. When Amazon Ring customers call in, they are talking to a Vapi-powered agent that handles 100% of inbound support volume, a deployment that went from zero to production in two weeks.

Vapi employs a stack of decisions to get to production:

1. Understanding which problems are even shaped for voice

2. How to design a conversation when the caller cannot re-read, scroll back, or click anything

3. How to give an agent a usable persona and guardrails

4. How to survive the messy middle of a real call

5. How to see what agents are actually doing once they are live

1. Pick the right problems before anything else

The first question is whether voice is the right interface for the problem at all. A useful matrix for that decision has two axes: impact and feasibility.

Impact captures what the deployment actually unlocks. If you were to replace your existing solution with voice, are you necessarily going to be able to scale your operation? What does that actually look like? What cost savings will you be able to unlock?

Feasibility captures what sits behind the voice agent. How mature are the external systems the agent has to interact with? Are the CRMs clean? Is there a sensible API, or is the company still running on something from 1988 with no documentation? Are there compliance considerations?

Plot a use case on those two axes and the answer becomes clearer.

• P1, very feasible and very high impact, is a no-brainer.

• P2, are things that are really high impact, but maybe we don’t know if the systems we are going to interface with are mature enough for us to actually put into production with voice. Since voice is technically the newest part of the stack a company is introducing, it needs the things it sits on top of to be modern too.

• P3, low impact but high feasibility, is doable but probably will not move the needle.

• P4, low feasibility and low impact, is where voice should be ruled out entirely.

What is useful about this framing is the discipline it imposes. The fact that a process is currently staffed by humans on a phone does not make it a voice-shaped problem.

2. Design voice as its own modality

Once you have identified that voice is the right modality for your use-case, you have to understand that voice and chat are different, and they reward different design choices. In chat, a user can take their time with a wall of text. They have a visual reference to come back to and can scroll to re-read. In voice, none of that holds. The listener is holding everything in memory as it gets spoken, and attention spans are noticeably lower when people are hearing things than when they are reading them. Three design principles fall out of this.

Keep turns short and lead with what matters. Instead of asking a question like

| “I found availability on Thursday, it looks like 2pm is open, would you like me to book that?”

you would say

| “Would Thursday at 2pm work?”.

You give the details right at the start, leave things open-ended, and let the listener engage with you in a back and forth. You also want to make sure that you are repeating facts, because something you said at the start of the conversation may not be in that user’s working memory by the time the call wraps up. So you would remind them:

| “Just to confirm, Thursday at 2pm. Should I go ahead and book that?”.

A lot of companies have also realized that the UX is very different for voice. ChatGPT’s voice mode has a soft limit of around 30 seconds of continuous speech. Responses get truncated and questions are left more open-ended so the user can engage back. The text version of the same product will happily produce paragraphs.

Another key aspect is that some information simply does not work over voice. You cannot read out a URL and expect the caller to start typing it. A calendar link is useless. Even a complicated street address is hard to deliver cleanly. The design question becomes what to surface verbally and what to send by text or email after the call ends.

3. Design differently for inbound and outbound

There are two categories that voice use-cases can often be classified into: Inbound and Outbound. Inbound is when the customer calls in and the agent picks up. Outbound is when the agent initiates and dials someone.

With Inbound, the customer is calling with intent. They have a specific problem or question they want solved, and the agent’s job is to figure out that intent quickly and route the call to whatever can handle it. A customer might call a financial services company and say “my credit card transaction was rejected.” That tells the agent the customer is dissatisfied with something related to credit cards, and from there the call gets handed off to an assistant specialised in those issues.

Outbound is the opposite. The agent is initiating, and the customer often does not know why they are being called. The conversation has to start open-ended. A sales agent calling about an unfinished credit card application might open with “I noticed you signed up for this service but left your application unfulfilled. Did you have any questions I could help you with?”. The customer might come back with a specific concern “I do not know why you are asking for my SSN”, at which point the call can route to a more specialised assistant. Outbound agents have to be built around a list of prompts they can work through, extracting information as the conversation evolves.

Another design difference is that inbound agents almost always need the ability to transfer to a human. Anyone who has called a customer service hotline knows the feeling of realising that AI cannot help and asking for a person. Outbound agents do not need that. If the caller does not want to be called, they hang up, and that is that.

4. Consider your voice persona

With a lot of these conversations, you want to make sure that when the customer is talking to a voice assistant, they are experiencing some semblance of warmth. Whether the conversation topic is something serious, or the customer is more relaxed, you want your agent to be able to take in those signals and respond accordingly. The agent should also speak at a pace that is reasonable for its target demographic and if in certain situations like outbound calls, it needs to be assertive, it should be able to take initiative instead of waiting for the human to tell them what to do. Most of this is unique to voice. Chat does not really have a pace, and chat users do not notice if a bot fails to take initiative the way voice users do.

The default approach to prompting a voice agent is to write what you would write for a chat assistant. Something like:

| “You are a helpful assistant, help the customer with their appointment.”

Even though this kind of prompt works, the agent takes the path of least resistance and responds pragmatically. The way you can work around this is, building a persona for the voice agent with defined identity, style, task, guardrails and tools.

Strict prompting also matters for latency. Teams that treat voice like chat usually leave tool calls to the model’s judgement, the way they might with Claude in a chat context: just trusting the model to call the right tool at the right time. While that works sometimes in voice because the underlying LLMs are the same, what happens is that any gaps in tool calling can be very visibly noticed by customers. A directive like “on these specific trigger phrases, call this tool immediately, do not reason about it” can cut tool-calling latency from around 3 seconds to 1. In voice, 3 seconds of dead air can feel long. The caller has no visual confirmation that anything is happening, and they start wondering whether the call has dropped.

For sensitive data, the system prompt is not enough on its own. Information like phone numbers, dates of birth, account information, needs deterministic guardrails. The safer approach is to prevent that data from being discussed live on the call, from leaking into the call’s context, and from being stored anywhere outside the call’s lifecycle. The system prompt is one layer of defence. Hard rules around what the model can ever see is another.

5. Plan for the edges before they break things

Once a voice agent is in production, the failures can look messier than chat failures, with a lot of them being physical.

Multi-agent handoffs are one of the messy ones. Consider an inbound call about a car insurance claim. The first agent figures out what the customer is calling about and routes the call to a claims-handling agent, who discovers the customer is past their 90-day claim window. The call gets routed to an edge-case agent, who resolves the situation and then has to route back to the claims agent to process whatever can still be processed. These flows are not linear and cycle back and forth. Context has to be carried across handoffs and the system has to know where it is in the workflow at any moment, which agent has visibility into what, and how to hand off cleanly when the situation changes.

Prompt injection also looks different in voice than it does in text. A caller might insist their date of birth is January 1st, 1970 when the database says June 2nd, 2015. Does the agent let what the customer says override what is in the database? The answer has to be encoded into guardrails, not left to the model’s in the moment judgement. The right design assumption is that some callers will do everything they can to break the system.

6. Build observability before you ship

The hardest part of running voice in production is observability, and it is the leg that separates demoware from systems that survive at enterprise scale. Building is easy now. Testing is easy. Where these systems break down, and where teams refuse to commit, is observability and the ability to adapt.

In chat, observability is mostly about the transcript. The session log captures what was said, what tools were called, and what went right or wrong. Voice has multiple modes of failure that do not exist in chat. The transcript is one layer. The audio artifact, the actual sound the caller heard, is another. The public phone network, an enormous system that sits outside the AI stack entirely, is a third.

A transcript can read perfectly while the audio is garbled. Static, clipping, dropped packets, voice synthesis artifacts that turn a clean sentence into something the caller cannot parse. Waveform analysis on the audio can verify that the caller actually heard a crisp response, not just that the model said the right words. Phone-network errors get logged separately, because a call that fails before it reaches the agent still counts as a failed call from the customer’s perspective.

Vapi’s observability stack exposes a lot of this to operators:

• Webhooks fire during the call lifecycle for anyone listening live

• End-of-call artifacts bundle everything for retrospective analysis

• A live call controller lets operators inject messages into a call in real time

• Aggregate monitors surface common failure modes across all calls: what percentage went through, what percentage failed, how they failed, and which modes were most common

An example of voice in the real world

Financial services is one of the spaces where voice agents are getting real traction today. None of the use-cases are glamorous, but they are exactly the kind of high-volume, structured, repetitive work voice is well-suited to.

The customer base in financial services also splits in interesting ways. On one end are young, tech-enabled customers opening their first accounts, comfortable with apps and chat. On the other are customers who have been banking with the same institution for thirty or forty years, who started before the internet was a thing, who are far more comfortable talking to a person than navigating buttons on a screen. For this second group, a well-designed voice agent is often a better experience than the app the bank is trying to push them toward.

Compliance and governance are still being figured out

The piece of the space that is evolving most quickly, and that operators are paying more attention to than they were six months ago, is governance.

Outbound calls often need to start with a disclosure that the call is being recorded. They may also need to disclose that the caller is interacting with an AI agent, either at the start or the end of the call. This varies by jurisdiction and is changing rapidly. The current US baseline is that voice agents cannot lie about being AI when asked directly. If a caller asks whether they are talking to a real person, the agent has to confirm that it is not. The caller also has to be given a path to a human if they ask for one.

Voice realism is the other concern. Modern voice models are now good enough that distinguishing a real human from a synthetic one is genuinely difficult. This raises a concern for impersonation, where someone can clone a familiar voice and use it to talk to a target. The industry is investing in voice fingerprinting, the ability to identify when a voice has been cloned from a real human. Most of that work currently sits with the voice providers themselves rather than with the orchestration platforms that sit above them.

The next phase across the industry is converging on governance and predictability. Tighter uptime guarantees, latency under load, call-level monitoring that treats every conversation as a production workload, stronger guardrails can clear escalation paths to humans when the situation calls for them.

These sessions happen regularly, and they get into the details most tutorials skip. It’s invite-only for course alumni, but we’ll keep sharing highlights here every week. If you want in, subscribe to The Nuanced Perspective to get notified.

Starting this week, we’ll be releasing new videos every week, across different levels and for different kinds of learners. There’s a lot of tool FOMO on YouTube right now. Every week, there’s a new agent, a new coding tool, a new workflow, and a new “must use this” claim. We want to bring systems thinking back into how people learn AI tools. Less chasing every new tool. More understanding what actually transfers across tools. So if that’s your vibe, consider subscribing. We’re making high quality, beginner friendly, engaging videos that help you understand AI tools from first principles.

Every week there is a new AI agent to learn. Claude Code, Codex, Cowork, OpenClaw, the next one shipping while you read this.

Most people respond by chasing each release, memorizing commands, and starting over when the tool changes. They stay busy and never compound.

I made my first YouTube video about the way out of that loop and it’s super beginner friendly.

Here is the idea: Every agent on the market runs on the same structure. A language model in the middle, four components wrapped around it, and one decision that lives with you.

Learn those once and you stop relearning tools. The next product that lands does not catch you off guard.

Video linked above. Tell us what resonates!

-Aish

Most people meet Claude Code as a list of features to learn that just keeps growing. There is CLAUDE.md, rules, skills, hooks, subagents and MCP, with something new shipping almost every week. It is natural to feel like you are perpetually behind.

What makes everything click is realizing these are not separate features competing for your attention. They all answer the same question, one that grows louder the longer you use the tool, of how to keep the right information in front of the model and the wrong information out. That is what context management means, and once you see it, the features stop looking like a menu to work through and start looking like a ladder you climb one rung at a time, moving up only when a specific frustration forces you to.

Why the context window decides everything

Claude Code is an agentic assistant that runs in your terminal. You give it a prompt, it gathers some context, takes an action, checks the result, and keeps looping until the work is done. What separates Claude Code from a chat window is that it can pull context straight from your files and codebase and take action across them, instead of you having to paste everything in by hand.

The catch is that the context window is finite. Every instruction you load competes for the same limited space, and when you overload it the model starts behaving like a person handed a fifty-page brief for a five-minute job. It skims and misses the thing you told it near the top because it was still working through something near the bottom.

This shows up in two ways that look alike at first but come from opposite causes.

• The first is context overflow, which happens inside a single long session. As the window fills with everything you have said and done, the older material gets crowded out, and the model loses the thread of what you told it near the start.

• The second is convention drift, which is almost the reverse problem. Nothing you tell the model gets saved anywhere lasting, so every new session begins from scratch, and a preference you mentioned once is simply gone the next time.

One problem is a window too full to hold everything you need, the other is having nothing at all that survives once the window closes. Every feature that follows works toward the same goal of loading the context that matters, at the moment it matters and leaving everything else out.

Level 1: Raw prompting

The first level is the one everybody starts with. You open the terminal, describe what you want, and it builds, with no setup, no files and no conventions to manage.

This turns out to be the right tool far more often than people like to admit. A quick bug fix, a one-off script, an afternoon spent exploring a codebase you have never seen before, anything that lives and dies inside a single session, all of it is well served by plain prompting. It is the foundation of how you talk to any model, and for trivial work it is genuinely all you need.

However, run anything substantial for long enough and you hit context overflow and convention drift. The agent forgets your patterns, you spend more time correcting it than building with it, and the friction is the signal that it is time to level up.

Level 2: CLAUDE.md, rules, and memory

The second level is where you stop repeating yourself and start writing things down.

CLAUDE.md is the core document, and the easiest way to think about it is as the README for how you and the model work together. It loads in full at the start of every session, which means it should hold only the things that are true every single time. Things like your conventions, a short description of the project, and your non-negotiables. The common rule of thumb is to keep it under a couple hundred lines because a CLAUDE.md that runs long suffers from the same problem as a bloated prompt and the model simply stops listening partway through.

Three principles keep it strong, and they happen to be the same three that make any prompt strong.

• Give it structure with clear headers and short bullets rather than long paragraphs.

• Be specific, since “use two-space indentation” does far more for you than “format code properly.”

• Keep it consistent, because the moment line one and line fifty start disagreeing, the whole file gets weaker.

Rules are how you keep CLAUDE.md from filling up with things that only matter some of the time. They live in a rules folder and load only when they are actually relevant, usually scoped to particular file paths. Imagine your default language is Python but one corner of the project needs TypeScript, which you barely know. You do not want an instruction like “explain TypeScript to me as if I were five” sitting in your context during every session about something else entirely, so you write it as a rule that wakes up only when TypeScript files are in play. You have taken something that does not need to be true all the time and made it true only when it matters.

The third piece is auto memory, and it is the one you touch the least. Claude writes it on its own, picking up build commands, debugging patterns, and the preferences it infers from your corrections. It is useful, but it is probabilistic, which means you are not fully in control of what it chooses to remember. When you want that control, a memory file you edit yourself gives it to you, though for most people the automatic version is good enough to leave alone.

You can write something into your CLAUDE.md and the model will follow it almost every time, but almost is the operative word. These are probabilistic systems, nothing checks the work for you, and you cannot enforce a rule from inside a markdown file no matter how clearly you phrase it. You are still the only quality gate.

Level 3: Skills

If there is one thing to take from all of this, it is skills.

A skill is a workflow you have repeated often enough to be worth saving, and what makes it efficient is the way it loads. Only a short description sits in your context at startup, and the full set of instructions loads only when the skill is actually invoked. That means you can keep a large collection of skills around without drowning your context window, which is the very thing every level so far has been working toward.

Every skill is a single skill.md file. The top of it holds a short description in a YAML header, and below that sit the detailed steps describing what the skill does, what inputs it expects, and where it looks. You do not even have to write any of it by hand. Point Claude at a prompt you keep reusing, or paste in an old chat transcript, and ask it to generate the skill for you.

Say you write LinkedIn posts from session transcripts on a regular basis. Rather than pasting the same long set of instructions every time, you can build a LinkedIn writer skill that takes a transcript and hand back a properly formatted post. So now you can just say “write a LinkedIn post from this” and the skill will load itself. The same pattern covers a data-mapping task that repeats across spreadsheets, a deploy checklist, or PR descriptions in your team’s exact format.

People often ask how skills differ from slash commands, and the honest answer is that Claude Code now treats the two almost identically under the hood, to the point where the official docs have blurred the line between them. The mental model that still helps is that a command is explicit, where you type the slash command and it runs, while a skill is implicit, where you describe what you want and the model infers that the skill applies. They are the same file and the same setup, and the only real difference is whether you summon the thing or it shows up on its own.

Skills are powerful, but they remain probabilistic, so the model follows your protocol almost always while you stay the reviewer.

Level 4: Hooks

Sooner or later, almost always stops being good enough.

When you are working with code, or with anything where a mistake is expensive to clean up, you start wanting determinism, a rule that fires every single time rather than whenever the model decides it seems like a good idea. A hook gives you that. It is a shell command you define that runs at a specific point in the session and lives entirely outside the model loop. This lets it cost zero context and run without fail.

Hooks fire on lifecycle events. A post-tool-use hook runs right after the model edits a file, and a session-start hook runs the instant you open a session. What you do with them ranges from serious to trivial. On the serious end, a use case could be that after editing one file in a project of five hundred, you can run a type check on just that file rather than the whole project, so you catch what you just broke instead of drowning in two hundred unrelated errors. You might also run a security scan before a task is allowed to complete, or load project context automatically whenever a session begins. On the trivial end, a fun example could be that a small sound plays every time Claude finishes a task. It saves nothing and it enforces nothing, but when you have kicked off a job that runs for ten minutes and wandered off to do something else, that little ping pulls you back the moment the work is done. Determinism does not have to be dramatic to earn its place.

A hook makes a single agent more reliable, but it is still a single agent working inside a single context window, and at some point the project is simply too large for that to hold.

Level 5: Orchestration

The fifth level is the one most people never need, which is the most important thing to understand about it.

Think about how a team scales. One person works perfectly well alone, and a group of five still functions as a group, but somewhere around fifty or a hundred people you are forced to start thinking in terms of systems, functions and structure. Claude Code scales along the same curve. When a single agent and a single context window can no longer hold the whole job, you split the work across multiple agents.

There are two modes for doing this.

• Subagents are isolated workers, each with its own context window, its own system prompt, its own tool access, and they report back to a main agent without ever talking to each other.

• Agent teams go a step further and let the agents message each other directly, which you want only when the work genuinely needs them to coordinate.

The reason to reach for subagents comes back to context, same as everything else, because a side task that would otherwise flood your main conversation with output can run in its own isolated context and leave your main agent clean.

There is a real cost to all of this though. Subagents consume tokens, and agent teams consume far more, by some estimates seven or eight times what subagents use. The harder cost is coordination, since running fifty agents in parallel means a single failure can ripple across every one of them. The coordination problem is real enough that managing agents is quietly becoming a genuine skill in its own right.

MCP

One term you will hear constantly is MCP, short for Model Context Protocol. It is not a level but the connection layer. Straight out of the box, Claude Code can see your local files, and MCP is how you give it access to everything else, from your Google Drive to your Notion, Slack and calendar.

People often ask whether MCP is just an API with a fresh coat of branding, and it is not, quite. It sits closer to a standardized wrapper around APIs, so you get the connection between tools without having to do the integration plumbing yourself. The value lands the moment your work lives somewhere Claude Code cannot reach on its own. If most of yours sits in Google Drive, an MCP connection lets it work there directly, instead of leaving you to copy and paste in and out all day.

The real power is in the combinations

So far each level has stood on its own, but in practice none of them work in isolation. The setups that genuinely save you time come from wiring the primitives together, and the logic is the same context principle you have been following all along.

Take CLAUDE.md paired with a skill. Instead of pasting a long brand voice guide into CLAUDE.md, where it eats context and weakens the file, you leave one line saying to follow the brand guidelines and move the full guide into a skill that loads only when you are writing something brand-facing. The always-true instruction stays always-on, and the heavy reference shows up only when it is relevant. Skills and subagents pair the same way. A LinkedIn writer skill can check whether a topic is worth covering, then hand the actual writing to a subagent so the drafts never flood your main session. Hooks and MCP cover the determinism side, where a hook can fire an MCP call the moment someone touches a critical file and send a Slack alert automatically, all at zero context cost.

The features are not a list to choose between. They are parts of one system, and the judgment is determining which part carries which job.

How to choose: a working mental model

The levels are not a ladder you are obligated to climb to the top. They are a set of responses to specific kinds of friction, and the simplest way to hold the whole thing in your head is to think in terms of what you notice and what you add in response.

• When the model gets a convention or command wrong twice and you want it fixed everywhere, that belongs in CLAUDE.md.

• When you keep typing the same prompt, or pasting the same playbook a third time, that should become a skill.

• When you keep copying data out of a tab the model cannot see, connect it through MCP.

• When a side task keeps flooding your conversation with output, hand it to a subagent.

• When something has to happen every time with no exceptions, enforce it with a hook.

• When a second repo needs the same setup, package the whole thing as a plugin.

The single non-negotiable in all of this is a lean CLAUDE.md. You can run the init command in any project and Claude will draft one for you from the files already there, after which your job is to trim it down until it is dense with signal and free of filler. Everything else is need-based, added the moment the friction shows up rather than in anticipation of it.

For example, a preference like banning em dashes could live in CLAUDE.md, a rule, a skill, or a hook, so the question worth asking about every preference you hold is how much the thing actually bothers you, and in what setting. The model not knowing who you are or what you work on is a good example, and that kind of thing belongs at the user level where it stays true across every project you touch. Other things only bother you in a particular context. An em dash in a throwaway response is fine, but an em dash in a LinkedIn post is not, so that preference belongs inside the LinkedIn skill as a hard rule rather than in your global settings. Annoyances that are universal go high and broad, and annoyances that are specific to a context go into the skill or rule that governs that context.

The hardest part of scaling these tools is not the tools. It is the judgment underneath them, the call about what context belongs where and what is safe to hand off. Get that right, and you build systems that keep working even when you are not in the room.

If you want a running start, there are open-source starter setups that will generate a sensible configuration folder for you in about a minute, with separate paths for engineers and for non-technical builders.

References

https://github.com/SethGammon/Citadel

This article grew out of a live Chai & AI session run by Akshat Kharbanda where he worked through the levels of how to actually set up Claude Code. These sessions happen regularly, and they get into the details most tutorials skip. It’s invite-only for course alumni, but we’ll keep sharing highlights here every week.

The first part of this series laid out the 2026 AI agent stack as nine layers, covering what exists at each one, which players to watch, and how the picture has shifted since 2025. Mapping the territory, though, is a different thing from charting a route across it.

The question every builder had after Part 1 is the same one they had going in. Which of these players do I actually pick?

It’s a harder question than it sounds, because every meaningful tool in the stack is racing toward feature parity with every other tool in its layer. Vector databases have already converged, runtimes are converging now, and coding harnesses are next in line.

So “which tool is best?” turns out to be the wrong question, since it assumes a stable answer that doesn’t exist, and you can’t reason about the choice in a vacuum anyway. You are better off going at it the way you would actually build it, from the model up, because each layer pulls you toward optimizing for something different. Get through all of them and the question of how to choose is much easier to answer.

The model matters less than it used to

The model layer comes apart into three questions that build on each other, first the models themselves, then inference, then how you route between them.

Start with the models. Frontier closed-source providers like OpenAI, Anthropic, and Google continue to dominate production, and the two capabilities that once set them apart, multimodal support and reasoning depth, have largely converged. Multimodal went from a Gemini lead in 2025 to a baseline capability in 2026, since nearly every serious model now takes images, PDFs, and other modalities natively. Reasoning depth has jumped just as far. Humanity’s Last Exam, a benchmark built to be brutally hard, sat around 9% for the best model not long ago and is answered at roughly 41% today. What that means in practice is that top-end capability has converged far enough that choosing a model on its benchmark scores is the wrong frame, because you are rarely picking the one model that can do the job. You are picking among several that can.

That pushes the decision down to inference, which is really a question about cost. For the same capability, inference has been getting about 10x cheaper per year. A GPT-4-equivalent answer cost roughly $20 per million tokens in 2022 and runs about $0.40 today, with the cheapest endpoints under $0.10, and Gartner expects inference to be around 90% cheaper again by 2030. There’s a catch worth holding onto, though. Cost per unit of intelligence keeps falling, but overall usage is climbing fast enough to cancel that out, so your bill doesn’t actually shrink even as each token gets cheaper.

Cheap inference only turns into an advantage once you route it well. Mature 2026 systems rarely lean on a single model, running multi-model setups where an orchestrator routes each request by complexity, sending roughly 80% of requests to cheap specialist models and reserving the harder 20% for frontier ones, while the orchestration itself stays on the smartest model in the stack because deciding where to route is the hard part. Specialist models, often distilled from larger ones for reasoning or vision-language work, increasingly stand in for one model trying to do everything, and you can see the same sub-agent pattern inside frontier harnesses like Codex and Claude Code. The shift in 2026 is that frontier models have gotten good enough at predicting which tasks need less horsepower that the routing decision is moving from the human to the model. We are not all the way there, but it’s no longer something a builder has to hand-tune.

What decides this layer in practice is cost, the routing that manages it, and the freedom to change your mind later. Your stack should be able to swap a model without a rewrite.

Retrieval, memory, and the year of connectors

The agent gets its reach into the outside world from the middle of the stack, and capability has largely converged here too, so the decision has moved from raw capability to operational fit.

Retrieval is where that shift already happened. Vector databases have effectively commoditized in 2026, with most of them offering very similar capabilities so the choice now turns on operational fit rather than raw capability. What separates them is which integrations a provider already offers, what infrastructure you are already running, and how cleanly it fits your auth and access layer. Graph databases are taking more share than they had a year ago, on both the consumer and enterprise sides, and Andrej Karpathy’s “LLM Wiki” post popularized the idea of structuring a knowledge base as a graph so agents can find what they need without scraping through everything. Graphs still trail vector stores mainly because of ingestion. Dumping raw data in works for small use cases, but building good-quality graphs at scale is a project in its own right, which is why it’s worth reaching for one only when your retrieval genuinely needs relationships. Sitting alongside both, managed retrieval services like Glean blend RAG with traditional search and hand you the whole pipeline, trading knob-level control for infrastructure you don’t have to run. Hybrid search, which combines vector, keyword, and metadata signals, is now the production default, and naive RAG on semantic similarity alone is rarely shipped.

Memory looks like a separate layer, but in practice it overlaps heavily with retrieval. As a user interacts with the system, you capture what matters, things like their preferences, a project’s status, or an event they mention, and surface it again on demand, which is really external knowledge retrieval pointed at the user. What makes memory its own discipline is that the ingestion side is noisy. Users say a great deal, some of it contradictory, some of it overriding what they said earlier, and a naive store turns all that into garbage retrieval. Every serious memory player is opinionated about how to handle it, with Mem0, Cognee, and Zep running vector plus graphs, Letta using hybrid search, and SuperMemory built specifically for coding. The architectures differ, but underneath they are all RAG with a harder ingestion problem, so the choice comes down to how active the project is and what migrating the store would cost you later.

Tools are where 2026 earned the name “the year of connectors.” Rather than building separate integrations to email, calendar, browsers, and web search, you increasingly reach for an aggregator that wraps many of them behind a single auth layer. Composio exposes dozens of SaaS connectors that way, Browserbase handles browser automation, Firecrawl handles web extraction, and Zapier and Make bring their own large libraries. Picking among them is almost entirely a question of how they connect, meaning which protocols they speak, how their auth model fits your security posture, and how thick the wrapper is, which is another way of asking how hard it would be to leave.

The protocol debate is still open

Part 1 framed MCP and A2A as connective tissue, which is the optimistic read, and the honest one is that the industry is split. MCP has wide adoption among model providers and is the default for most major harnesses, but it hasn’t become the HTTP of agents yet. An active counter-camp argues that CLI tools and existing APIs are enough, and that another standard just means another spec to maintain, with OpenClaw the notable example of a harness that leans CLI-first.

From Google’s agentic commerce protocol to fintech-specific standard, new protocols keep getting proposed for specific verticals too. The trouble is that a protocol only works once the industry aligns on it, and 2026 has far too many options on the table for that alignment to have settled.

Protocol-based integrations are cheap to maintain, but only if the protocol survives, whereas custom integrations have always carried a long maintenance tail regardless of what happens to the standards around them. So lean on a protocol when your team and your customers are already aligned on one, and stay light when you are building into an environment where the question is still unsettled. Bet on the protocols you can afford to be wrong about.

The decision that matters most: the harness

Once you have a model, the components, and a way to wire them together, you reach the choice that shapes everything sitting above it. More than the vector database or the runtime, the harness you choose to live inside is the decision that defines the build.

A harness is the wrapper around the model that turns it into something you can actually work in. Codex, Claude Code, Cursor, and OpenClaw are the products people spend real working hours inside, and the quickest way to see why the choice is structural is to picture building one yourself.

Say you want a basic coding agent that can answer “explain how authentication works in this codebase and who changed it recently.” You wire up a model. It needs to see the directory, so you add a shell-execution tool, then a file-read tool when it has to open things, then ripgrep when it has to search the repo. You teach it Git. You sandbox it so it can’t run git rm -rf. You add compaction because the context window blows up after three turns, then approval gates for anything destructive. Every one of those tools needs its description tuned, and every one breaks in its own way as model versions change.

That’s the build-it-yourself path. The harness path is a single command, codex on the terminal, with all of that wiring already shipped, tested, and opinionated. The opinions won’t fit your use case perfectly, but for most generic needs they hand you a working starting point before you write a line of code.

The bigger shift in 2026 is that harnesses are no longer just for coding. The same primitives, file reads, shell execution, search, sandbox permissions, compaction, turn out to be what you need for a PM drafting PRDs, a trip-planning agent reading PDFs, or a research agent summarizing reports, so Codex and its peers are becoming general-purpose knowledge-work harnesses rather than coding tools that happen to be useful elsewhere.

How far you can take one depends on its customization surface, which spans a spectrum. At the light end is an MCP config that wires a single external tool into the harness, and at the heavy end is a full plugin that bundles skills, scripts, and MCP servers into a distributed package, with skills, hooks, and AGENTS.md or CLAUDE.md files filling the space between.

Choosing among harnesses is the highest-stakes version of every judgment call in this piece. A handful of things separate a safe bet from a risky one:

• How alive the project is. Most of the major harnesses are open-source, including Codex, OpenClaw, deepagents, and LangGraph, and Claude Code itself was leaked into the open. You can download Codex and ask it to explain its own architecture, where it stores plugins and how it manages context, which tells you more about its maturity than any feature page.

• How much it connects to. For a harness, customization and integration are really the same surface, spanning everything from MCP servers and skills to hooks, plugins, and internal tools you wire in through configuration.

• How well it documents itself. The docs need to read cleanly not just for your team but for the coding agents that will write automation against it, and plenty of fast-moving tools ship with documentation that’s already broken.

• How much it locks you in. Plugin formats aren’t standardized, so a skill written for one harness won’t carry over to another.

• How thick the opinionation runs. OpenClaw leans on CLI tools and stays light on MCP while others commit to it fully, so the right answer depends on the stack you already run.

Evals and Governance: proving it works, and keeping it safe

The two vertical rails, observability and evals on one side, governance and security on the other, are where the choosing changes character, because they are less tools you slot in next to the others than systems you build around everything else.

On evals, the 2026 shift is away from one-shot evaluation and toward a continuous improvement loop. LLM judges have become the default for grading agent outputs, mostly because at the edge of model capability it’s hard to specify in advance what good behavior even looks like. Code-based evals still have a place for specific, measurable metrics, but they can’t carry the load on their own. Tools like Arize Alyx now read your observability data as an agent and surface common failure patterns on their own, so you are no longer inspecting traces by hand to find what is breaking.

The real skill, though, lies less in picking the eval tool than in knowing which evals to build, and that knowledge comes from production data rather than a checklist. Multi-agent and multi-turn evaluation is still hard. You can grade against broad goals, like whether the agent solved the customer support ticket, but the specific things worth testing only become clear once you have watched the system fail with real users. An eval suite on its own won’t make a product good. What does is the loop of shipping, watching where it breaks in production, and feeding those particular failures back into the tests. The teams with the most reliable agents in the field are rarely the ones with the largest suites so much as the ones who have made that loop tight.

On governance, the shift has been from optional to non-negotiable. Real-time scanning is now standard, with very small models running in tens of milliseconds to check outputs without adding meaningful latency. Indirect prompt injection, where attacker-controlled content arrives through email, web pages, or documents the agent reads, has become common enough that vetting new data sources is routine practice. Defense in depth, sandbox policies, and approval gates have stopped being merely good practice and become a control-plane decision driven by regulation, with the EU AI Act and its equivalents elsewhere moving the whole area from goodwill into compliance.

With these two rails, the real question is less which vendor you pick than whether you have the capability at all, and what it costs you to be missing it. On every serious enterprise contract now, the procurement conversation opens with audit trail, human override, and policy enforcement, and the model question is often settled by the end of the first meeting.

Putting it together: A Financial Analyst Copilot

Suppose you are building a Copilot for a hedge fund. A user types in a stock ticker, and the agent pulls live price data, scans recent news, runs the firm’s standard thesis checks, drafts a memo, and posts it to Slack at 8 AM daily.

Trace it through the stack and you touch every layer. Slack is the part users see, but the agent loop itself runs elsewhere. Behind it, the harness decides which runtime carries that loop and which capabilities come pre-wired. The tools layer hands the agent a market-data API for the price and web search for the news, plus an internal retriever that reaches into the knowledge and retrieval layer for the firm’s own prior research. You would want it to sharpen over time too, learning which past memos landed and what each user likes to see, and that is the memory layer at work. The model layer pulls everything together into the final memo, with the heaviest reasoning on a frontier model while smaller ones handle routing and citation checks. Underneath it all run the two rails. Observability catches failed tool calls before they reach the user, and governance enforces approvals the moment the agent reaches for a sensitive source.

Almost any agent you build runs through these same layers and no single one of them decides whether it ships well. A model, a harness, tools, retrieval, memory, and two rails of monitoring and governance each come with their own crowd of competing players, and each carries its own trade-off between what’s cheapest today and what’s survivable in eighteen months. Which brings back the question: With this many choices, and every option racing toward parity, how do you actually pick?

Five dimensions that survive feature parity

There is rarely one winning player in any of these categories, and the leader today can be matched within weeks, so the honest answer to “which tool?” is not a tool at all but a way of deciding that outlasts the features themselves. Five dimensions do that work, three of them weighted heavily and two moderately, and the aim isn’t to score every option but to stop arguing about features when the real decision is about something else.

Adoption. Will this tool still be here in eighteen months? Look past the launch posts to the open-source activity, the health of the community, and how many people are actually talking about it on Reddit and X. A niche tool a friend built is fine for a side project, but for production the real question is whether the team has the runway and the community to keep shipping when the next model release breaks something.

Integration. This is where the weights shifted most in 2026. The agent is becoming the interface, and people will spend less time clicking through a tool’s UI than they will having an agent talk to it, so the question stops being whether it has a nice dashboard and becomes whether it speaks the protocols your agents already use. If it exposes no MCP server, no A2A surface, no native function-calling, you are back to writing custom integration code and carrying its maintenance queue.

Learnability. This has two sides to it. The first is how hard the tool is for your team to pick up, and the second, increasingly the one that matters, is how easily your coding agents can. A lot of AI tools today have documentation that is broken, out of date, or missing their newer features, and a coding agent that can’t read the docs can’t write reliable code against them, so the cost shows up in every change you push afterward.

Cost (with lock-in). This isn’t really about the sticker price but what it takes to leave. If a tool is so opinionated or so deeply embedded that switching means a rewrite, then a cheaper-looking option can be the expensive one. A tool that costs more today but lets you swap providers next quarter is often cheaper across the whole timeline than one that’s free until you try to get out.

Flexibility. It’s closely related to lock-in but not the same thing. What you are asking is which abstractions the tool sits on and how thick its wrapper is around them. A thick wrapper is quick to start with and slow to migrate away from, while a thin one takes longer to ship but reads cleanly when you swap the pieces underneath.

None of these five should feel new. They were the quiet logic behind every choice on the way up, from cost and swappability at the model to integration and fit at the vector store to all five at once at the harness. The dimensions just put a name to the pattern you were reaching for anyway.

The next twelve to eighteen months

The five dimensions hold steady but what moves is how much weight each one carries.

Runtimes and harnesses are consolidating. A year ago, picking a runtime was a major architecture decision, and now most are reaching feature parity, with the same thing happening to harnesses like Codex, Claude Code, and Cursor as they commoditize the way models did. The harness is also outgrowing coding, as companies reposition these tools into general knowledge-work harnesses that build your inbox, act as a chief of staff, schedule your meetings, or drive your task tracker, which means people whose day job isn’t shipping software will be working through them too. Adoption weight rises, because picking the harness that survives matters more than picking the one with the slickest feature today.

Protocols will win the wiring, eventually. Some standardization is coming, even if what the standard turns out to be is still an open question, because the direction of travel is one-way. As it firms up, lock-in cost falls on protocol-aligned tools and rises on the ones that route around it.

Memory and retrieval blur into one context layer. The cleanest production teams are already pulling the decision out of application code and into a unified layer that works out at request time whether to pull from an external store or from memory. Flexibility weight goes up, so the move is to pick storage that lets you change your mind without a migration.

Governance moves from optional to mandatory. As AI products go mainstream, audit trails, explainability, and human override stop being nice-to-haves and turn into requirements. Most products aren’t built for this yet, while the ones that win enterprise deals will be.

Eval infrastructure beats shipping first. Being early to an autonomous agent counts for far less than having the eval infrastructure to keep improving one. AI products need break-in time and real signals from how people actually use them before they get good, so the differentiator is the improvement flywheel, building the right evals, seeing whether the product works in production, and looping those user signals back into the evaluation system. Adoption weight on observability and eval tooling rises sharply.

Where to start

If you are at the start of a project, put the framework before the comparison chart. Score the alternatives across the five dimensions, weigh them against your own stack and timeline, and pick the tool whose score holds up when the next round of feature releases lands. Whatever wins on features today probably won’t be what wins next quarter, but whatever wins on adoption, integration, learnability, cost-with-lock-in, and flexibility usually still will.

If you are already mid-project, run the same audit over your most consequential picks and find the one that scored worst on lock-in, because that is where your next painful migration is going to come from. Plan it now, while the cost of it is still manageable.

This piece is a follow-up to The AI Agent Stack in 2026. It grew out of a lightning session that gave practitioners a clear mental map of the stack today and a way to reason about each piece when building an AI product. Similar sessions happen regularly in our Chai and AI community and All our Maven cohort members get access to them, check out our cohort here.

An agent used to be something you ran on your laptop and watched; now it runs on its own in the cloud, kicked off for every issue, reviewing every pull request and monitoring every system. The apparatus around the model even has a name: the harness, the layer that covers everything in an agent system except the model itself.

That shift rewrites the question you have to answer. You are no longer debugging one agent but rather, as Arize CEO Jason Lopatecki put it, running a “fleet” of them, which is closer to managing an autonomous workforce than operating a tool, and the infrastructure for that does not exist yet. Underneath all of it sits a plain and uncomfortable question: once your agents are loose with real users, at that scale, can you even tell when one of them breaks?

Uber, Cursor, BlackRock, LG U+, and AWS are all shipping agents to production. Put a roomful of these teams together and the same finding surfaces from every one of them: when an agent fails in production, a better model is almost never what fixes it. The fix is somewhere less obvious. What follows are the lessons that showed up across all of them, the ones worth keeping in mind before you build evals for anything that runs in production:

1. You cannot test your way to safe. Instrument production and keep domain people close enough to catch what the tests miss.

2. An eval you do not trust is not an eval, and evals lose trust when the work of keeping them current gets skipped, not when the tools fall short.

3. Build evals around how the system breaks, not around the model, because a better model does not fix structural failures.

4. The most durable fixes are organizational, and they slow you down before they speed you up.

5. None of it works if the agent cannot reach the context that is current.

6. The moment agents watch your fleet, you have to watch those agents too.

7. The harness, not the model, decides what a fleet remembers and reuses.

Instruments catch some failures, people catch the rest

A voice agent booking rides for Uber took a call from a mother heading to SFO. Her kid, in the background, said “I want pizza too.” The agent heard it, dropped the airport route, and started navigating toward the nearest pizza place.

Teams often run worst-case analysis and red-team their agents, which catches a lot. But the range of things a real user might say is effectively unbounded, so no test suite covers the whole tail of it. A kid asking for pizza in the middle of a booking might not have been anyone’s risk register, not because it is unimaginable but because the list of things like it never ends. The failure that takes you down in production is usually the one that did not make your list. What caught this one was a number already being watched by a conversational designer: average turns per session, which jumped from around 4 to 15 or 20 and tripped an alert.

BlackRock’s worst near-miss is the contrast. A model advising on portfolio management produced a regulatory rule that did not exist. The guardrails caught it, but only by accident: the checks that flagged the anomaly were not designed to catch fabricated citations, and someone happened to be reviewing by hand and noticed the mismatch. The invented rule was already on its way to a portfolio manager who would have used it to make investment decisions. This near-miss was less a detection failure than a trust failure. The model’s output was treated as reliable when it should have been treated as a claim to verify.

One team caught its failure because it had instrumented the right signal in advance, and the other caught its failure because a person who understood the domain happened to be looking. Neither can be left to luck. You cannot enumerate every way an agent will break, so the work splits in two: instrument the signals that show something has gone wrong at all, and put people who know what a wrong answer looks like close enough to catch the rest. The first is a tooling problem and the second is a people problem, and most of the teams ended up reorganizing around these.

If you do not trust your evals, you do not have evals

Instrumenting the right signals sounds straightforward, but after a year spent trying to close the gap between “we have eval tools” and “those tools actually change what we ship,” Uber’s platform team concluded the tooling was never the barrier. Its agent platform supports teams across the spectrum, from ops teams building their first personal productivity agent to engineering teams building customer-facing ride and delivery experiences at the scale of 200 million consumers across 70 countries. Teams already had eval platforms, both ones Uber built and commercial offerings, so the barrier was never capability. It was friction. By the time a team thought through what datasets to create, what evaluators to write, how often to run them, and where to surface results, they had already shipped, and eval became another day’s problem.

If you have ever set up an eval suite and then watched it go stale within a month, you can probably relate. The datasets stop reflecting what users actually do, the evaluators measure what was easy to measure at launch, not what matters now, and nobody updates them because the team’s attention has moved to the next feature.

Uber’s fix was to make tracing and evaluation a default rather than an opt-in decision. When teams deploy for the first time using Uber’s managed agent solution, they get tracing enabled automatically in every environment, with an Arize workspace configured and permissions already granted. For teams building on Uber’s code-first SDKs, tracing requires two lines of code. Once traces were flowing, the platform team took the agent config plus the trace data and auto-created evaluators specific to each team’s agent. Teams that had never heard of “LLM as a judge” immediately understood when they got a Slack alert saying their tools were contradicting each other 30% of the time.

The deeper change was in the questions. Uber stopped asking “did you hit 90% on your metric?” and started asking four fundamental ones:

1. Do you trust what your evals are telling you?

2. If your evals flagged a regression in your last push, would you actually stop the push?

3. When did you last update your dataset? If the answer is months, that dataset is worse than no signal.

4. How fast can you test a new model? If swapping in a newly released model is a backlog item rather than a 24-hour exercise, your eval infrastructure is too brittle to keep up with the pace of model releases.

Even all of that leaves the hardest part unsolved. When the question was raised on the online eval challenge: in production, you do not have ground truth, so how do you evaluate outputs in real time? BlackRock’s current answer is sampling outputs for human review on a rotation, which they acknowledged is not ideal. They too hoped to find better answers to that problem, which connects back to the broader theme: even the teams with the most resources and regulatory pressure are still iterating on this.

Build the eval around the failure, not the model

Some failures do not move when the model improves, because they come from how the system is built rather than how smart it is.

Cursor runs its own coding product on itself, which gives a clear view of what breaks. Roughly 30 to 40% of their internal code changes now merge without a human reviewer. Instead, they have a system of internal agents that handle PR review, security scanning, CI-failure detection, test coverage, and documentation checks, with a risk score classifying each change so that low-risk ones in well-tested areas auto-approve and higher-risk ones still go to a human. Anyone who has used a coding agent for a week has met the three failure modes they describe. The agent loses the thread of what you already decided and contradicts itself; it takes a vague task and confidently builds the wrong thing and when it cannot figure out how to do something, it hands you code that compiles but fails, rather than admitting it is stuck. Better models do not make these disappear.

The fabricated rule from BlackRock was the same kind of problem from a different angle, a hallucination that no model upgrade reliably removes. So is the drift AWS flagged, where an agent that worked last week behaves differently today because it wrote something to its own memory that changed its reasoning path. This is called temporal drift, and it means the same agent, on the same input, is not a fixed quantity over time.

Cursor runs two kinds of eval continuously.

1. Capability evals ask what the agent can do, start low, and climb as the system improves.

2. Regression evals ask whether it still does what it used to, sit near 100%, and flag any dip as breakage.

They also push verification past prompt engineering by configuring agent machines to mirror a real developer’s setup as closely as possible, down to installed applications. One agent went and downloaded Spotify, because the agent had taken “match the developer environment” literally. This became an internal anecdote to share. They go that far because an agent that can run the actual application, check its behavior through a computer-use loop, and record video of the result as evidence, catches failures that no text-level eval would.

Cursor applies the same idea to the rules its review agents use. When a reviewer leaves a reaction or reply on an agent-generated review, that signal feeds into the agent’s learned rules, and over time it starts catching the issues that the reviewer cares about. The rules are not specified up front; they emerge from the loop.

The hardest fixes are organizational, and they slow you down first

Across the teams furthest along, the fix was not the model. It was a change in who owns the problem and when they get pulled in. And every version of it slowed things down before it sped them up.

BlackRock is the clearest case, and it starts with that fabricated rule. Its clients are institutions deciding what happens to other people’s retirement money, in a domain governed by the SEC, FINRA, and the OCC, so the near-miss was a trust failure it could not afford. BlackRock stopped treating responsible AI as a layer on top of the product and made it the product - with the model, the interface, and the prompts built around a central trust system. The guardrail suite now runs eight categories aligned with the NIST AI Risk Management Framework: five general ones covering toxicity, jailbreak and prompt injection, bias, off-topic content, and PII, and two specific to capital markets. One detects language that crosses into individualized investment advice, with thresholds that move by jurisdiction, and since no industry benchmark existed, the team built it from primary SEC and FINRA rulings in collaboration with their legal team. The second, sensitive content guardrail detects references to confidential or regulated firm-specific information, like questions about whether BlackRock influenced a presidential election. The last category is the person themself, each member on the team and whoever is reviewing and working on these systems, acts as the eighth guardrail.

The harder change was organizational. Legal and compliance used to arrive at the end to sign off on a finished product, which took eight weeks. Now they join the PRD stage. The early version was a tough courtship, since lawyers generate ten edge cases for every two conditions an engineer defines, but once embedded they became the team’s biggest allies, and the metric that tracks time from engineering-complete to compliance sign-off dropped from eight weeks to two. The slow, expensive thing done upfront is exactly what bought the speed later.

The same shift shows up elsewhere. Uber handed eval ownership to conversational designers instead of engineers, on the logic that the people closest to the customer know what “correct” sounds like in ways a platform team cannot specify in advance, and after the pizza incident that team moved to weekly eval and dataset updates run by the designers. In both cases the tools already existed; what changed was who held them and when.

The same logic runs one level up, in who AI-native companies hire. The defining new role is the forward-deployed engineer, a domain expert who sits in front of customers. Harvey hires ex-lawyers. Rogo hires ex-Goldman bankers. Domain experts are expensive and slow to bring on, but hiring them puts the people who know what correct looks like closest to the work. Once you accept the cost upfront, the speed compounds afterward.

None of it works without context

Underneath the eval and observability story sits a quieter problem that decides whether any of it pays off. An agent is only as good as the context it can reach, and keeping that context current is harder than building the agent. Two companies hit it from opposite directions.

PromptQL started from a blunt observation: internal documentation has been failing for thirty years, and agents made it urgent because they depend on current context to produce anything useful. When the context goes stale, the outputs go bad, adoption falls off, and the project dies. The reason docs rot is not that teams do not care, it is that maintaining them has always been someone’s second job, and second jobs do not get done. So PromptQL built a team-facing AI that grows the company wiki as a byproduct of how people already work. It sits in the team’s Slack-style channels, tracks what it learns, and at natural breakpoints offers a short list of candidate facts that a human approves with a click. Over time, these micro-edits compound into a comprehensive knowledge base. Since switching to this in late March 2026, a 70-person team’s wiki grew to nearly 3,000 pages at around 200 edits a day, and internal direct messages dropped sharply because the quick status-check questions now go to the AI. When two engineers were mid-argument about a design decision, the system offered to record the disputed fact, and neither confirmed it, because the fact was the thing they were debating. After they settled it, the system offered again and they accepted. The system recognized the difference between an open question and an established fact, and it waited for the humans to close the loop. If you are building any kind of automated knowledge capture, this distinction between contested and settled knowledge is a design requirement that is easy to overlook.

LG U+, one of Korea’s three major telecoms (30 million subscribers and 150,000 calls a day across 17 contact centers), hit the same context problem compounded by language, format and scale. Their internal knowledge base of 30,000 consultation guides uses legacy Korean operational documents that mix flowcharts, tables, conditional logic, and answers buried in diagrams rather than text. Standard PDF parsers garble these documents because they can not see the layout structure.

LG U+ solved the parsing problem by giving their model two inputs simultaneously: the page as an image and the extracted text. The model reconciles what the words say with how they are arranged. For retrieval, public embedding benchmarks proved useless. Telecom-specific Korean is full of plan abbreviations and domain jargon that no public model has ever seen. “WFDN” stands for “Worry-Free Data with No Extra Charges,” a specific data plan. Picking a model based on high MTEB (Massive Text Embedding Benchmark) scores did not translate to accurate document retrieval, so LG U+ built their own embedding model with hard-negative training (training on documents that look similar but mean different things) and a two-stage domain adaptation process. The resulting model outperformed every frontier embedding model they tested on their domain.

The two responses are opposite ends of the same instinct. PromptQL harvests context from work that is already happening. LG U+ engineers a bespoke stack from the ground up. Either way, generic AI fails on specific problems, and the work of making it specific never finishes.

Another interesting shift is in that of token usage and the focus on efficiency. Earlier on companies had leaderboards where burning more tokens was rewarded and now everyone is trying to cut costs. Cost forced LG U+’s sharpest decision, the squeeze every finance leader is starting to feel as one-time token budgets get blown through and CFOs demand a return on every token. At 150,000 calls a day, running a frontier model on every interaction is not an option, so LG U+ offloads high-traffic tasks to fine-tuned small models in the 0.6 to 4 billion parameter range, on the argument that small language models are the future of agentic AI, and reserves the large models for complex planning. The full system cut call handle time 16% across its contact centers, with answers surfaced in real time so human agents stop putting callers on hold. It runs as a closed loop where production traces feed automated evaluation that finds failure patterns and redeploys improvements continuously.

The watchers need watching too

Solve detection and ownership and you uncover the next layer. Running a fleet means two things you do not have yet: a way to see what the agents are doing, and a harness solid enough to run them for long stretches without you. Both are being built right now.

Seeing them comes first. Once observability is in place, it generates more data than a human can read, so teams build agents to read it, and now they are back at the original problem one level up: how do you know those agents are working? AWS framed this as a liability problem before an engineering one. Ten agents on a developer’s laptop have a small blast radius. A thousand agents running in your cloud account while that laptop sits closed have a large one. You cannot tell a regulator an agent caused the violation. Once you automate a business process with agents you cannot observe, you own the liability either way. The technical reasons it is hard stack up fast: agents are non-deterministic, so the same input twice will not return the same output, and error compounds across every hop from agent to agent to tool because you are multiplying probabilities at each step.

This is the gap Arize built its Observe 2026 launches around, and the lineup reads as a checklist of what the problem actually requires.

1. Signal reads production traces in the background and surfaces emerging failure patterns on its own, grouping related traces into investigation reports with root-cause analysis and suggested next steps.

2. Fleet observability tracks state, token usage, and trajectories across every running agent.

3. Agent orchestration runs fleets of repo-aware agents that investigate failures, propose fixes, and create evals.

4. Harness-as-a-Judge has agentic judges inspect production traces and write reusable labels, instead of leaning on predefined rubrics that go stale.

Paired with AX for capturing and querying traces and Alyx for reasoning over them to find causes rather than correlations, it all amounts to agents watching agents. What nobody has answered is who watches those agents. When it came up, the honest reply was that there is no established practice yet. Keep humans in the loop for observability agents at first, bias toward false positives over missed issues, and tighten as confidence calibrates.

The harness decides what the fleet remembers

The harness is the part that actually keeps a fleet running for long stretches without you. Hermes, OpenClaw, Claude Code, and Cursor’s internal system now differ most in how they handle memory, because the harness is what decides what a fleet remembers and reuses. Nous Research built Hermes as an open-source coding harness, originally an internal tool, because OpenClaw was not holding up over longer sessions, and its answer was tiered memory.

• Short-term memory is a markdown file scoped to the current task.

• Conversation-level memory persists as files about the current session.

• User-level memory is a vector database that holds what you told the agent across sessions, so a detail from a month ago gets pulled back into context when it becomes relevant.

On top of that sit more than 80 built-in tools and as many skills, where a skill is a markdown file the agent writes for itself after finishing a task; if Hermes expects you to repeat a workflow, it saves the skill without being asked and updates it when you change the workflow. The design has a quiet payoff. Prompt caching only helps when the cached content is stable, and because skills are plain markdown loaded as user messages, a repeated workflow hits the cache every time instead of rebuilding context from scratch, which is a real cost and latency win.

An engineer from Hermes, described using Hermes to run research experiments asynchronously by texting the agent on Telegram while out doing errands, getting results back an hour later, and compiling them into a JSON file for a research paper. He also had Hermes scour Facebook Marketplace daily on a scheduled task to find a specific car. A month and a half later, the agent notified him that it found one, and he bought it. He now uses Hermes Agent instead of Claude Code for his production commits. The underlying model is the same, but the harness provides the memory tiers, scheduling, and messaging that make longer-running autonomous work practical. Hermes runs locally, works with any model provider including fully local open-weights models on air-gapped devices, and doesn’t require a subscription. The entire codebase is open source, and his recommendation for adding features is to ask Hermes itself to modify its own code and submit a PR.

Learn from the playbook

Line the talks up and the same progression runs through all of them.

1. Build the agent

2. Find you cannot tell if it is working

3. Build observability

4. Find the observability data is too much to read

5. Build agents to reason about it

6. Find those agents need evaluating too

At every step the reflex was to reach for a better model, and at every step the fix was something else: instrument the signal before you need it, make the eval a default rather than a someday task, hand ownership to the people closest to the domain, design for the failures the model will not fix, and feed the agent context that is actually current.

The thread tying those together is that none of the fixes were quick and none were free. Instrumenting before the incident, embedding lawyers at the PRD stage, hiring domain experts, building a context pipeline that maintains itself: each one slowed the first lap and paid back on every lap after.

So pick one agent you have already shipped and run the same four-question check as Uber on it. If the answer to any of them is “no” or “I don’t know,” you have the same problem every team at this conference was working on. The difference is that they are working on it in the open, and their fixes, tracing on by default, evaluators generated from agent configs, wikis built from conversation, small models trained on the actual domain, judges that write their own labels, are concrete enough to start copying this quarter.

*All images were generated using slides from the conference.

References

https://mitchellh.com/writing/my-ai-adoption-journey

https://openai.com/index/harness-engineering/

https://arxiv.org/abs/2506.02153

This article grew out of insights captured by Prahitha Movva at the Arize Observe conference. Arize is a LevelUp Labs partner and our AI Builder Pass includes $650 in Arize AX Pro credit for 12 months of free subscription, so you can build evals with AI assistance the way it is being done in 2026. Learn more about our maven cohort on Evals and Designing Improvement Flywheels for AI Products.