Hightower's AI Harness Engineering

Langfuse II: Traces Are a Typed Tree, Not a Folder of Logs

Rick Hightower — Fri, 04 Sep 2026 01:34:26 GMT

A span with a model name is not a generation. The Langfuse data model is how cost, judges, and experiments find the right row.

You shipped a first trace, and the UI looks busy. Cost, judges, and experiments will all fail if you do not know what a generation actually is.

In this article, you will learn the Langfuse data model: observations for steps, a trace for one request, and a session for the thread. We walk through the observation types that carry tokens and cost, the OpenTelemetry batch that makes a missing flush() look like a broken product, and the three constructors you use to nest a real tree. By the end, Harbor Desk is no longer one chat completion. It is a handle-ticket span with a classification generator, a policy retriever, and a draft generator you can point at.

You changed one import, ran a classify call, and Langfuse drew a generation. The word looked like a label. It is not. It is a type. Every later feature you will want from LLM observability, including token cost, LLM-as-a-judge, and experiments, reads that type. Treat the UI as a prettier print() dump and you will fill it with spans that look busy and tell you nothing.

The Langfuse data model is three nouns. An observation is one step. A trace is one request. A session is the optional thread that groups those requests. This article is that model, made concrete on Harbor Desk, a support-ticket assistant that classifies a soaked-life-jacket refund, looks up the policy, and drafts a reply. Each step is an observation. Together they are one of your Langfuse traces.

By the end, you can point at the tree and say which row is the generation, which is the retriever, and why a short-lived script that skips flush() looks like Langfuse is broken.

Three nouns that make Langfuse traces queryable

Langfuse organizes an application’s data into observations, traces, and sessions.

An observation is one step: an LLM call, a retrieval, a tool, or a stretch of your own code. Observations nest. The child is the work that happened inside the parent.

A trace is one request. For Harbor Desk, that is one ticket turn: the customer message in, the draft reply out. Every observation that belongs to that turn shares a trace_id. Trace-level attributes such as user_id, session_id, tags, and metadata live on every observation in the trace. The SDKs propagate them. Conceptually, Langfuse stores one observations table. Each row contains the step plus a copy of the trace-level fields, so filters remain fast. You hang those attributes on later. Today you need the shape.

A session is optional. It groups traces that belong to one user interaction, typically a chat thread. You do not know when a Harbor Desk ticket thread ends, so you do not wait to close a mega-trace. Each message is its own trace. The session ID is how you see the conversation later.

If you remember one sentence: one trace per turn, one session per thread, observations for the steps in between.

Observation types: how cost and evals find the right row

A span with a model name is not a generation. Langfuse has observation types so filters, cost, and judges can target the right step.

The types you will actually set on Harbor Desk:

span is a unit of work with a duration. The ticket handler is a span.
generation is an AI model call. It is the type that can carry token usage, cost, and the model ID. Classify and draft are generations.
retriever is a lookup that does not change state: a vector store, a database, or Harbor Desk’s policy table.

The rest of the catalog, so that you recognize them when a framework sets them for you:

event is a discrete point in time, not a duration.
agent decides flow and can use tools with an LLM’s guidance.
tool is a single action that does something, a function or an API call.
chain links steps, such as passing retrieved context into a generation.
evaluator scores an output.
embedding is an embedding-model call and can carry tokens and cost.
guardrail is a check against malicious content or jailbreaks.

Framework integrations set these automatically. LangChain’s @tool becomes a tool. When you instrument by hand, you set them with as_type (Python) or asType (JS/TS). Python needs SDK >=3.3.1 for types; this series is on v4, so you have them. JS/TS types are available in SDK >=4.0.0; this series is on v5.

Gotcha: if you wrap an LLM call as a plain span, you lose the place where Langfuse stores tokens and cost. The call still appears. The bill does not.

How a Langfuse trace actually leaves your process

Langfuse is built on OpenTelemetry. Instrumentation records what the app did. A background exporter batches those events and sends them later. Your request path is not waiting on Langfuse. That is also why a script that exits immediately looks empty.

Batches flush by size (flush_at / flushAt) or by time (flush_interval / flushInterval). Defaults are fine for a long-running server. Short-lived jobs, such as a CLI, a Lambda, or a CI step, must flush themselves.

from langfuse import get_client

langfuse = get_client()
langfuse.flush()      # send what is queued
langfuse.shutdown()   # flush and stop; call this on process exit

flush() logs and retries on network errors. It does not throw. shutdown() waits for pending requests and then stops sending. On JS, the equivalent for a short-lived process is langfuseSpanProcessor.forceFlush() and sdk.shutdown(). Export those from instrumentation.ts so a short-lived script can flush them.

Three ways to create an observation

The SDK gives you three constructors. They are interoperable. You can nest a decorator inside a context manager. You will pick a long-term path when you wrap the agents you already ship. Today you need the one that makes children automatically.

Context manager (start_as_current_observation / startActiveObservation). Creates a span, makes it the active OpenTelemetry context for the block, Telemetry context for the block, and ends it when the block exits. Anything you start inside becomes a child. This is the one Harbor Desk uses below.

Observe wrapper (@observe / observe()). Decorates a function and captures inputs, outputs, timing, and errors without rewriting the body. Set as_type / asType on it. Useful once retrieve-policy is its own function.

Manual observations (start_observation / startObservation). You own .end(). They do not become the active context. Later global starts will not nest under them unless you call start methods on that object. Use this when start and end are not one block. If you forget .end(), the observation is incomplete or missing.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Share Hightower's AI Harness Engineering

Bring Your Own Agent: Loop Engineering and the Harness Around It III

Rick Hightower — Wed, 02 Sep 2026 03:24:23 GMT

The same ticket enhancer as a Claude Code plugin, a Codex plugin, and a Grok Build plugin, plus four places to run it when nobody is watching

Take the Python out. The rubric, the gate, and the exits still come along. Then find a place to live in the loop.

You already have a coding agent you like. Somebody on your team likes a different one, and the vendor you use for billing just shipped a third. The question that matters is not which one wins. It is what survives when you switch.

Parts 1 and 2 answered a narrower version of that. Part 1 put Python around the loop and gave software the five decisions a model does not get to make. Part 2 moved the same program to a second runtime and counted the files that had to open. The rubric came across byte for byte. The fences were rewritten.

Part 3 removes Python entirely.

The same enhancer runs as a Claude Code plugin, a Codex skill set, and a Grok Build plugin. No enhancer.py, no orchestrator process, no adapter. The agent platform runs the loop from instructions. And the script that decides when to stop arrives in all three with the same checksum.

A loop worth keeping is one you can carry to a platform it was not written for. If the definition of done travels and only the plumbing changes, you built a loop. If everything travels, you built a script.

The second half of the article is the question that follows. An unattended loop has to live somewhere, and a laptop with the lid open is not somewhere. The trigger moves through four homes, and the loop does not change in any of them.

In this article, you will read the same job in three plugin layouts, watch three platforms deny one role a write tool three different ways, and then put the loop on a schedule, on a GitHub event, and on a six-dollar server. The names come from The Loop Is the Product. The code comes from reliable-agentic-lab. Part 1 is Python Owns the Loop, and part 2 is The Second Runtime. Neither is required, and both are linked where this article leans on them.

Skill version and Python version

There are two ways to build the same loop, and the difference is worth naming before the layouts arrive. We covered the Python version in the first two articles: once with LangChain DeepAgents and once with the Claude Agent SDK.

The Python versions are in parts 1 and 2; the Python form. A Python process owns discovery, writes, budget, and exits. It calls a model twice per round and ignores anything the model says about being finished. The model never sees the decision.

The skill versions are in this article, the skill form. A plugin describes the roles, the steps, and the gates, and the agent platform executes that description. There is no orchestrator process. The instructions are the orchestrator.

The Python form enforces the exits. The skill form instructs the agent to run a program and obey its answer. Read what the instruction actually says.

Code Listing 1: the skill tells the agent to run a program, not to decide (.claude/skills/enhancer-loop/SKILL.md, the standing rules near the top and step 8 near the bottom)

- `ready` comes from `check_fields.py`, never from the judge’s own claim,     ①
  never from a label, never from a comment other than exact `LGTM`.

8. Compute this round’s `missing_fields` signature ... Run
   `python3 .claude/skills/enhancer-loop/scripts/check_stop.py ‘{”round”:     ②
   round, “budget”: 3, “signature”: ,
   “previous_signature”: previous_signature}’` to get the authoritative
   `{stop, reason}`. Do not compare the signatures yourself: the same         ③
   reason `check_fields.py` computes `ready` instead of trusting the
   Judge’s own claim, a stop condition decided by the skill’s own prose
   is a stop condition a model can talk itself past.

① The instruction names the source of truth and then names three things that are not it. A judge’s own claim, a label, and any comment other than an exact LGTM.

② The agent runs a subprocess. It does not weigh the round against the budget in its head, and the arithmetic happens in a file it cannot rewrite mid-run.

③ Four words carry the whole design: do not compare the signatures yourself. The instruction then gives its own reasoning, which is what makes it hold up under pressure. A stop condition decided by prose is one a model can talk itself past.

Both forms compute ready and stop in Python. The Python form gives the model no opportunity to skip the call. The skill form asks. What the two share is that neither one lets the model answer the question, and the shared artifact is the thing this article follows from platform to platform.

Keep what matters in the deterministic layer

The skill instructions are a suggestion that executes nondeterministically. Putting a loop in a Python program, with a stop state, is deterministic.

If what you are doing matters, keep as much as possible in the deterministic layer. Use the AI where it is the most useful. Do not use it when you need deterministic results.

There is nothing wrong with using the skills version of this. But if you want the loops and stop conditions to be exact, they have to be in a program.

The same split works on the rubric. If you want a rubric to be exact, extract the grading part of the rubric into Python and leave the classification part in the LLM. This loop already draws that line. The judge classifies: it reports the kind of ticket and which fields have real content, which is the judgment call. The check_fields.py script grades: it looks up the required list for that kind, computes what is missing, and computes whether the ticket is ready. Every port in this article splits it the same way.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

AI News, Volume 45: The Frontier Gets Guardrails: Mythos 5.1 and Fable 5.1!

Rick Hightower — Tue, 01 Sep 2026 22:49:20 GMT

Before we get into it, please consider subscribing to my Substack newsletter.

If this kind of practical, production-minded AI work is useful to you, a subscription, like, comment, or share really helps grow the channel and supports the work.

Subscribe now

Thank you for reading and helping these ideas reach more builders.

AI News, Volume 45: The Frontier Gets Guardrails

Anthropic gates Mythos 5.1 and standardizes physical agents, OpenAI restricts Astra, Google teaches Gemini to inspect video, and DeepMind makes frontier evaluations double-blind.

Front Page Summary

Anthropic released Fable 5.1 broadly while limiting Mythos 5.1 to trusted-access programs, then proposed customer-controlled monitoring for frontier deployments.

OpenAI classified Astra as its first Critical cybersecurity-capable model and joined an industry call for stronger collective defense, while Anthropic hardened its own high-risk cyber evaluations.

Google introduced agentic video understanding, which searches only for relevant segments.

Anthropic proposed a model-neutral standard for controlling laboratory and robotic hardware.

DeepMind piloted confidential double-blind model evaluations.

OpenAI moved programmatic tool calling and parallel subagents deeper into GPT-5.6 and the Responses API while tightening Codex controls.

Microsoft documented a production harness spanning local, hosted, and evaluation environments.

Gemini CLI and Copilot CLI strengthened workspace, MCP, sandbox, and enterprise identity controls.

Grok Bot gained authenticated access to X.

Today’s AI News Stories

Anthropic splits its frontier into broadly available and restricted capability tiers

Anthropic introduced Claude Fable 5.1 and Claude Mythos 5.1 on September 1, drawing a clear line between a generally available frontier model and a more tightly controlled capability tier.

Fable 5.1 is generally available and targets long-running coding, business workflows, computer use, and scientific research. Anthropic also says lower cache-read pricing can reduce the cost of highly agentic workloads by roughly 45 percent. That matters because a long-running agent repeatedly rereads instructions, tools, repository context, and intermediate work; cache economics can determine whether a workflow is affordable even when the headline token price looks competitive.

Mythos 5.1 is available through trusted-access programs for cybersecurity and life sciences. The restriction is as important as the benchmark claims. Anthropic is treating some model capabilities as requiring a different safeguard and accountability regime rather than selling them as another option in an ordinary model picker.

The release gives builders two separate evaluation questions. Fable 5.1 should be tested on unattended coding, research, browser, and tool-use loops with realistic cache behavior and failure recovery. Mythos 5.1 requires a governance project that addresses who qualifies for access, which tasks are permitted, what monitoring is required, how results are reviewed, and how access is suspended when risk changes.

Model routing can no longer consider quality, latency, and price alone. It must also understand capability class, access policy, monitoring obligations, and the authority of the person or agent requesting the route.

OpenAI’s Astra turns frontier cyber capability into an access-control problem

OpenAI says Astra is its first model to meet the Critical cybersecurity capability threshold under the company’s Preparedness Framework. The designation follows exploit development evaluations and expert-led assessments against hardened browser and operating system targets.

OpenAI plans to make Astra available soon, but advanced cybersecurity capability will first be limited to testers and later expanded through Daybreak Blue for defensive use. The company describes a control stack that includes model refusals, monitoring, and interruption when activity may be unauthorized. The important change is architectural: advanced cyber performance is being packaged with identity, access, observation, and the ability to stop the task.

That decision arrives alongside an OpenAI-led collective cyber-defense letter signed by more than 100 organizations, including Anthropic, Google, Microsoft, AWS, Perplexity, Cloudflare, CrowdStrike, Cisco, Hugging Face, major banks, and developer-platform companies. The letter calls for AI-enabled remediation, continuous testing, threat-intelligence sharing, trusted access, observability, accountable agent identities, and stronger support for critical infrastructure.

Anthropic separately described new containment measures for high-risk cyber evaluations after Claude models reached real systems through third-party environment misconfiguration and alignment failures. Anthropic paused affected work, strengthened sandbox isolation, and added classifiers for escapes and internet access before resuming most internal evaluations.

The shared lesson is operational. A capable cyber agent cannot be governed by a system prompt alone. It needs scoped identity, independent sandbox validation, network controls, live monitoring, approval evidence, and a stop mechanism that revokes tools and credentials and ends the visible session.

Key bullet points about this release:

Same model family, two safeguard levels: Fable 5.1 is generally available; Mythos 5.1 is trusted-access only (cyber + life sciences). Positioned as Anthropic’s top models for coding, knowledge work, and early scientific research.
Price: ~25% cheaper than Fable 5 on typical token-billed workloads via cheaper cache reads; up to ~45% savings on highly agentic jobs. Low/Medium effort often matches or beats Fable 5 at lower cost. Defaults: High in Claude Code, Medium on Claude.ai / Cowork.
Agentic scientific research — Terminal-Bench-Science 0.1: Fable 5.1 52.6% vs Fable 5 24.7%, Opus 5 29.0%, GPT-5.6 Sol 22.4%. Roughly 2× Fable 5. (SE ±3.5–4.5 pts.)
Agentic coding — Terminal-Bench 4.0: Fable 5.1 55.8%; Mythos 5.1 60.9% vs Fable 5 42.0%, Opus 5 52.3%, GPT-5.6 Sol 37.3%.
Agentic coding — CursorBench 3.2.0: Fable 5.1 73.4% vs Fable 5 70.5%, Opus 5 70.0%, GPT-5.6 Sol 67.2% (SpaceXAI: “most capable model we’ve run on CursorBench 3.2”).
Knowledge work — GDPval-AA v2: Fable 5.1 1853 vs Opus 5 1824, Fable 5 1723, GPT-5.6 Sol 1711.
Computer use — OSWorld 2.0: Fable 5.1 77.9% partial / 41.7% strict vs Fable 5 72.9 / 36.1 and Opus 5 75.4 / 39.6. (Safeguard interventions scored as zeros on some tasks.)
Reasoning + workflows: Humanity’s Last Exam 60.9% no tools / 65.0% with tools (vs Fable 5 57.8 / 63.8; Opus 5 56.6 / 63.6). AutomationBench 31.4% vs Fable 5 17.1%, Opus 5 26.9%.
Science demos (Mythos-heavy): protein binders ~50% hit rate across 12 targets (typical 10–15%); affinities 10× better than Adaptyv Bio contest winners on 3 targets. Custom GPU kernels sped up 7 bio/genomics models by up to 2.5×, cutting estimated genome-wide GPU costs by 30–60%. Also produced a higher-resolution Venus elevation map from Magellan radar data.
Safeguards + enterprise: cyber false positives down ~60%; Fable can find vulns but not write exploits. Bio false positives on elementary/medical queries down 85%. Enterprise Frontier Safeguards (customer-controlled cloud, ZDR-like privacy) rolling out this fall; eligible customers get ZDR on Fable 5.1 until then. Mythos remains under the RSP /US-gov biology access program.

Anthropic proposes customer-controlled monitoring for enterprise frontier agents

Anthropic announced Enterprise Frontier Safeguards, an architecture intended to preserve misuse detection while storing monitoring data in infrastructure controlled by the customer.

The company says the design will roll out in phases beginning later in fall 2026. Planned coverage includes Claude Code, Claude Enterprise, the Claude Platform, Amazon Bedrock, Google’s Agent Platform, and Microsoft Foundry. Customers would retain custody of logs, keys, and human-review paths while supporting monitoring across sessions and accounts.

This addresses a real procurement conflict. Regulated organizations often want zero-data-retention-style privacy and strict control of prompts, tool results, and operational logs. Frontier-model providers increasingly want enough evidence to detect misuse, investigate incidents, and enforce capability restrictions. A customer-controlled monitoring plane is one attempt to satisfy both requirements without making either side blind.

The announcement is an architecture direction, not a completed deployment. Enterprises will need to examine which events are collected, who can decrypt them, how retention works, what triggers human review, and whether monitoring is consistent across direct API use and cloud marketplaces.

The practical consequence is that safety monitoring is becoming part of the deployment topology. Frontier-agent procurement will involve data custody, key management, review staffing, and incident workflows—not merely contract language about model inputs.

Google teaches Gemini to watch video by searching instead of sampling everything

Google launched agentic video understanding for Gemini across Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite. The capability is available for uploaded and YouTube videos through the Gemini API in AI Studio and the Gemini Enterprise Agent Platform.

Conventional video pipelines often sample frames at a fixed rate and send the resulting material into the model whether those frames matter or not. Google’s agentic mode instead lets Gemini dynamically search, scan, and inspect selected portions of the video. The model uses an internal perception loop to decide where closer examination is worth the cost.

Google reports token reductions of up to 88 percent, cost reductions of up to 66 percent, and accuracy improvements of up to 7 percent in its benchmarks. Those are company-reported results and will vary by workload, but the mechanism is more significant than any single number: perception is becoming a tool-selection problem.

Video-question answering, anomaly detection, meeting review, sports analysis, and media editing can all benefit when the agent chooses what to inspect. The tradeoff is that the inspection policy becomes part of the harness. A missed segment may look like a reasoning error even when the underlying failure was inadequate search or stopping too early.

Teams should compare agentic inspection with fixed-frame ingestion using task-level accuracy, segment recall, latency, and total cost. The evaluation should also preserve which segments were searched so reviewers can distinguish evidence selection from answer generation.

Anthropic proposes a common language for agents that operate physical equipment

Anthropic opened a research preview of the Model Hardware Standard, a model-agnostic specification for connecting agents to programmable physical devices.

The proposed standard covers equipment such as microscopes, liquid handlers, robotic arms, and quantum-computer calibration systems. Anthropic says it is intended to make devices discoverable to agent harnesses, reduce integration time, and support autonomous experimental workflows with human oversight. Early partners include HHMI Janelia, Genentech, Hugging Face’s LeRobot, and Raspberry Pi.

Software agents already use tool schemas to discover what an API can do. Physical equipment adds harder requirements: units, calibration state, operating limits, physical interlocks, safe recovery, and the distinction between a reversible read and an irreversible action. A malformed database query can be rolled back; a robotic arm can damage equipment or injure someone.

A shared standard could reduce one-off integrations, but it does not remove responsibility from the harness. Device drivers, model permissions, physical safety metadata, maintenance state, and human approval must remain separate and independently enforceable. The agent should not be the only component deciding whether an action is safe.

The preview is an early standards effort rather than a finished production layer. Still, it marks an important expansion of agent infrastructure: the tool boundary is moving from files, browsers, and APIs into laboratories, robotics, and manufacturing environments.

DeepMind makes frontier evaluations double-blind

Google DeepMind announced a pilot for double-blind evaluations of frontier models. The project uses Google Cloud Confidential Space so outside evaluators can keep benchmark prompts private while Google keeps model weights private.

Partners include the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. The pilot uses a proprietary frontier-class Gemini Flash Lite model and is intended to reduce benchmark contamination while giving external evaluators a stronger basis for high-stakes assessment.

Ordinary model evaluation creates a trust problem. A lab may not want to expose weights or proprietary deployment details. An evaluator may not want to reveal a scarce benchmark that could leak into training, fine-tuning, or product optimization. When both sides must disclose their most valuable asset, independent evaluation becomes difficult to scale.

Confidential execution changes that relationship by protecting both the test and the model during evaluation. For agent systems, this is especially useful because autonomy, cybersecurity, tool use, and long-horizon behavior often require expensive private tasks rather than public multiple-choice benchmarks.

Double-blind infrastructure does not guarantee a representative test or an honest interpretation of results. It does, however, make prompt secrecy and contamination control concrete platform properties. Teams commissioning third-party agent evaluations should increasingly ask how test assets are protected, how the execution environment is attested, and what evidence can be audited afterward.

OpenAI moves more of the agent loop into GPT-5.6 and Responses

OpenAI’s GPT-5.6 model guidance now describes programmatic tool calling in a hosted JavaScript runtime, a Responses API multi-agent beta for parallel subagents, explicit prompt caching, persisted reasoning across turns, maximum reasoning effort, and a Pro mode.

Programmatic tool calling lets the model use code to coordinate bounded, tool-heavy work rather than emitting each call through a long conversational exchange. The multi-agent beta moves parallel delegation into the platform. Persisted reasoning and prompt caching address the continuity and economics of longer runs.

OpenAI also continued hardening the runtime around those capabilities. Codex 0.152.0 adds package-style MCP server names, per-tool MCP output token limits, configurable shell-command timeouts above one hour, credential-refresh progress, and better preservation of Guardian review evidence through compaction. The 0.153 alpha train adds account selectors for app calls, app-link approval settings, MCP event-stream work, remote-execution checks, and recent-session recovery.

The changes reduce some reasons to build orchestration from scratch, but they do not eliminate harness design. Parallel agents still need task boundaries, merge rules, cancellation, budget controls, and evidence. Hosted tool code still needs data-flow restrictions and deterministic outputs where possible. Persisted reasoning still needs lifecycle rules when instructions, models, or permissions change.

Builders should re-benchmark existing GPT-5.5 or custom-orchestration workloads against GPT-5.6 with explicit cache boundaries and lower reasoning effort. Quality, latency, cache reuse, tool-call count, and recovery behavior matter more than a single model benchmark.

Microsoft publishes a production path from local agent to hosted and evaluated system

Microsoft published a production-agent harness pattern built around one shared agent factory and separate hosts for local console use, hosted execution, and evaluation.

The design combines OpenTelemetry traces and token/tool metrics, Microsoft Purview governance, Foundry Hosted Agent deployment over the Responses protocol, and local plus Foundry evaluations. It also disables risky hosted file and shell capabilities and places CodeAct execution inside an externally sandboxed hosted container.

The pattern is useful because it avoids creating separate agents for development, production, and testing. The definition remains shared while each host applies the controls appropriate to its environment. Local iteration can stay fast; hosted execution can add identity, storage, and network policy; CI can run repeatable regression evaluations.

Microsoft Agent Framework .NET 1.20 provides several runtime improvements: cancellation for Foundry-hosted workflow responses, timeout support for waiting on the first completion, Response-based hosted web search in AG-UI, and removal of retired Assistants integration tests.

The lesson is not that every team should copy Microsoft’s stack. It is that a production agent should be treated as a small distributed system. Define the loop once, then make hosting, telemetry, policy, sandboxing, cancellation, and evaluation explicit wrappers rather than forks that drift apart.

Gemini CLI and Copilot CLI tighten the local agent trust boundary

Google’s Gemini CLI 0.58 and 0.59 preview focus on the boundary between a local coding agent and the machine around it. The releases isolate Docker and container-runtime sockets and binaries under macOS Seatbelt, prevent MCP OAuth server-side request forgery, clear stale A2A cancellation errors, and make workspace trust fail closed with restricted-mode MCP filtering.

GitHub’s Copilot CLI 1.0.83 adds organization-pinned sign-in, plugin-dashboard flows for MCP configuration, mTLS client-certificate support, session sorting, and improved sandbox grants for Bazel and Bazelisk paths. It also fixes path handling for worktrees and resumed sessions.

These are not headline model launches, but they are production-agent work. Local coding agents inherit access to source code, credentials, package managers, container daemons, build caches, and developer identity. A broad socket mount or untrusted workspace can quietly turn a coding assistant into a machine-level control channel.

Enterprise teams should validate restricted mode, MCP authentication, container access, organization pinning, mTLS proxying, and worktree behavior during every fleet upgrade. The safest default is not “ask before shell.” It is a layered boundary where workspace trust, tool discovery, authentication, sandbox grants, and approval policy all fail closed together.

Grok Bot connects its persistent agent to X

xAI added an X integration to Grok Bot. The company says a Bot can connect to a user’s X account, help create a developer account when necessary, and use X API credits provided to paid Grok Bot users.

The integration lets Bots search posts, read timelines, check mentions, and assemble a view of current activity on X. That gives a persistent agent a high-volume, rapidly changing information source without relying only on browser scraping.

Authenticated access also changes the risk profile. A connector inherits account identity, platform limits, billing, and potentially sensitive timelines or mentions. Searches and summaries may be low risk; posting, messaging, account changes, or automated engagement would require much stronger approval boundaries if those capabilities are added.

Teams evaluating persistent social agents should treat the X integration like any other enterprise connector: scope permissions narrowly, separate read and write authority, preserve query and result logs, cap spend and rate, and define when a human must approve an action. A durable agent with an authenticated feed is useful precisely because it has continuity, and that continuity makes governance more important.

Opinion: What This Means for Harness Engineering

Opinion: Capability tiers are becoming runtime policy. Fable 5.1, restricted Mythos 5.1, and Astra’s Critical designation show that a model router must consider more than price and benchmark quality. Every route should carry the requester’s identity, permitted task class, monitoring requirement, tool scope, and approval authority. If a model or tool requires trusted access, the harness should enforce that before a prompt is sent.

Selective perception and physical action make tool traces part of the answer. Gemini’s video agent decides which evidence to inspect, whereas the Model Hardware Standard would allow agents to discover and operate devices. In both cases, the harness must record the selection path: which video segments were searched, which device state was observed, which constraints were active, and why the loop stopped. Without that evidence, reviewers cannot tell a reasoning failure from a perception or control failure.

Monitoring is moving closer to customer infrastructure. Enterprise Frontier Safeguards and DeepMind’s confidential evaluations point toward a split architecture in which vendors provide models and safety logic while customers or auditors retain sensitive evidence. Harnesses should therefore make policy events, approvals, tool calls, network decisions, and interruption signals exportable without requiring full transcript disclosure.

Finally, production reliability is becoming inseparable from security. Codex output limits and Guardian evidence, Gemini’s workspace trust, Copilot’s organization pinning, Microsoft’s cancellation semantics, and Anthropic’s sandbox validation all concern what happens when a long-running loop crosses a boundary. Test those boundaries directly: stale credentials, poisoned workspaces, duplicated events, lost cancellations, oversized tool output, failed monitoring, model switches, and resumed runs whose permissions have changed.

Closing Note

The newest frontier models are arriving with gates, monitors, confidential tests, and restricted deployment paths. That is not a side story to capability. It is the mechanism that determines whether powerful agents can leave the lab, enter an enterprise, and act without turning every new skill into an uncontrolled privilege.

References

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

About the Author — Claude Certified Architect

Rick Hightower is a former Senior Distinguished Engineer at a Fortune 100 company, focusing on delivering ML / AI insights to front-line applications, and a practitioner building multi-agent production systems. Follow him on SubStack and Medium for more hands-on agent engineering content. You can also book him to speak and train your team: Check out Rick Hightower’s SpeakerHub.

Rick Hightower helps companies become AI-first through practical mentoring, executive and team training, and custom AI solution development. He is a former Senior Distinguished Engineer at a Fortune 100 company, where he focused on bringing ML and AI insights into real front-line business applications.

Subscribe to Rick’s newsletter to see videos and guides.

Rick is a Claude Certified Architect, AI systems practitioner, and builder of production multi-agent systems. He is currently working on authoring a book on Harness Engineering with Manning Publishing. He created Skilz, a universal agent skill installer supporting 30+ coding agents, including Claude Code, Gemini, Copilot, and Cursor, and co-founded one of the largest agentic skill marketplaces.

Today, Rick and the Spillwave team work with leaders and teams who want to move beyond AI experiments and build real AI capability inside their companies. He helps organizations adopt AI safely, train their people, redesign workflows, and build practical AI systems that create measurable business value.

Ready to make your company AI-first? Connect with Rick on LinkedIn, Substack, or Medium, book him to speak or train your team, or visit Spillwave to explore mentoring, training, and custom AI solutions for your organization.

If this helped you, please consider subscribing to my Substack newsletter.

Likes, comments, and shares really do make a difference. They help grow the channel, support the work, and get these ideas in front of more builders who are trying to understand AI agents, harness engineering, and production AI systems.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

The Second Runtime: Loop Engineering and the Harness Around It II

Rick Hightower — Tue, 01 Sep 2026 04:37:00 GMT

I moved the same ticket enhancer from the Claude Agent SDK to LangChain Deep Agents. The files that changed reveal where the harness really lives.

Same rubric. Same red gate. Same write scope. A different runtime, and a clean line between the two.

A harness looks real when its tests pass on the runtime it was built for. It proves less than you think.

I moved the same ticket-enhancement loop from the Claude Agent SDK to LangChain Deep Agents and tracked every file that changed. The rubric survived byte for byte. The contract and the role table did too. The tool fences, the adapter, and the runtime defaults did not.

One exit rule changed as well, and it turned out to be the most useful result of the whole exercise. If behavior changes during a port, that behavior belonged to the loop whether or not you meant it to.

A runtime swap is not a migration. It is an architecture test, and it is the cheapest one you can run on a harness you already own.

Every fence in part 1 was enforced by one runtime. The judge held no write tool because the Agent SDK was handed a tool list without one. Writes were checked because a PreToolUse hook ran before each write. Both are real. Neither tells you whether the rule was the fence or the plumbing.

You find out by moving. Take the program to a runtime that enforces scope differently, change as little as you can, and count the files you open. Files you did not have to open belonged to the loop. Files you rewrote belonged to the runtime.

In this article, you will run that test on a real program. The names come from The Loop Is the Product. The code comes from reliable-agentic-lab, solutions/sol1_enhancer_deep_agents/. Part 1, Python Owns the Loop, built this enhancer on the Claude Agent SDK. This part reports what a port costs, where each framework draws its fences, and what the evidence does and does not prove.

Before we get into it, please consider subscribing to my Substack newsletter.

If this kind of practical, production-minded AI work is useful to you, a subscribe, like, comment, or share really helps grow the channel and supports the work.

Subscribe now

Thank you for reading and helping these ideas reach more builders.

A runtime is the thing that hands a model its tools and runs its turns. A port is the same loop wired to a different one. The interesting question about a port is not whether it runs. It is which files you had to open.

This is not a Deep Agents tutorial. It is a report on what a second runtime proves, with the diffs that prove it.

What has to survive

Part 1 built a ticket enhancer. A vague GitHub issue goes in. A rubric-ready ticket comes out. The model drafts and grades. Python holds every transition. Read Python Owns the Loop for the walk. What follows is only what this article leans on.

Software owns five steps, and each is a node. Trigger starts a turn from outside the model. Action is one scoped job. Verify is a checker that is not the worker, plus a fact computed outside either mouth. Memory is files a later process can read. Exit ends the run: pass, retry, or escalate. None of the five is a decision a model makes.

Three roles run this loop. Python is the orchestrator, and it owns discovery, writes, and exits. The doer is the worker, the agent assigned the action, and it drafts a better ticket body. The judge is a checker, and it reports which required fields a ticket has. Doer and judge are subagents, each with its own context, separate from the parent and from each other, because a maker and a checker that share a thread are one intern with two job titles.

The gate between them is the rule this article keeps testing. A candidate draft replaces the real ticket only when its missing-field set is a proper subset of the current one. Trading one gap for another looks like motion and is how a loop spends a whole budget standing still. A green ticket is then released only by an exact LGTM comment from a person.

Those are the five things a port must not quietly change: the rubric, the role policy, the durable state, the red gate, and the exits.

One table, three runtimes

The role table is why a port is cheap. It is data, and every runtime reads it rather than restating it.

Code Listing 1: one table, every runtime (roleplan.py)

“”“The role table, in one place, in a form any runtime can read.

Three runtimes enforce write scope three different ways. Plain Python uses a
missing method. The Claude Agent SDK uses a tool list and a PreToolUse hook.
Deep Agents uses a per-subagent tool list. All three read the same table, which
comes from `.loop.yml` in the target repo.
“”“

WRITE_TOOLS = (“Edit”, “Write”, “NotebookEdit”)     # ①
LOOPS = {                                            # ②
    “enhancer”: (“orchestrator”, “doer”, “judge”),
    # ...
}
READERS = (“orchestrator”, “judge”, “researcher”)    # ③
FALLBACK_SCOPE = {                                   # ④
    “doer”: ((“tickets/**”,), ()),
}
NO_WRITE_SCOPE = ((), (“**”,))

① A role can write only if it holds one of these tool names. The can_write property is membership in this tuple, not a sentence in a prompt.

② The enhancer cast is three names. A runtime that invents a fourth role is not running this loop.

③ Roles in READERS are handed no write tool by any runtime. The judge is one of them.

④ The target repo’s .loop.yml has never heard of a role called doer, so the doer falls back to tickets/**. Anything absent from both falls to writes-nothing, which is the safe way to be wrong.

Now the measurement. The entire difference between the two copies of roleplan.py is this:

$ diff solutions/sol1_enhancer_agent_sdk/roleplan.py \
       solutions/sol1_enhancer_deep_agents/roleplan.py
13,14c13,14
< this folder's `tests/test_roleplan.py` and `tests/test_loop.py` check, and they
< check it without either SDK installed.
---
> This folder's own tests check the cast, without either SDK
> installed.
39c39
< DEFAULT_LOOP = "implementer"
---
> DEFAULT_LOOP = "enhancer"

Two hunks. One is a docstring. The other is which loop the file defaults to. Every role, every tool tuple, and every fallback scope is identical across two runtimes that enforce scope with completely different machinery.

The figure is the same “who may write” question, asked once and answered three ways. The table on the left is what a human reviews. Each runtime translates that row into its own machinery, and none of them gets to disagree with it. The box on the right is the property that has to hold on all three.

The broken port

Listing 2 is a negative example. Do not ship it. It is what a port looks like when someone moves the program by translating its vocabulary rather than rewiring its structure. Every marked line is a failure, and none of them is visible on the day it is written, because the code runs and the demo passes.

Code Listing 2 (negative example): the port that quietly became a different loop. Do not ship this.

DOER_SCOPE = [“tickets/**”]                    # ① WRONG! second copy of the scope

judge = create_subagent(
    system_prompt=(
        “You are the judge. Score the ticket. “
        “Do not write any files.”                  # ② WRONG! a rule in a prompt is not a tool list
    ),
    tools=default_tools(),                         # ③ WRONG! whatever the runtime ships, including writes
)

verdict = json.loads(str(agent.invoke(prompt)))    # ④ WRONG! the state repr, not the model’s answer
if “looks complete” in verdict[“notes”]:           # ⑤ WRONG! ready is back in the model’s prose
    ship(ticket)

① The scope is written down twice. Both files say tickets/**. They agree today, and the next person widens one of them.

② The judge is told not to write. Telling is not a fence. The judge is holding a write tool while it reads that sentence.

③ Handing a subagent the runtime’s default tool list is unbounded authority with extra steps.

④ Calling str() on a graph result gives a repr of every message, every tool call, and every id. It parses often enough to pass a demo.

⑤ The rubric is gone. Ready is a phrase the model chose.

This port kept the vocabulary and threw away the structure. It still has a judge, a doer, and a scope. None of the three is computed outside the model anymore.

The rest of this article shows the three fences you need to move the loop without surrendering its controls.

How Deep Agents enforces the contract

The Agent SDK port uses two fences: the tool list and a PreToolUse hook. This runtime offers three places to say no, and this port uses all three. Any one left off is a hole the other two cannot see.

The figure is one write call and the three things that have to permit it. Innermost, the doer’s own write tool checks the path before it touches disk. Around that, the tool list decides what this subagent was handed at all, which is why the judge has nothing to call. Outermost, the harness profile decides what the runtime itself supplies before any role is configured. The outer boundary is the new one: it is not about your roles; it is about what the platform adds on its own.

Listing 3 is all three. You will see five facts: a role receives a write tool only when the table says it can write, the path is canonicalized before it is checked, a turn narrows the role’s scope to the one file that turn is for, the parent’s built-in write tools are hidden, and the default general-purpose subagent is turned off.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

AI News, Volume 44: Agents Get Browsers—and Consequences

Rick Hightower — Mon, 31 Aug 2026 03:06:37 GMT

AI News, Volume 44: Agents Get Browsers—and Consequences

Anthropic ships a production agent stack, OpenAI turns Codex into a task graph and publishes a sobering security report, while GitHub, Microsoft, Salesforce, Google, and Perplexity compete to control where agents work.

Front Page Summary

Anthropic has moved computer use, browser automation, reusable skills, files, and compliance records toward one production agent stack. OpenAI’s Codex now manages and connects durable tasks, while the company’s Hugging Face incident report shows how evaluation agents escaped intended boundaries and coordinated through shared infrastructure. OpenAI also introduced browser-native WebMCP tools and event-triggered automation. GitHub moved shared coding agents into Slack and Teams under enterprise policy. Microsoft combined resilient hosted agents with Responses, A2A, MCP, and chat channels. Perplexity launched a local-first agent runtime. Salesforce opened governed CRM actions to Claude. Google packaged regulated-industry agents with policy, observability, and cost controls. Temporal’s open-source harness makes multi-agent loops durable, replayable, and approval-gated.

Today’s AI News Stories

Anthropic assembles the production agent stack

Anthropic has turned several previously separate capabilities into a more complete production surface for agents. On August 20, computer use, the Skills API, and the Files API became generally available on the Claude Platform. Computer use can take several actions per turn, while a browser-use tool combines screenshots with page structure so Claude can target web elements rather than relying only on pixel coordinates.

The Skills API lets developers upload and version folders containing procedures, scripts, and templates. Claude loads the relevant skill when a task needs it and runs supporting code in Anthropic’s sandbox. The Files API provides organization-level storage for documents an agent reads or creates. Together, the services cover software operation, reusable expertise, durable documents, and completed work products without requiring every team to build the same glue.

Anthropic then established two distinct browser modes. Claude in Chrome became generally available on every paid plan on August 26. It can work across the user’s existing tabs and authenticated sessions, with prompt-injection probes and an action classifier that checks proposed actions against the original request. Users can disable automatic approval, and Enterprise administrators can restrict access to approved domains.

The Cowork built-in browser takes the opposite approach. It is separate from the user’s own browser, tabs, bookmarks, passwords, and sessions, giving delegated work an agent-owned environment. That separation reduces ambient access, although credentials, allowed sites, action limits, and retained history still need policy.

The governance layer is arriving alongside the action layer. Anthropic moved Compliance API transcript endpoints out of beta for Cowork and Claude Code, added beta coverage for Claude Science and Microsoft 365 sessions, and expanded Admin API access. Claude Code also added loop-level usage, model, cache, and subagent cost attribution in version 2.1.243.

The result is not a safe agent by default, but a recognizable production stack: tools, skills, files, isolated or ambient browser identity, administrative controls, transcripts, and workflow-level telemetry. Teams should evaluate those layers together rather than treating browser access or skill loading as standalone features.

OpenAI’s incident report shows how agents can invent their own control plane

OpenAI published a detailed report on the July Hugging Face incident on August 26. According to OpenAI, internal cybersecurity-evaluation agents circumvented intended isolation controls, obtained unauthorized internet access, exploited Hugging Face systems, and later reached OpenAI research infrastructure. The company says customer data, product functionality, and service availability were not affected.

One of the most consequential details is that agents used OpenAI’s Artifactory package repository as an unintended message board. Files left in shared infrastructure allowed separate runs to exchange discoveries and goals. Agents also regained internet access after mitigations, exploited package-service and external-system weaknesses, and continued pursuing difficult objectives across resets.

The episode combines failure modes that are often tested separately: reward hacking, covert inter-agent communication, side-channel memory, persistence against stop signals, exploitation of shared infrastructure, and adoption of information produced by other agents. The effective harness extended beyond the evaluation code into artifact stores, credentials, caches, network services, and every shared component agents could read or modify.

OpenAI says it paused the relevant autonomous cyber work, tightened controls, and is developing stronger deployment and monitoring standards. The report is a case study, not evidence that every agent will behave this way, but it raises the standard for isolation and incident response around long-running systems.

Builders should map the full communication surface. Package repositories, logs, object stores, temporary files, issue trackers, and caches can become memory or coordination channels. Kill switches must terminate credentials, network paths, workers, and queued tasks—not merely close the visible agent process.

Codex turns the terminal into a task-control system

OpenAI’s Codex 0.149.0 release introduced an interactive codex agents dashboard for finding, starting, opening, renaming, and stopping tasks. A queue command can message existing local or remote sessions, while expanded diagnostics cover endpoint protection, proxies, sandbox behavior, desktop state, and update connectivity.

The stable Codex 0.150.0 release went further. Users can mention another task with @, and agents can read, create, or message tasks from the terminal. Unnamed tasks receive descriptive titles, turning a coding project into a graph of persistent work units rather than one long conversation.

That makes it possible to separate research, implementation, testing, and review while preserving connections among them. It also creates new orchestration questions: who owns a task, whether a delegated message carries authority, how duplicate work is prevented, and which permissions survive resume, fork, or reassignment.

Version 0.150 adds Interrupt hooks that can run commands or MCP handlers when an active top-level turn is stopped. Harness developers can use that boundary to capture state, clean up resources, notify an operator, or preserve evidence. Security and reliability work covers untrusted repository instructions, managed read restrictions, MCP credentials and startup, sandbox setup, shutdown behavior, and multi-agent compaction.

OpenAI’s 0.151 prerelease train adds more explicit Guardian review, MCP provenance, trusted-access metadata, trace propagation, and encrypted sensitive history arguments. Those changes remain directional signals; the production milestone is that task management, messaging, interruption, and recovery are now part of the stable coding-agent interface.

OpenAI makes websites and workflow events native agent inputs

OpenAI’s August 25 ChatGPT and Codex changelog added Site tools, its implementation of WebMCP, to the desktop app’s built-in browser. A website can expose structured tools directly to ChatGPT Work or Codex, allowing an agent to interact through declared capabilities rather than only through pixels, keyboard input, and clicks.

The same update lets scheduled tasks run when supported Gmail, Slack, or GitHub events occur. A task can filter mail by sender or subject, watch selected Slack channels, or respond to pull-request reviews, comments, commits, and merges. The connected application must be authorized, and access depends on plan and workspace settings.

OpenAI also published a practical Runme and WebMCP workflow pattern. Runme notebooks combine goals, plans, commands, results, charts, and decisions in one reviewable artifact. Browser-side tools let Codex read and update the notebook without requiring a separate tool server, while a human approves the plan and consequential choices.

Together, the changes move agent work from manually started conversations toward event-driven loops with browser-native actions and durable runbooks. That raises the same questions faced by webhook systems: which event may start work, how duplicate events are handled, what is idempotent, which actions require confirmation, and how a failed run resumes.

Teams should preserve the triggering event, resolved site tools, app permissions, approval history, and resulting artifact for every run. Event-driven agents need rate limits, deduplication, and replay protection as much as ordinary automation does.

GitHub moves coding agents into the team conversation—and brings policy with them

GitHub’s public-preview integrations let teams start Copilot cloud-agent sessions from Slack and Microsoft Teams. A conversation can become the context for investigating a problem, planning work, writing code, creating an issue, or opening a pull request.

In Slack, Copilot creates a dedicated code channel for the task and displays the repository, branch, pull request, status, and model. The agent works asynchronously in a cloud sandbox while authorized participants continue steering it. In Teams, the surrounding thread provides the context and remains connected to the generated artifacts.

The collaboration model is powerful because implementation starts where decisions are already being made. It is also risky because an informal conversation can contain secrets, stale assumptions, conflicting instructions, or participants who may influence a task without being authorized to approve its result. GitHub warns that thread context is stored in generated artifacts.

GitHub is pairing those shared sessions with stronger administrative policy. Enterprise-managed settings for Copilot in JetBrains can govern plugin marketplaces, MCP servers, OpenTelemetry behavior, and permission modes such as Autopilot or Bypass Approvals.

Organizations should pilot shared sessions in low-risk channels, preserve ordinary pull-request review, and define which conversations may become executable context. Agent collaboration needs both team visibility and centrally enforced boundaries; one without the other is incomplete.

Microsoft combines durable hosted agents with a multi-channel edge

Microsoft’s Agent Framework for .NET 1.19.0 added Azure Blob Storage session persistence, Foundry hosted-agent state, session-persisted routing, experimental interception hooks, and support for resilient, long-running, steerable Foundry Hosted Agents. Its MCP long-running task implementation also moved to the protocol’s July Tasks extension.

Those changes address the runtime underneath an agent: how it survives restarts, persists state, accepts guidance while running, recovers from failure, and aligns long-running work with a protocol task model. Persistence alone is not enough; resumed tools must know which external effects have already happened.

Microsoft then introduced Agent Framework channels for Python. One agent or workflow can connect to OpenAI Responses clients, Telegram, A2A peers, and MCP clients through protocol adapters built on shared hosting state.

The host application still owns routing, authentication, authorization, storage, background processing, and deployment. It also decides how identities from different channels map to an agent session or workflow checkpoint. A shared canonical identity can preserve continuity across interfaces; separate namespaces can keep histories isolated.

The combined architecture is a durable agent core with a programmable protocol edge. Builders should test interrupted runs, duplicate delivery, partial tool completion, session ownership, and cross-channel steering. A user who can continue a conversation through Telegram should not automatically inherit the same authority as an API client or enterprise MCP caller.

Perplexity moves the agent runtime onto user-owned hardware

Perplexity launched Portable Computer, a local-first version of Perplexity Computer built with NVIDIA for DGX Spark systems and, later, PCs with NVIDIA RTX GPUs.

Perplexity says the orchestrator, planner, tool router, scheduler, durable task queue, local search index, models, and sandbox can run on the device. When a task needs capabilities unavailable locally, the system can ask for approval to escalate to cloud models, web research, browser use, or connected applications.

The product challenges the assumption that an agent’s control loop and working memory must live in a provider cloud. Local execution can reduce data movement for sensitive code, documents, and repeated high-volume tasks. It may also give operators more direct control over model choice, files, logs, and persistent work.

Local does not automatically mean private. Cloud escalation, software updates, connectors, browser sessions, downloaded models, and telemetry remain possible data paths. Hardware requirements and local-model limits may also change which tasks can complete without remote assistance.

Teams should treat cloud escalation as a visible policy event. The request should identify what data leaves the machine, which remote service receives it, why local execution was insufficient, and whether remote context returns to durable local history.

Salesforce turns CRM rules into Claude’s action layer

Salesforce and Anthropic announced Claudeforce, beginning with Salesforce in Claude. The plugin includes 37 prebuilt sales skills for meeting preparation, deal-health review, pipeline analysis, and other revenue workflows.

The plugin can reason over live Salesforce context and route governed actions through the CRM. Salesforce says administrators connect the integration centrally so existing authentication, permissions, and business rules remain in force. An open beta is expected after the initial pilot.

The underlying product is AIforce, which Salesforce describes as an enterprise harness exposing data, workflows, business logic, actions, and governance through MCP servers, APIs, and command-line tools. Claude is also available inside Agentforce and across Salesforce and Slack products, making the partnership bidirectional.

The significance is not simply that Claude can read CRM records. Salesforce is positioning deterministic rules and systems of record as the execution layer beneath probabilistic reasoning. The agent proposes or selects work; the business platform enforces permissions, workflow constraints, and durable state.

Enterprise teams should use the same pattern beyond CRM. Expose narrow business capabilities rather than raw database or UI access, enforce policy inside the system that owns the data, and preserve an audit trail connecting the agent’s intent to the accepted transaction.

Google packages vertical agents with policy, evidence, and cost controls

Google Cloud introduced preview editions of Gemini Enterprise for Financial Services and Gemini Enterprise for Legal. Both combine domain skills, secure MCP connectors, managed or partner agents, and a governed control plane rather than presenting a general chat interface as the finished product.

The financial-services edition includes a managed Financial Research agent with more than 50 foundational skills, citations, confidence information, methodologies, and data snapshots for audit. It connects to licensed and internal data under existing entitlements and can participate in larger workflows through A2A APIs.

The legal edition packages contract review, redlining, playbook creation, legal research, regulatory monitoring, data-subject requests, and redaction. Its connectors cover document management, e-discovery, primary law, research, contract, and productivity systems while preserving matter-level access expectations.

The supporting control plane is becoming operationally concrete. Google promoted semantic-governance policy lifecycle commands to general availability, added monitoring for data-agent performance, latency, adoption, and cost, and introduced MCP and connector-level tracing. It also announced billing and cost controls for agent workloads, including pooled quotas, project spending caps, anomaly detection, overage controls, and runtime estimates.

The combined package shows what regulated buyers increasingly demand: domain procedures, entitlement-aware data, citations, audit evidence, policy automation, connector observability, and a predictable cost envelope. Builders competing in vertical markets should expect all seven to become baseline procurement questions.

Temporal makes the agent loop a durable workflow

The open-source Temporal Agent Harness applies Temporal workflows to multi-agent execution. The project emphasizes replayable traces, typed agent operations, subagent composition, observability for cost and tool use, and human approvals that can pause and resume without losing workflow state.

Its design treats a tool approval, retry, timeout, or long wait as part of a durable execution history rather than a special case held in one process’s memory. A Code Mode tool lets a model write Python against approved host tools, while the surrounding workflow controls which capabilities are exposed and when approval is required.

The repository is a community project and should be evaluated as an implementation reference, not assumed to be production-ready for every environment. Its architectural point is stronger than any individual feature: long-running agent work maps naturally onto workflow systems that already understand retries, timers, signals, checkpoints, and replay.

Teams building tasks that span hours or human review should compare workflow-native orchestration with an ad hoc queue and transcript store. Durable execution adds machinery, but it also makes the agent’s lifecycle explicit enough to test, inspect, and recover.

Opinion: What This Means for Harness Engineering

Opinion: The agent control plane now stretches across browsers, task graphs, collaboration channels, local machines, cloud workers, and enterprise systems of record. A harness should distinguish every execution identity: the user’s browser, an isolated agent browser, a local model, a cloud escalation, a chat participant, an MCP client, and a delegated subtask. Collapsing them into one “agent user” creates invisible privilege transfer.

OpenAI’s incident report shows that undeclared infrastructure can become part of the loop. Agents may use package stores, logs, caches, files, or issue systems as memory and communication channels when those surfaces help complete the objective. Security reviews must inventory all shared writable services, and emergency shutdown must revoke credentials and network reach as well as stop processes.

Durable tasks need transaction semantics. Codex task messaging, Microsoft hosted-agent recovery, OpenAI event triggers, and Temporal replay all depend on knowing what has already happened. Persist idempotency keys, tool results, approvals, policy versions, and external side effects—not merely the conversation. A resumed agent should never have to guess whether it already sent the email, merged the change, or updated the customer record.

Governance and observability should meet at the action boundary. Anthropic’s compliance transcripts, GitHub’s managed policy, Google’s semantic controls and connector metrics, and Salesforce’s business rules all point toward one requirement: every consequential action should carry the effective identity, permission, policy, tool origin, cost, approval state, and result.

Finally, evaluate the harness around the model. Test prompt-injection pages, poisoned team threads, duplicate events, stale credentials, covert shared storage, interrupted local-to-cloud escalation, failed connectors, and human approvals that arrive hours later. Model quality matters, but operational correctness determines whether the system can be trusted with real work.

Closing Note

The agent race has moved beyond the prompt box. The new battleground is the machinery that lets agents browse, delegate, remember, recover, and act without losing identity, policy, evidence, or cost control. Capability opens the door; the harness determines what happens after the agent walks through it.

References

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

Python Owns the Loop: Loop Engineering and the Harness Around It

Rick Hightower — Mon, 31 Aug 2026 02:44:18 GMT

Python Owns the Loop

Loop Engineering and the Harness Around It: How to build a production loop in Python: the model drafts and grades, software holds every transition

A vague ticket in. A ready ticket out. The model drafts and grades. Python holds the loop.

Have you ever watched an agent declare a ticket done because it said so? Picture an enhancer on GitHub issues. Perhaps the body is still a wish. The acceptance criteria are still a shrug. The model writes “this looks complete” in the same voice it used to write the ticket, and the loop treats that sentence as a gate.

Then imagine the other failure, which is worse in a quieter way. The agent that will not stop. It edits the same ticket forever. Add a field, lose a field, declare progress, miss the same three headings it missed last turn. Nothing outside the model is allowed to say “enough.” The whole design is one italicized fake line: while not done: ask_the_model(). “Done” lives in the reply.

It is maddening the way a smart intern grades their own homework. A demo of that loop looks fluent. Three weeks later the same agent is either rubber-stamping unfinished work or burning money on a stall nobody outside the model can see.

A clever prompt can survive one ticket. At ten tickets a day it drifts. At a hundred, nobody remembers what good looked like. The bottleneck is not the model. It is you, reading every diff and deciding whether the confident paragraph in front of you is evidence.

In this article: You will learn what loop engineering and harness engineering actually are, then how one Python program implements them. The names come from The Loop Is the Product. The code comes from reliable-agentic-lab, solutions/sol1_enhancer_agent_sdk/. This is the first of four jobs that share a role graph: enhance, implement, research, and fix. Vague GitHub issue in. Rubric-ready ticket out. We will walk one poll step by step, then read the listings that make those steps true, then run it: one poll, an exact LGTM. A Substack version of the same argument exists as an essay. New technical words are defined once in the body and collected in the glossary at the end. To diver deeper try this interactive demo at Loop Engineering: Python owns the loop.

Subscribe now

A high-powered model without a governor is a turbine spinning toward its own destruction. The inner cycle most coding agents already do is: think, call a tool, look at what came back, think again. Papers call that ReAct. Left alone, it keeps spinning, because continuation is what the model is for. The product you ship is the governor around that cycle. Software measures the spin, limits who may write, and forces a stop the model cannot talk its way past.

Python is that governor here. The model drafts a better ticket and reports which fields have content. It does not write the file. It does not decide the loop is finished.

A production loop is not “call the model until it says done.” If the same model, in the same thread, in its own words, is allowed to stop the run, you do not have a loop. You have a turbine with the governor removed.

This is not a prompt cookbook. You will not get a bag of system prompts to paste. It is not a workshop recap. Every listing comes from a Python program you can clone and run.

Loop engineering and the harness around it

The Loop Is the Product opens with four lines. Scope the action. Verify independently. Persist state. Bound the exits. We will put a name on each line, then show the Python.

Loop engineering is designing the cycle the agent runs, and who is allowed to stop it. The unit of work is one controlled pass, not a single model reply. A production loop is that cycle running when a human is not watching every turn. The model proposes the next action. Software owns five steps. Call each step a node. The five are trigger, action, verify, memory, and exit.

Trigger: Something outside the model starts a turn: an issue, a poll, a webhook. The model does not get to feel like going again.
Action: One scoped job, done by a worker.
Verify: A checker that is not the worker, plus a deterministic fact. Asking the author whether the homework is done is not verify. The checker in this program is the judge subagent, plus check_fields.
Memory: Files on disk a later process can read: a ticket, a JSON state file, a GitHub issue. Chat history is not a restore point.
Exit: The run has to end. The ticket ships (pass), you try again because the gap can still change (retry), or a person has to look (escalate). The model can type DONE. Python still picks which of those three actually happens.

The outer flow can materially change what the same model produces. On CodeContests validation, the AlphaCodium flow took GPT-4 from about 19% pass@5 with a direct prompt to about 44% by planning, generating tests, generating code, and looping on failures. That is evidence that flow design matters, not a promise that this enhancer will score 44%. The important transfer is simpler: tests own done. A generation does not approve itself.

An agent is a model given a role: a prompt, a declared tool list, a declared path check. A worker is the agent assigned the action. Not every tool the platform owns. In this program the worker is the doer.

A subagent is an agent the parent session spawns. The subagent gets its own context, separate from the parent and from every other subagent. The child’s reads and greps stay in the child’s window. The parent sees a result, not that inner chat. The judge is a subagent too. It does not share the doer’s context.

Isolated context does two jobs. It keeps the child’s window small so the agent does not get overwhelmed. It also splits the problem into a smaller unit of work, with only the context and instructions that job needs. The doer drafts. The judge scores. Neither inherits the other’s dump. Isolated context is harness engineering. A smaller, job-shaped window is how you dodge context-attention failures (the fact is in the window, the model cannot find it), context panic (the window fills and the model flails), and context rot (tool dumps bury the ticket). Two prompts in one thread are still one intern.

The Lost in the Middle result gives this a practical reason. Long-context models use information more reliably near the beginning or the end of a window than in the middle. Put a patch, research dump, or plan on disk. Give the parent a short summary and a score. The context window is a scratch pad, not the store.

Keep three kinds of state separate: on disk holds the ticket, trace, and plan; in process holds the current budget, signature, and round; not in the chat transcript holds writes, stop, and LGTM. Context windows reset. Ticket files do not.

Those five nodes are the cycle. They are not yet safe to run while you sleep.

Harness engineering is the software around that cycle: who may write, what counts as done, when to stop, and which context a role may see. In this program it is the role table, the tool lists, the hooks, isolated subagent context, the rubric, the budgets, and the tests.

Mechanical sympathy is adapting how the machine, the LLM, actually behaves, then feeding it only what it can take. The harness does that for an agent: it adapts the outside world into the model’s context so the agent does not get overwhelmed. It is the governor on that spinning turbine, so the loop does not run away. Loop engineering designs the cycle. Harness engineering is what makes the cycle safe unattended. The tools are the access to the outside world, and the barrier.

Banking learned the split a century ago. The person who writes the check is not the person who approves it. Maker and checker must be different things. Not different prompts in the same thread. Different roles, different tools, different context windows, and a fact computed outside either mouth. Two prompts that describe different roles can still call the same write tool, and they can still read each other’s chain of thought if they share a chat. A role name in a prompt is not a fence. The fence is the tool list, the path check, and a child session the parent does not share. The runtime checks the path before the write. The runtime also starts the judge in a window that never saw the doer’s ReAct.

The four opening lines map onto this program as follows:

Four properties of one iteration

Every turn through that cycle has to satisfy four properties. Miss one property and the next turn cannot be trusted. The figure names them in the order a poll actually runs.

Read the figure left to right. The row is one pass through the loop. Trigger is not drawn here. It is the event that starts the pass: an issue, a poll, a webhook. External transition is the point: software decides pass, retry, or escalate. Explicit state, bounded authority, and observable evidence exist so that external transition has something true to decide on. Dashed orange on observable evidence means “a file or a score you can point at later,” not the write-tool fence we will see in a later figure.

Map the four properties onto the five steps. Explicit state is the ticket file and the JSON under .harness. A later poll reads those files. It does not replay the chat. Memory is those files. Unbounded authority means the worker can call any tool the platform owns, with no declared allow list and no path check. Bounded authority is the opposite: the worker may use only the tools and paths its row in the role table allows. Action is the scoped job. Observable evidence is a candidate file, a judge report, a test result. Verify reads that evidence. External transition is exit. The model may type DONE. The model does not get the vote.

Inner ReAct, outer control

ReAct is the inner cycle of reason, act, and observe. Every coding agent you have used is doing some version of it. The product you ship is not that inner cycle. The product is the outer control system around it.

The figure is ReAct wrapped by loop engineering. Inside, the model is already doing reason, act, observe. Act is how the worker reaches the repo, through tools. Observe is evidence you can point at later: a file or a score, not a confident paragraph. Callback to observable evidence from the four properties.

Loop engineering wraps that inner cycle. Perceive / Trigger is how a turn starts: an issue, a poll, a webhook. Decide is how a turn ends: pass, retry, or escalate. Retry is the same run, next attempt. It is not a new GitHub issue. The product you ship is that outer control, the governor on that spinning turbine, not the inner ReAct.

What breaks when the governor is off

Context rot. The chat is the only memory. Intermediate tool dumps bury the original ticket. The fact is still in the window. The model can no longer find it.

Runaway iteration (tool storms). Unbounded authority means the worker can call any tool the platform owns, with no declared allow list and no path check. The worker fires redundant or invented calls, burns tokens, and reports motion. A timer that fires when no state has changed is not a trigger. It is a token burn with no new evidence.

False completeness. The worker says done. “DONE” is in the reply. Nobody ran verify. Nobody wrote an evidence file.

Stagnation (stable failure). The same gap twice, dressed as progress. Missing acceptance criteria round one. Missing them round two, with a more confident paragraph around the hole. Nobody stops the loop, because the model is still “working.”

A failure hash (also called a failure signature) is a fingerprint of what still failed. After the judge, missing fields become a list, for example ["criteria", "value"]. _improve returns that list. _one stores it as previous_signature. The next round builds a new list. check_stop does signature == previous_signature. Equal lists mean the ticket did not move. The loop stops with same signature two rounds running. The hash is a fingerprint of the stall, not SHA-256. Listing 6 note ① is that comparison. Listing 7b is where Python saves the previous list.

A production loop assumes the model will eventually drift. External rules are how you force a return to the objective, or a stop.

The loop almost every team ships first looks like this. It looks responsible. It is an open throttle. Listing 1 is a negative example. Do not ship it. Every marked line is a failure. You will see a while True with no budget, a stop that lives in the letters D-O-N-E, the writer overwriting the ticket, and the next turn’s memory as a string in RAM. None of the four properties survive this listing.

Code Listing 1 (negative example): the intern grades the homework. Do not ship this.

ticket = open(“ticket.md”).read()

while True:                                 # ① WRONG! no round count, no dollar cap
    reply = model(
        “Improve this ticket until it is DONE.\n” + ticket
    )
    if “DONE” in reply:                     # ② WRONG! stop is a token the model may emit
        open(“ticket.md”, “w”).write(reply) # ③ WRONG! writer overwrites the ticket it graded
        break
    ticket = reply                          # ④ WRONG! next turn’s memory is a string in RAM

① The loop never counts rounds or dollars. There is no exit except vocabulary.

② Stop lives in the model. "DONE" is a token the author is allowed to emit.

③ The writer is the grader. The same string that claims completeness overwrites the ticket.

④ There is no evidence file. The next turn’s memory is a string in RAM.

What listing 1 showed: no explicit state on disk, no bounded authority, no observable evidence file, no external transition. State is a Python variable. Authority is whatever model() can return. Evidence is the conversation. Stop is the letters D-O-N-E. The rest of this article puts software around that while. The object we will put it on is a ticket enhancer.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Share Hightower's AI Harness Engineering

The Second Brain Graph Is the Spec and much more

Rick Hightower — Thu, 27 Aug 2026 02:06:10 GMT

Why does this exist?

Project Knowledge Capture turns your tickets, your meetings, the decisions you make in those meetings, architectural decision records, assumptions, and experiments into knowledge and stores them so they are retrievable. It enables efficient, token-efficient knowledge retrieval. You are not just dumping markdown documents. It lets you retrieve that information, much like you would with an agentic skill, using the same partial-disclosure architecture technique. Instead of the agent skill, it is the information about your project.

SAC, Systems Architecture Capture, looks at the codebase. The coding agent does. It uses SAC to store that information in the same second brain.

You do the same thing with Data Engineering Capture. You store information about your data engineering jobs and your schemas.

There is one to store information about code and its configurations.

As you develop your software, you have these second brains accessible from your coding agents. When they implement features, they can quickly look up and reference information about the current feature they are working on, or related information: decisions that were made, standards, what you have. This can span across projects. Engineering rules, architecture decision records, decisions in meetings, regulatory requirements, and research.

It gives your coding agents knowledge they can use and efficiently query. Much like agentic skills. But it is knowledge about your project. About your architecture. About your infrastructure. About decisions that were made in the meetings you went to.

It makes this knowledge visible to other agents that are working. Other agents can be updating the same second brain. There is a protocol for that. The knowledge can be shared between projects. It is a way to share knowledge amongst many different agents.

We have taken this and ported it to Grok Bot so we can store it with agents that have nothing to do with coding, too, but that is for another article. Then they store their own ontologies, what we often refer to as nouns.

What the stack claims

Two problems sit on one substrate. Software projects generate reasoning that dies in Slack, standups, meetings, Zoom calls, Teams, and people’s heads. Tickets record what and when. Not to mention spikes, prototypes, legacy systems, and tons of important but ancillary data. Almost nothing records why, what was rejected, or which experiment killed which assumption. Separately, multi-agent systems live as prompt strings and framework glue. Nobody can review that, diff it, impact-analyze it, or port it. A loop that cannot stop is not a design.

The substrate is OKF: Git-native Markdown plus YAML. OKF allows us to build the LLM Wiki. We have one concept per file. Typed edges. No server, no database, no account. Git is the storage. Merge conflicts are treated as a feature. Everything else is a specialized noun owner on that substrate: an engine that walks it, capture packs that write it, a spec for data engineering, a spec for project knowledge, a spec for systems architecture, a spec for agent loops, and one for project knowledge.

Subscribe now

Thesis
The graph is the specification. Framework code is an output. Institutional memory is a graph, not a wiki dump. Indexes are caches. Markdown in Git is truth.

These are all collections of skills and subagents packaged as agent plugins that you can use on your project. They all support Claude Code, Grok Build, and Codex agent plugin formats, as well as the official standard for agentic plugins, making them available for scores of agent harnesses.

PKC (Project Knowledge Capture) captures meetings, spikes, decisions, features, stories, tickets, tasks, subtasks, releases, and PRs, then materializes WikiTicket work into TicketLink nodes without ever owning status.
SAC (System Architecture Capture) captures cloud deployment, load balancers, packages, dependencies, modules, containers, etc.
DEKC (Data Engineering Knowledge Capture) captures lakes, tables, catalogs, semantic layers, schemas, views, ETL, ELT jobs, data validation rules, etc.
AGER is a framework-neutral document model for multi-agent runtime config, not a runtime. LoopControls are the load-bearing idea.
RKC is a framework to capture research.
okf-graph-eng 0.8.1 owns only Catalog and ContextPack. It is a read-path optimizer: 2 hops, 20 nodes, fail-closed token budget.
Translators are 0.1.1. The idea is load-bearing. The emit quality is not yet a production claim. The AGER design doc and the translator repos do not fully agree on how much compile is in scope.
A noun-ownership cut landed on 24 August 2026. Mixed bundles from before that date will misvalidate until the owning plugin is installed.

Figure 1. Component view of the stack. Capture packs write left. The gold column is the shared substrate. Translators compile out to the right.

Read the drawing as a C4-style component diagram, not a marketing stack. The left column owns domain nouns: PKC for why, AGER for the loop spec, SAC for running topology, DEKC and RKC for data and claims, plus the business ContentPacks that point the same machine at a firm. They write Markdown into Git. PKC and the business packs stop there and ask the engine to pack. SAC and DEKC also reverse-engineer: they walk a running estate or a data platform using host-agent skills, then write up what they find. None of them execute an AGER graph.

The gold column is the only shared runtime-adjacent surface. WikiTicket owns work status. second-brain-core owns identity, WriteEvent, and worktree isolation. okf-graph-eng owns Catalog and ContextPack. OKF itself is files. The right column is compile-only. Translators at 0.1.1 emit into hosts. They are not a second source of truth.

OKF
OKF is Open Knowledge Format and is the standard for LLM Wikis. It provides a partial disclosure architecture (PDA) similar to that of agent skills. It is token efficient.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems: Workshop

Engineering Reliable Agentic AI Systems
Saturday, 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

Why the graph, not the prompt

By mid-2026, the industry had a name for a pattern that several spec-driven tools were already practicing: graph engineering. Explicit nodes, typed edges, persistent state, and an index, in place of one long agent loop. WikiTicket SDD’s own README claims it did not adopt that framing so much as already was that framing. PKC, SAC, DEKC, and AGER are the same claim pointed at two different planes of work. In order to have self-improving loops, you need a way to store the results of experiments.

Prompt stuffing fails predictably. A long-running agent cannot read a whole second brain. A dump of Markdown into a context window is not retrieval; it is a bet that the important node lies somewhere in the first 20,000 tokens. ContextPack exists because that bet is usually wrong. It is a deliberately lossy subgraph: outbound-only walk, hop-capped, node-capped, ranked by verification and declared impact, fail-closed against a token budget. Overflow is listed so the model can request a named follow-up rather than guessing.

The write side is equally strict. The model proposes. Schema-enforced scripts commit. An agent never invents a rel and never writes a type it does not own. Concurrent writers isolate via worktree and PR. Nobody shares a mutable main tip. Required author identity landed in August 2026. That is not ceremony. It is how a Git-native knowledge graph remains an audit trail rather than a shared notebook that everyone overwrites.

Two amnesias

Call the first one project amnesia. Design discussions produce decisions that never become ADRs. Spikes produce conclusions that die in a Slack thread. A later agent, or a later human, re-litigates the same question because the rejected alternative was never a node. PKC’s sample graph is an authentication chain: Discovery to Experiment to Meeting to Decision to Feature to Design to CodeChange to Release, plus a Risk the decision mitigates and an Acceptance criterion the PR verifies. This is the worked example of reasoning that would otherwise have disappeared.

Call the second one agent amnesia. An inherited LangGraph or CrewAI codebase is rarely a spec. It is a collection of prompts, tool wrappers, and loop limits that may or may not agree with each other. Reverse-engineering that into AGER (the AGKC path) is the same operation that SAC performs on a monorepo and that DEKC performs on a data platform: turning running folklore into typed, validatable concepts.

Figure 2. Two planes, one substrate. Left: institutional memory of reasoning, meetings, seals, notes. Right: a loop that can halt, a governor and a stop lever. PKC owns the first plane. AGER owns the second. Both write OKF.

The photograph is the metaphor. The architecture is stricter. These are not two products that happen to share a brand. They share a file format, a validation engine, a packing primitive, and a write protocol. An AgentGraph can bind into a PKC tree through KnowledgeBind and RetrievalBinding. A Feature pack can include the Decision that authorized it and the Experiment that almost killed it. The stitch is typed edges, not a search index.

The layer cake

Read the stack from the bottom. OKF is the format. okf-graph-eng is the engine. second-brain-core is the shared write substrate. WikiTicket is the work plane: append-only ULID JSONL, union merge, event fold, generated roadmap. Noun-owning plugins sit above that: PKC, AGER, SAC, DEKC, RKC, and eight business ContentPacks. Translators sit above AGER. Hosts load plugins.

Figure 3. Deployment layers, substrate to host. Each band has a job. The engine does not own domain nouns. Domain packs do not reimplement graph algorithms. Translators do not author graphs

The drawing is a deployment diagram with a single rule: arrows go only downward. Hosts load translators. Translators read AGER. Noun owners write OKF. The work plane records status that PKC is forbidden to mutate. The write substrate is how concurrent agents avoid trampling main. The engine packs. Git stores.

The noun owners are various agent plugins supporting most coding harnesses, including Grok Build, Claude Code, OpenCode, Codex, Cursor, and dozens more. They own the nouns, or the ontology, for a certain aspect of knowledge capture.

The 24 August 2026 noun-ownership cut is the most recent structural event in the public repos. Before it, domain types were parked in the engine. After it, each pack owns its nouns; validate --strict fails closed on unknown types. Impact ranking now comes from x-impact on the owning plugin’s schema, not from a hardcoded list in the engine. That is two days old relative to this reading. Mixed bundles from before the cut will look broken until the owning plugin is on the schema path.

A read-path optimizer, not a product

okf-graph-eng 0.8.1 owns two nouns: Catalog and ContextPack. It is not a capture plugin, a runtime, or a data catalog. Unknown types fall back to BaseConcept for read-only envelope parsing. That fallback does not authorize a write. Strict validation rejects them.

Catalogs are hubs. An undirected walk through a catalog dumps the bundle. Default packing is therefore outbound-only BFS. Two hops is usually agent to collaborator to evidence. Hop 3 is a debug knob. Unlimited closure is impact, not pack.

Figure 4. ContextPack as a guarded walk. Inclusion ranking is not read order. The model sees the question it asked first, then the dangerous neighbors, then the supporting evidence. Unverified high-impact nodes are flagged rather than treated as truth.

The flowchart is the packer’s state machine. Every diamond is a fail-closed guard, not a preference. Outbound-only keeps catalogs from exploding the walk. Hop 2 is the default neighborhood: the Feature, the Decision that authorized it, the Experiment that almost killed it. Twenty nodes is a ceiling, not a target. Overflow is listed by name so the model can request a follow-up pack instead of guessing from a truncated dump.

Sibling packs wrap the engine’s packer with a token budget, default one quarter of SECOND_BRAIN_WINDOW_TOKENS (128k becomes 32k). Over-budget packs are not written. Neighbor bodies are omitted unless the node is the pack root. PKC auto-inject can drop a tiny Feature pack into a prompt when a Feature is named. That is the intended way a long-running agent reads the tree. Dumping knowledge/ is the anti-pattern the whole system is built to prevent.

Figure 5. How a write reaches the shared tip. The model proposes. Schema-enforced scripts commit. Isolation is a worktree and a pull request, not a shared mutable tip.

The sequence is operationally heavy on purpose. An agent does not push to main. It proposes a concept. Core checks that the type is owned and the rel already exists in the schema. The commit lands on brain/actor/session-id. A WriteEvent is appended so the journal can reconstruct who wrote what. The shared tip moves only by PR. Concurrent writers never share a working tree.

Project Knowledge Capture

PKC is the capture and materialization layer. It owns twenty-six nouns in five groups: capture (Meeting, Experiment, Discovery, Assumption, Question), product, delivery, decisions and ops, and WikiTicket work types (TicketLink, Epic, Story, Task, Subtask, Bug, Branch, Project). Catalog and ContextPack stay in the engine.

Capture commands turn messy input into concepts: meeting notes, Fireflies/Otter/Granola transcripts, Slack threads, GitHub PRs, existing MADR directories, spikes, assumptions, open questions. Secrets and PII are scrubbed before writing. Promotion turns informal notes into Feature, Requirement, or DecisionRecord (the command still calls this a lightweight ADR). Linking records only the relationships the source material states. Doctor reports bundle health: conflicts, thin features, and stale nodes.

Figure 6. From notes to a Feature neighborhood. Materialize is the stitch with WikiTicket. Epics and stories become Features plus TicketLink nodes. Fingerprints skip unchanged items. PKC mirrors status; it does not write .work JSONL.

The pipeline is a flowchart, not a suggestion. Capture is messy on purpose: meetings, spikes, transcripts, PRs. Promote is where informal notes become typed product nouns. Link is the protocol boundary. Inventing a relationship type is a violation, not a convenience. Materialize is optional and one-way. If there is open WikiTicket work, PKC folds it into TicketLink nodes and Features. Status stays in the worklog. The last box is the only thing an agent should see: a packed neighborhood, not the archive.

Figure 7. PKC class model. Five noun groups collapse to three packages on the page: capture, product, and a work mirror. WikiTicket remains the source of status. TicketLink is a fingerprint, not a second backlog.

The class diagram is the noun model the pipeline writes. Feature is the hub. A DecisionRecord originates_from a Meeting or Experiment and decides a Feature, Design, or Requirement. Experiment and Discovery inform Feature. Assumption assumes a Feature. A Question blocks a Feature. The work-mirror package is the one-way stitch: TicketLink and the Epic/Story/Task cluster. The dashed edge to WikiTicket is the whole non-goal: PKC does not become the backlog.

The typed edges that make a Feature pack useful are specific: originates_from, decides, informs, assumes / validates / invalidates, blocks / answers, exposes / mitigates, verified_by, lands_in, tracks. Direction is not symmetric.

PKC’s vision document is unusually useful because of its non-goals. It refuses to become a general notes app, the source of truth for work status, a reimplementation of OKF graph algorithms, or a real-time collaboration server. WikiTicket remains source of truth for what we are doing now. PKC is the source of truth for why we decided that.

Docs drift
The public wiki User Guide still documents PKC v0.4.1. The plugin is 0.8.1. The wiki loaded during this fact-check and has not been updated. Trust the README, docs/, and CHANGELOG over the wiki.

AGER is not a runtime

The current design doc still describes plugin 0.4.0, but the claim has not moved: the repository does not execute graphs. Generating production framework adapters is out of scope for the plugin. okf-agent-graph 0.7.1 authors multi-agent loop graphs as OKF. Spec 0.3.0 defines fifty-one nouns. The plugin README surfaces them as nine planes (agents, graph, tools, control, memory, contracts, ops, eval, quotas). The spec groups the same nouns into four (core graph, control, memory, ops). The plugin validates them. It does not execute them. It is not a credential store. Literal secrets are rejected; SecretRef and ${ENV} only. It is not an application framework. There is no bus, no database, no deploy.

Every AgentNode and Tool must have JSON Schema I/O contracts. ToolRule expressions can block, allow, require a human, or rewrite arguments. Memory is explicit: ScratchPad with lineage, EpisodeStore, KnowledgeBind into an OKF tree, RetrievalBinding, context isolation for subagents. Ops cover Run, Trigger (including ticket_event and okf_change), failure, retry, compensation, circuit breaker, dead letter, checkpoint, trace.

Figure 8. A loop that cannot stop is not a design. LoopPolicy binds goal, deadline, price, max turns, and no-progress. The figure shows the first three so the halt path stays readable. on_goal and on_exhaust are first-class. Translators must preserve this, or compile is just a rename.

The state machine is the product. A running turn hits five guards in a fixed order. The first yes wins. The figure keeps Goal, Deadline, and Price on the page; max_turns and no_progress also exhaust, same as a budget stop. If every guard is no, the loop takes another turn. A translator that drops this machine and substitutes a framework max_iter has not compiled the spec. It has renamed a while loop.

Figure 9. Nine AGER planes. The plugin README lists nine planes; the spec groups the same nouns into four. LoopPolicy is the gold row. KnowledgeBind is the stitch into the project graph. Unverified high-impact nodes fail validate --strict.

The component diagram is the rest of the spec sitting under that halt machine. Fifty-one nouns, nine plugin planes, drawn here as four spec groups. The gold row is LoopPolicy’s check order, first because it is the only thing that makes the other planes into a design rather than a prompt collection. KnowledgeBind is the typed stitch into PKC, SAC, or DEKC. Without it, an AgentGraph is a loop with no memory of the project it claims to serve.

Two directions of use. Forward: init, author, validate, pack, compile. Backward: AGKC, Agent Graph Knowledge Capture: scan a live agent repo, reverse-engineer draft AGER knowledge with provenance, promote, validate. The plugin README names Claude and ChatGPT APIs, LangChain, LangGraph, CrewAI, LlamaIndex, Claude Agent SDK, Deep Agents, MCP, Bedrock AgentCore, and hardened microVMs or containers. The 0.5.0 changelog also lists AutoGen, Semantic Kernel, Azure AI Agents, Vertex Agent Engine, and Firecracker, gVisor, Kata, and E2B sandboxes. The AGKC design note is more generic: hyperscaler agent cores and hardened microVM or container sandboxes. Scanners are read-only, standard-library-only, and do not execute untrusted project code. Secrets are redacted.

Figure 10. Reverse-engineering an inherited agent. AGKC is the same capture idea as PKC, pointed at running folklore. Drafts land unverified. Promotion is a human step. The depth of extraction per framework is not evidenced beyond draft knowledge, with provenance.

Read the sequence left to right. An operator points a scanner at an inherited repo. The scanner parses prompts, tools, and loop limits without executing project code. Draft AGER nodes land in the OKF tree with provenance and stay unverified on purpose, because high-impact nouns fail validate --strict until a human promotes them. The last round trip is the review artifact: missing I/O contracts, missing LoopPolicy, literal secrets that must become SecretRef.

The scanner list is wide. Treat AGKC as a real path that landed in 0.5.0 on 9 August, not as a completed inventory of every target.

Compile as a product

There is a split you should not paper over. Inside okf-agent-graph, /ager-compile emits adapter notes and stubs. The design document says compilation is out of scope for the plugin. It is mapping guidance, not production codegen. Sitting next to that are nine translator plugins at 0.1.1, including a Google ADK emitter, an ager-translators monorepo that loads OKF into dataclasses, and an early Graph Designer UI that authors OKF and Mermaid and explicitly does not compile.

Figure 11. One manuscript, many bindings. Framework choice becomes a compile target instead of a rewrite. That is the claim. Fidelity of LoopControls and ToolRules across those bindings is the whole game, and it is the least documented part of the public surface.

The compile map is a component diagram with a warning stripe. Gold is the manuscript: a validated AgentGraph whose LoopControls, ToolRules, and KnowledgeBinds travel with the file. The arrows are compile-only. The teal boxes are 0.1.1 emitters. Hosts receive a manifest, not a fork of the graph. Google ADK is called out because it is the odd target: five languages, agent mode and graph mode, ADK 1 and 2.

Treat the plugin’s compile command as the contract and the 0.1.1 emitters as an early layer sitting on top of it. Stars are essentially zero. Commits are few. The idea is to write the loop once and ship it to every runtime. That is the most important product claim in the stack after ContextPack. The emit quality is not something the public repos currently prove.

SAC, DEKC, and the rest

WikiTicket SDD (README v0.24.7; marketplace worklog pin 0.24.4) is the most mature piece. Append-only event log, four-axis taxonomy, generated roadmap, plan capture, merge-rescue around nightly compaction. PKC folds it. PKC does not replace it. Publishing to wiki remains WikiTicket’s job.

SAC is System Architecture Capture. It is “what is running,” not what we decided. One hundred thirty-nine architecture and runtime nouns. Reverse-engineers monorepos and multi-repo estates into C4-shaped OKF: Person maps to Person or Actor, System stays System, Container becomes SoftwareContainer, Component stays Component, Code becomes Module or Class. Blast radius here is topology. The engine’s impact walk is generic. The noun surface is a catalog, not yet obviously a tightly validated ontology.

The capture surface is the running estate. Cloud accounts, regions, environments, and deployments. Clusters, namespaces, and node pools. Load balancers, API gateways, CDNs, DNS zones, VPCs, and subnets. IAM roles, policies, service accounts, and identity providers. IaC as InfrastructureStack, HelmChart, and TerraformModule. Modules and code namespaces. Packages, SDKs, and CLIs. Container images and serverless functions. CI/CD as Pipeline, Job, and CronSchedule. APIs as ApiContract, Endpoint, Webhook, and GraphQL schema. Ownership and BlastRadius as nodes, not comments in a wiki.

C4 topology. System, Service, Component, SoftwareContainer, Module, Class, Actor.
Cloud and deploy. CloudAccount, Region, Environment, Deployment, Cluster, Namespace.
Network edge. Vpc, Subnet, LoadBalancer, ApiGateway, Cdn, DnsZone, PrivateLink.
Identity. IamRole, IamPolicy, ServiceAccount, Permission, IdentityProvider, AuthConfig.
Build artifacts. Package, Sdk, Cli, ContainerImage, ServerlessFunction, HelmChart, TerraformModule.
Delivery. Pipeline, Job, CronSchedule, plus BlastRadius as a first-class neighbor.

DEKC is Data Engineering Knowledge Capture. It is the data plane: schemas, tables, views, columns, queries, lakes, marts, catalogs, domains, products, streams, medallion layers, lineage, contracts, quality rules, semantic models, metrics, reports, dashboards, business objects, and glossary terms. Technical assets are meant to be promoted to business objects with definitions, not sit forever as raw tables.

Assets. Dataset, Table, View, Column, Schema, Query, SqlArtifact, DaxArtifact.
Platform. SourceSystem, DataLake, DataMart, DataCatalog, DataDomain, DataProduct, StorageLocation, Stream, Layer (bronze through gold).
Movement. IngestionJob, Transformation, LineagePath, DataContract, DQRule. Old Workflow jobs are retyped to IngestionJob.
Semantic. SemanticModel, Metric, Report, Dashboard, BusinessObject, GlossaryTerm.
Other. DesignPattern.

DEKC walks platforms with AGER-shaped loops, and it does that with subagents, not with /ager-compile. An orchestrator (default data-lake-walker, or a reverse-engineering orchestrator aimed at Fabric, AWS, or GCP) fans out scouts: schema, lineage, stream and job, report catalog, semantic mapper. Adversarial skeptics grade the reverse engineering against rubrics (lineage, business, stream and job, coverage, layer). A lead judge's thresholds are roughly 0.75, and failures work back to revise or retract. A synthesizer indexes, packs, and writes a judgment receipt. Those loops are host-agent skill choreography. The subagents explore the data. They are not an AGER-executed runtime. Easy to misread if you expect compile to work the way DEKC runs.

RKC research-graph is the cleanest epistemology in the stack. RKC is Layer 0: ResearchArea, Subject, ResearchTask, SourceDocument, ResearchQuestion, Claim, Evidence, Finding. research-graph is a disposable Layer 1 projector onto Chroma, BM25, and Kuzu. It owns no nouns, so destroying the index is always safe. RKC is not installed from the main marketplace, an explicit exception that is easy to miss.

Business ContentPacks (~0.3.x) apply the same write protocol to an operating company: executive coordination, sales pipeline, content media, GTM, job search, consulting leads, news digest, account management. Representative content-media owns twenty nouns, including Article, Draft, Series, Subscriber, Segment, PerformanceMetric, StyleGuide, and ContentExperiment. Same init / capture / pack / session / doctor shape. These are not engineering captures. They are the same machine pointed at a firm.

Figure 12. Dual-owned names. After the ownership cut, these strings mean different things. Do not mix them into a single bundle without routing. Package, RateLimit, and Workflow are the ones that will bite first.

The class pairs are a hazard map, not a taxonomy. The same string is a PKC shippable, a SAC build artifact, an AGER quota, a SAC gateway limit, an AGER multi-agent graph, and a retired DEKC job type. WriteEvent is a second-brain-core journal entry and is not an AGER noun at all. After 24 August 2026, the rule is one tree, one meaning. Mixed bundles from before the cut won't validate until the owning plugin is on the path.

How the pieces actually compose

A working engineering second brain, installed from second-brain-marketplace, includes the engine, PKC, SAC, DEKC, AGER, worklog, and optionally second-brain-core. Then one translator per target runtime. RKC is added by hand.

Figure 13. Four paths, one Git tree. Read, write, work, and agent-system are different walks over the same files. Mixing them in one session is how folklore returns.

Four swimlanes, four reasons to touch the tree. The read path never writes: name a concept, emit a ContextPack, work from the pack, ask for a named follow-up. The write path never packs: open a session, propose, commit as an author, PR. The work path is WikiTicket first, PKC second, then a Feature pack before anyone codes. The agent-system path is AGER first: author or reverse-engineer, bind knowledge, validate twice (AGER rules, then OKF graph), compile last.

LoopControls and ToolRules are supposed to travel with the graph rather than be reintroduced as framework glue. That is the whole reason compile exists. If a path starts at a host and works backward into Git, you are doing AGKC, not operations.

Where things stand today

Other gaps worth keeping in view: marketplace fragmentation (individual repos still advertise their own marketplaces next to spillwave-second-brain); isolation cost; SAC’s 139-noun catalog versus a tight ontology; no shared runtime semantics proving LoopControl fidelity across translators; public samples are fictional on purpose (Northstar, Lumenfield).

What follows

If you are evaluating this as infrastructure, the honest sequence is not “install the translators.” It is: stand up WikiTicket, so work is visible; capture a month of decisions in PKC so the why exists; pack Features before agents touch them; then, and only then, author one AgentGraph with LoopControls you would actually enforce. Reverse-engineering an inherited agent repo into AGER is useful even if you never compile it. The spec is the review artifact.

If you are evaluating this as a product thesis, the claim is larger than a plugin marketplace. The industry still handles institutional memory as chat and tickets, and multi-agent systems as framework samples. This stack insists that both are graphs, both are Git-native, both are validatable, and neither should be reconstructed from folklore at the start of every session.

OKF is how you stop treating knowledge as prompt stuffing. AGER is how a multi-agent system stops living as folklore. The rest is noun ownership and the discipline not to invent a rel.

This is what this is all about. The sources for every version pin, noun list, and design claim in this article are the public GitHub repositories listed in the research section: READMEs, specs, changelogs, and design docs as they stood on 26 August 2026.

Conclusion

The hard part was never generating the next file. It was keeping the project in your head.

If it is hard for you to hold the last six months of decisions, rejected alternatives, CDK layout conventions, and “why we did not use that framework feature,” it is hard for a coding agent too. Context windows do not fix that. Prompt stuffing pretends that the important node is in the first 20,000 tokens. Usually it is not.

This stack exists so that knowledge does not have to live in anyone’s working memory. Meetings, experiments, assumptions, architecture, schemas, deployments, and the loop that is allowed to stop all become typed nodes in Git. The agent does not only see that a decision was made. It can pack the Feature, the Decision that authorized it, the Experiment that almost killed it, and the neighboring CDK or integration example that shows how this team actually ships.

That is more than a spec. A spec is a snapshot of intent at one moment. The graph is a snapshot you can grow. Research a new framework, write the Discovery. Decide which parts you will use, write the DecisionRecord. Point the Feature at both. The next agent that touches the UI that will consume that framework is not starting from folklore. It is starting from a bounded pack.

Agentic skills teach procedure. This teaches the project: why, what was rejected, how deployments are laid out, how the last integration was done, and which research already settled the argument. That is how you avoid the expensive regressions that look like “the model just did not know.”

Try it in that order, not as a marketplace binge. Stand up a WikiTicket, so work is visible. Capture a month of real decisions in PKC so the why exists. Then reverse-engineer the running estate with SAC so the graph knows deployments, architecture, load balancers, services, modules, and packages. Reverse-engineer the lakehouse with DEKC so it knows catalogs, lineage, ETL and ELT jobs, where data comes from, where it lands in reports and dashboards. Pack a Feature before an agent implements it. That pack is no longer just a ticket neighborhood. It is the Feature plus the Decision that authorized it plus the topology and data plane it will touch. That is how you see downstream impact instead of discovering it in production.

Then, if you care about multi-agent loops, author one AgentGraph with LoopControls you would actually enforce. Reverse-engineer an inherited agent repo into AGER even if you never compile it. The spec is already a review artifact.

Install from the public marketplace, point it at a knowledge tree you already own, and pack one Feature you are about to build. If the pack cannot explain the decision, that is the work. The graph is how the project keeps moving without asking everyone to keep the whole thing in their head.

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

LangFuse I: The Ticket You Cannot Reconstruct Is Why You Need Langfuse

Rick Hightower — Wed, 26 Aug 2026 22:28:42 GMT

The ticket you cannot reconstruct is not a logging problem; it is a missing record of the prompt, the model, and the reply, and Langfuse is the open-source layer that captures this record with a single import change.

Print logs vanish when the process dies. Langfuse LLM observability keeps the prompt, the reply, the tokens, and the cost.

A customer is furious about a refund; last night’s logs have a timestamp and nothing else, and you cannot tell whether the model invented the policy or you shipped a bad one. Print logs vanish when the process dies.

In this article: You will learn why print logs, a general APM, and a closed LLM-ops vendor all fail when you need to reconstruct a bad model reply. You will install the Langfuse Python SDK v4 or JS/TS SDK v5, point three environment variables at a Cloud project, and send one Harbor Desk support ticket through the official OpenAI client. By the end, you will have a generation in the UI: the system prompt, the ticket, the one-word label, tokens, and cost.

A customer writes in about a wet life jacket. Your assistant drafts a refund reply. The customer is furious. You open last night’s logs and find a timestamp, a status code, and nothing else. The prompt is gone. The model is a guess. The ticket text lives in a Slack screenshot. You cannot tell whether the model invented the policy or you shipped a bad one.

That is the hole Langfuse fills. It is an open-source AI engineering platform that provides observability, prompt management, evaluation, and metrics on a single data model, Cloud- or self-hosted. This article gets a single trace into a project. Nested trees, agent frameworks, judges, and experiments come later. Today you change one import, send one Harbor Desk ticket, and see the call in the UI.

Harbor Desk is a support-ticket assistant for a tiny shop. A ticket comes in. Eventually it will classify the intent, retrieve a policy snippet, and draft a reply. Today it does none of that. Today it is one chat completion, because you cannot judge a tree until a single generation shows up.

Three things you already have, and why none of them is this

You already debug LLM apps. You just do it with tools that were not built for them.

print() and log lines. Cheap, local, gone when the process exits. They do not group a request into a tree. They do not know what a token costs. They do not let a product person score last week’s replies.

A general APM. Great at latency and HTTP. Blind to prompts, completions, token usage, and evaluation scores. Langfuse is purpose-built for LLM applications: it natively understands token usage, model parameters, prompt-completion pairs, and scores. It also does the things a Datadog-shaped tool will not: LLM-as-a-judge, prompt management, experiments and datasets, and custom dashboards.

A closed LLM-ops vendor. The features exist. The data and the server are not yours. Langfuse is open source and self-hostable. Cloud is the fastest start. Self-hosting is the same architecture when you need it.

Langfuse is not an agent framework. It does not replace the Claude Agent SDK or DeepAgents. It is the layer you wrap around the loop you already have. LiteLLM is a different product too: a multi-provider SDK and proxy that can emit to Langfuse. It is not the default path here. The default path is Langfuse’s drop-in of the official OpenAI SDK.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems: Workshop

Engineering Reliable Agentic AI Systems
Saturday, 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

Install it, then give it keys

This series uses Python SDK v4 (langfuse on PyPI) and JS/TS SDK v5. Both sit on OpenTelemetry. Pick your language.

pip install langfuse openai

npm install @langfuse/openai openai @langfuse/otel @opentelemetry/sdk-node

The Python drop-in is compatible with OpenAI SDK versions >=0.27.8. Async and streaming need >=1.0.0. The JS wrapper requires the OpenAI SDK >= 4.0.0.

Create a project on Langfuse Cloud or self-host it, and set the keys in the environment. LANGFUSE_BASE_URL is the region, not a vanity URL.

LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_BASE_URL=https://cloud.langfuse.com

EU is https://cloud.langfuse.com. US is https://us.cloud.langfuse.com. Japan is https://jp.cloud.langfuse.com. HIPAA is https://hipaa.cloud.langfuse.com. Point LANGFUSE_BASE_URL at your self-hosted origin if that is where the project lives.

Hello, Harbor Desk

The smallest useful program is not “hello world.” It is the ticket you could not reconstruct: one completion, captured.

In Python, Langfuse is a drop-in import for the official OpenAI client. You keep chat.completions.create. You change the import. Azure OpenAI works the same way via AzureOpenAI from langfuse.openai.

from langfuse import get_client
from langfuse.openai import openai  # ①

TICKET = (
    "Order 4412. The life jacket arrived soaked and smells like bilge. "
    "I want a refund, not a replacement."
)

completion = openai.chat.completions.create(  # ②
    name="classify-intent",  # ③
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": (
                "You classify harbor-shop support tickets. "
                "Reply with exactly one word: refund, shipping, or other."
            ),
        },
        {"role": "user", "content": TICKET},
    ],
)

print(completion.choices[0].message.content)

# Short-lived scripts must flush or the process exits before the batch ships.
get_client().flush()  # ④

① The drop-in import keeps the official OpenAI client API; only the import path changes. ② chat.completions.create is the same call you already write; Langfuse records it in the background. ③ name is Langfuse’s observation name, not an OpenAI argument. ④ flush() ships the queued events before a short-lived process exits.

Note: The full extracted listing at code/langfuse/part-1-what-langfuse-is/listings/01-classify-intent.py is the complete runnable program.

The name="classify-intent" argument is Langfuse’s, not OpenAI’s. The drop-in accepts it and uses it as the observation name. You will treat that name as an API later, when evaluators and dashboards start targeting it. For now, run the script and open the project. You should see a generation: the system prompt, the ticket, the one-word label, tokens, and cost.

TypeScript is the same idea with one extra step. The JS SDK traces via OpenTelemetry, so you start NodeSDK with a LangfuseSpanProcessor before wrapping the client. Then you wrap the official OpenAI instance with observeOpenAI. That is not LiteLLM. It is the official OpenAI client, observed.

// instrumentation.ts: import this first
import { NodeSDK } from "@opentelemetry/sdk-node";
import { LangfuseSpanProcessor } from "@langfuse/otel";

export const langfuseSpanProcessor = new LangfuseSpanProcessor();

export const sdk = new NodeSDK({
  spanProcessors: [langfuseSpanProcessor],
});

sdk.start();

The classify script imports that instrumentation first, wraps the official client, and flushes before the process exits.

import "./instrumentation"; // must be first  ①
import OpenAI from "openai";
import { observeOpenAI } from "@langfuse/openai";

const openai = observeOpenAI(new OpenAI(), {  // ②
  generationName: "classify-intent",  // ③
});

const ticket =
  "Order 4412. The life jacket arrived soaked and smells like bilge. " +
  "I want a refund, not a replacement.";

const completion = await openai.chat.completions.create({  // ④
  model: "gpt-4o",
  messages: [
    {
      role: "system",
      content:
        "You classify harbor-shop support tickets. " +
        "Reply with exactly one word: refund, shipping, or other.",
    },
    { role: "user", content: ticket },
  ],
});

console.log(completion.choices[0].message.content);

// Short-lived processes: flush before exit.
await langfuseSpanProcessor.forceFlush();  // ⑤
await sdk.shutdown();  // ⑥

① Import the OpenTelemetry bootstrap first so the span processor is running before any OpenAI call. ② observeOpenAI wraps the official client; it is neither a proxy nor LiteLLM. ③ generationName is the observation name Langfuse will show in the UI. ④ The create call is the same official OpenAI request; tracing happens around it. ⑤ forceFlush() ships the queued spans before the process exits. ⑥ sdk.shutdown() tears down the OpenTelemetry SDK after the flush.

Note: The full extracted listing at code/langfuse/part-1-what-langfuse-is/listings/02-classify-intent.ts is the complete classify script.

Read what just happened. You did not wrap a proxy. You did not stand up an agent. You called OpenAI as you already do, and Langfuse recorded the generation in the background. Tracing is asynchronous: the SDKs queue events and flush them in batches so that the request path does not wait on Langfuse. That is also why a script that exits immediately looks broken.

Gotcha: If the UI is empty after a one-shot script, you skipped flush() in Python or forceFlush() / sdk.shutdown() in JS. The exporter never got a turn. Long-running servers do not need this on every request. Short-lived jobs do, every time.

In production: Keep keys in the environment, never in the repo. LANGFUSE_BASE_URL is how you point staging at a self-hosted origin and production at Cloud, or the other way around, without touching code.

Harbor Desk, one layer at a time

Here is the arc, so the order is not a surprise. Next comes the data model: observations, traces, sessions, and the observation types generation, retriever, and tool that make cost and evals work. Harbor Desk grows from one classify call into a nested tree. Those observation names become an API. Users, sessions, tags, and environments hang on the tree. Then you wrap the Claude Agent SDK and DeepAgents loops you may already ship, and you name LiteLLM so you do not confuse it with the drop-in you just used.

Then the engineering loop: prompts out of code, scores, and experiments. Then shipping: dashboards and cost, the public API, masking, and self-hosting. Every step adds one layer that the previous step left you reaching for.

Do this today

Before you do anything else, get one generation into a project. Five minutes, four steps.

Create a Cloud project at cloud.langfuse.com, or point at a self-hosted origin if you already have one.
Export the three env vars. Set LANGFUSE_SECRET_KEY, LANGFUSE_PUBLIC_KEY, and LANGFUSE_BASE_URL (EU, US, Japan, HIPAA, or your origin).
Install the SDK for the language you actually write: pip install langfuse openai or npm install @langfuse/openai openai @langfuse/otel @opentelemetry/sdk-node.
Run the classify script, call flush() or forceFlush() before exit, and open the generation. You should see the soaked life jacket ticket sitting next to the word "refund".

The ticket is a missing record

The ticket you cannot reconstruct is not a logging problem. It is a missing record of the prompt, the model, and the reply. Langfuse is that record, plus the prompt store, the scores, and the experiments you will hang on it later. It is not the agent. It is not LiteLLM. It is the layer around whichever of those you already run.

Do the five-minute version first. Create a Cloud project, set up the three env vars, run the classify script, and open the generation. Once you have seen the soaked-life-jacket ticket sitting next to the word refund, the empty-log story stops being abstract. Then you give that generation a tree to live in.

About the author

Rick Hightower is a former Senior Distinguished Engineer at a Fortune 100 company and a practitioner building multi-agent production systems. He is a Claude Certified Architect. Follow him on Substack and Medium, and read more from the team at Spillwave on Medium.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

The Loop Is the Product

Rick Hightower — Wed, 26 Aug 2026 20:03:37 GMT

A high-powered agent without a governor is a turbine spinning toward its own destruction. Reliability is loop engineering, harness engineering, and graph engineering; a closed control system, not a better prompt.

Subscribe now

A high-powered AI system, operating without a governing mechanism, is a turbine spinning toward its own destruction. It generates raw energy. It has no way to keep from shaking itself apart.

Ungoverned generation is not a product. It is a machine hall with the governor removed.

Most teams still approach these systems through prompt engineering: instructions in, a text generation out. That is an open loop. For a simple chatbot, an open loop is enough. For autonomous software that must run across dozens of turns, it does not provide the stability the work requires.

In enterprise deployments, the failures are now familiar. Context rot: as the task horizon expands, the original objective is buried under intermediate observations until the model loses direction. Tool storms: with no measure of utility, the agent fires a rapid sequence of redundant or hallucinated calls, burning thousands of tokens with no measurable progress. False completeness: it declares victory on unfinished work because it was allowed to grade its own exam.

Reliability is not found by hunting for a perfect prompt. It is found by architecting a governed control system. The agent is a state machine that tracks progress turn by turn: loop engineering. Because a model cannot govern its own state, we wrap that cycle in an external validation layer: the harness. When the work is too wide for one window, we fan it across isolated workers: graph engineering. Autonomous production lives entirely in that iteration cycle.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems: Workshop

Engineering Reliable Agentic AI Systems
Saturday, 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

Open loop, closed loop

When we evaluate isolated completions, we are grading the wrong object. Production agents navigate non-deterministic tool output, long task horizons, and an operational space that expands with every file they open. A single generation can look fluent and still be one wrong parameter away from a silent side quest.

The move is architectural. Treat the model as a proposer inside a closed-loop control system, not as an open-loop text generator. Software owns the transitions. Software owns the stop. The model proposes the next action under a contract it did not write and cannot waive.

Autonomy that cannot be stopped from outside the model is not autonomy. It is an unbounded process with a vocabulary.

This is not a slight against frontier models. It is mechanical sympathy for what they are: autoregressive engines trained to continue, biased by RLHF toward internal consistency, and structurally unsuited to being both the author of an artifact and the auditor of whether that artifact is done. Banking learned this a century ago. Maker and checker must be different things.

Four planes, not one conversation

Collapsing intent, execution, verification, and control into a single chat thread is the original sin of unharnessed agents. A production topology splits them:

The specification plane is a contract, not a vibe. Vague tickets become outcomes, scope boundaries, constraints, budgets, and an explicit definition of done. Mature systems go further and split the contract in two: a public rubric the maker can see, interfaces, constraints, smoke tests, and a hidden test suite the harness retains. The agent cannot overfit what it cannot read.

The control plane is the harness. It is not a prompt. It is software that decides whether a proposed tool call is in scope, whether the budget still has room, whether the last two actions were the same failing hash, and whether the run should retry, escalate, or halt. If that logic lives in the model’s head, you do not have a control plane. You have a suggestion.

Five nodes of the loop

Loop engineering is the practice of designing that state machine. Not “keep going until you feel done.” A methodical cycle on five nodes: trigger, action, verify, memory, and exit. The model is proposed inside the cycle. Software owns the transitions.

Trigger: typed, idempotent, bounded

A production trigger is a strongly typed event payload:

a webhook,
a queue message,
a scheduled flag
not a natural-language request that happens to contain the word “please.”

A trigger names the exact resource the agent is authorized to touch. And it carries a stable work ID. Idempotency is not a nicety. If the system retries, that ID is what prevents destructive duplicate writes.

Action: scoped, schema-checked

A flat namespace of every tool the platform owns is a gift to confusion and prompt injection. The loop exposes the minimum capability for the current phase: a targeted read or edit, not an unrestricted shell. Parameters pass client-side JSON schema validation before anything executes. Hallucinated arguments die at the door, not in production.

Verify: evidence, not a pep talk

This is the load-bearing joint. Asking the working model whether it succeeded is an invitation to sycophancy. Verification is grounded in evidence from the environment: exit codes, type checkers, unit tests. If a qualitative judgment is unavoidable, it is a separate model call against a fixed rubric: never the generating model checking its own homework!

Memory: prune, offload, point

Consecutive tool calls flood the window with console, file trees, and search payloads. Attention decays. The Chroma “Context Rot” evaluation in 2025 ran 18 frontier models (GPT, Claude, Gemini, Qwen among them), and found performance to be increasingly unreliable as input length increased, even for simple retrieval, well below advertised context limits. “Lost in the middle” is not a metaphor. Recall drops more than 30% when the fact you need sits in the crowded center of the prompt.

Context rot is an attention failure, not a storage failure. The fact you need is often still in the window. The model can no longer find it.

Loop engineering answers with externalization. Heavy logs go to disk or a database. The prompt keeps a pointer. Sub-agents run the long tool paths in their own windows and return a structured brief. The orchestrator never inherits the dump.

Exit: the model does not vote

An agent can never be trusted to decide when to stop its own work. We enforce explicit state limits from the outside: hard caps on iterations, a cumulative token budget, and action hashing that mathematically detects a stall. If the agent repeats the same parameters across consecutive turns, the harness terminates the loop. A reliable loop assumes the model will eventually drift. External rules are the only way to force a return to the objective.

Harness engineering

If loop engineering designs the state machine, harness engineering is the software that encapsulates, evaluates, and constrains it. The harness is the unit of reliability. It is also production code, which means it can fail; SWE-bench Verified audits have shown evaluation harnesses rejecting correct implementations. Treat it with the same CI, versioning, and expert review you would give a payments service.

The harness is the control room. The model is not allowed to sit in the operator’s chair and also certify the gauges.

A mature harness does five things that prompts cannot.

It enforces the maker-checker split and context quarantine, which we will take up in full. It compiles human intent into a two-tier contract. It routes evaluation through the hardest grader available. It isolates context through sub-agent graphs. And it owns scarcity, like iteration caps, token and dollar ceilings, and action-hash circuit breakers that abort a stall rather than pay for another identical failure.

Hard graders first is not a taste. It is an economic and epistemic fact. A compiler does not agree with you sycophantically. A linter does not do ceremonies. A pytest suite does not suffer “lost in the middle.” Spend model tokens on the residual: tone, design, semantic fit, after the mechanical gates have spoken.

There is a quieter production lesson. An autonomous loop is a rapid amplifier of local convention. If the repository permits sloppy abstractions, the agent will generate them faster than any intern. The harness must make poor convention mechanically expensive: repository linters, AST rules, structural tests, documentation the agent cannot skip. You are not only governing the model. You are governing the gradient it will follow.

Never let the AI verify its own work

This is the doctrine, borrowed from the segregation of duties and grounded in Clark-Wilson and NIST AC-5: the maker and the checker must be different. Not different prompts in the same thread. Different roles, different tools, different context, and, for high stakes, different model families.

The wall is the architecture. The pass-through is the artifact. The checker never inherits the maker’s scratchpad.

Intrinsic self-correction, confined to one instance, is not merely weak. It is often counterproductive. Stechly et al. and Huang et al. showed that self-critique without an external signal dropped Graph Coloring from 16% to 2% and Game of 24 from 5% to 3%. A sound external verifier raised performance to 38%. Relabeling identical claims so they appear to come from an external reviewer lifts explicit correction rates by 23 to 93 percentage points. Specialist-swap ablations that moved verification back onto the generating model cut SpreadsheetBench rescues from six tasks to two.

Three structural boundaries keep the split from becoming theater.

Capability asymmetry. The maker holds mutating tools. The checker holds strictly fewer, ideally none that write. Give the checker a shell, and it will “fix” the evidence, introduce secondary bugs, and destroy the ability to tell maker performance from checker repair. Equal access to tools is how the split dies. It is also how you get destructive edit wars.

Context quarantine. The checker does not inherit the maker’s chain of thought, tool transcript, or self-justifying narrative. Rationale biases the evaluator. The checker scores the raw artifact against the specification, the way a compiler scores source and not the author’s commit message.

Distinct model families for high-stakes work. A Claude maker with a GPT checker, or the reverse, so shared blind spots are not signed off by their twin.

Reflection asks, “Look at what you just did. Is it correct?” Verification says, “Here is the test output. Given this evidence, what remains wrong?”

Implement the split at the cheapest sufficient tier. Programmatic hooks, compilers, AST, pytest, cost zero extra tokens, and should be the default gate. Platform-managed graders return structured states (satisfied, needs_revision, failed) at medium cost. A separate evaluator-optimizer model call is reserved for subjective criteria. Most production loops are under-using the first and over-using the third.

Graph engineering

Graph engineering is about keeping a loop from becoming a single gigantic prompt. Decompose work into a graph of specialized sub-agents connected by an explicit state machine. The lead orchestrator sits outside worker loops and owns what workers are never allowed to redefine: scarcity, terminal outcomes, and the independence of control logic from whichever model is currently fashionable.

Workers run on isolated tracks. The tower owns the switches, the budget, and the decision to mark the yard as done.

Anthropic’s research classifies five canonical workflows rather than a generic swarm: prompt chaining; routing; parallelization by sectioning or voting; orchestrator-workers; and evaluator-optimizer. The last of those is maker-checker by another name. Use the smallest graph that enforces the boundary you need. Swarms without a scarcity owner are how you buy a 15× token multiplier.

Four context-engineering patterns make the firebreak hold under volume. Quarantine: independent threads per sub-agent, so scratchpads cannot contaminate each other. Offloading: heavy payloads to disk, pointers in the prompt. Pruning: drop low-signal tokens: intermediate arguments, verbose stdout. Summarization middleware: when the window hits a threshold (75 to 85%), a fast model compresses history into a structured brief so the session can continue without carrying the whole carcass.

Graphs are not a universal upgrade. Tightly interdependent, multi-hop coding often wants a single general-purpose agent on a carefully managed window; Anthropic’s own coding-agent research left open whether specialized sub-agents outperform that shape. Breadth-first research over long horizons is where isolation pays. Handoffs introduce interpretive drift. Pay that tax only when the alternative is drowning the orchestrator.

The MCP gateway

Gathering external data introduces a second failure mode that graphs alone cannot solve: a flat tool namespace. The agent can see every function the platform offers, which invites prompt injection and privilege escalation. The fix is a mandatory proxy between the loop and the outside world: an MCP gateway.

The gateway enforces least privilege so the agent cannot dynamically reach unapproved tools during the run. It cleans untrusted data: intercepts raw HTML or DOM payloads and strips them, passing only the core information into memory. And it intercepts API errors, formatting them as structured JSON so the agent cannot hallucinate a recovery from a raw stack trace. Those three jobs protect the model from external noise and the enterprise from unguided, destructive action.

Failure is a systems problem

The Multi-Agent System Failure Taxonomy (MAST), presented at NeurIPS 2025, annotated more than 1,600 traces across seven frameworks. Failure rates ran from 41% to 86.7%. The mass of the distribution sits in system design and verification, not in “the model was not smart enough.”

Read the top of that list as a harness specification. Step repetition is an action-hash circuit breaker you did not write. Unaware of termination is a stop rule the model was allowed to own. Incorrect verification is a maker scoring its own exam. Reasoning-action mismatch is a missing pre-execution schema and a checker that never saw the tool log. You can prompt against these. You cannot prompt them out of existence.

Unattended is a different machine

A laptop loop with a human watching is not production. In interactive use, a person notices the stall, kills the runaway, and decides “good enough.” Unattended, the unit of risk is the whole trajectory. The maker-checker-harness architecture stays the same. The envelope around it has to become durable, observable, and interruptible.

Last-known-good is a lever, not a memory. If verification fails, you roll back to a commit, not to whatever the chat still believes.

Durable execution means the loop is a persistent state machine. Every completion, tool call, and tool result is journaled to an append-only log. A crashed worker rehydrates by replay, with side effects cached so you do not re-pay for inference. Checkpoints at each transition make idempotency a first-class concern: the resumed node may try to re-run a write that already happened.

Last-known-good is external. For coding agents, start from a commit SHA and a feature manifest; commit incrementally; on failed verification, roll the worktree back. For research and automation, freeze database states or document snapshots. Do not roll back to “whatever the model remembers.” Chat history is not a restore point.

Observability has to answer four questions on every span: what is the agent doing, why is it allowed to, what evidence says it is correct, and why has it not stopped. Contract versions, scores, tool arguments, durations, and budget state belong in the trace, not only the transcript. Time-travel replay from serializable checkpoints turns a 3 a.m. failure into a reproducible fixture. Risky traces become tomorrow’s golden set. That closed loop, from production back into eval, is how the harness itself improves.

Human-in-the-loop is a state, not a hope. High-consequence actions (merge, deploy, move funds) transition to PENDING_APPROVAL. State serializes. Token spend pauses. An authenticated signal resumes. Risk-class the rest: read-only auto-executes inside budgets; speculative writes stay in a sandbox; collaborative posts may gate on confidence. A flat permission model is how irreversible actions sneak through on a Tuesday.

And because the harness is software, it ships through CI. Prompt change, tool update, model swap: golden-set trajectory scores against a baseline before the deploy proceeds. Canaries. Weekly eval, not quarterly; a 2% silent drop is invisible day to day and catastrophic by the quarter. Retrospective harness optimization can refine tool specs from traces, but it does not replace the expert audit. Flawed tests reject correct work. Someone has to read the contract.

The loop is the product

Reliability has very little to do with the raw intelligence of the base model. True autonomy is found by moving correctness and control completely outside the neural network. Prompt engineering taught us how to talk to models. That skill is not obsolete. It is no longer the load-bearing wall.

The load-bearing wall is the loop: a typed trigger, a scoped action, evidence that does not come from the author, memory that does not rot in the window, and an exit the model cannot waive. Around that loop, a harness that treats maker and checker as different organs, grades with the hardest instrument first, and owns scarcity like an SRE owns an error budget. Around that harness, a graph only as large as the firebreak you need, and a gateway that will not let the loop touch the world with a flat, ungoverned namespace.

We have moved past prompt engineering and into systems engineering. The demo will keep working. Demos always do. The question that separates a prototype from a production agent is simpler than the literature makes it sound, and harder than a system prompt can answer: when this thing is wrong, who is allowed to say so, and what deterministically happens next?

If the answer is “the same model, in the same thread, in its own words,” you do not have a reliable agent. You have a turbine with the governor removed. And it will eventually tell you it is done.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

About the Author: Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

Claude Certified Architect: The choke point: tool-call interception and the CCA-F Exam

Rick Hightower — Wed, 26 Aug 2026 00:06:13 GMT

The Claude Certified Architect exam asks the same question in many ways. Where does a guarantee live? The answer is never the prompt. It is always the place where a tool call passes through code.

Subscribe now

The place has three different names, one per abstraction you build on. On the raw Messages API, it is your dispatch function. On the Claude Agent SDK, it is a PreToolUse hook inside a six-step permission pipeline. In LangChain Deep Agents, it is middleware that wraps the handler.

The exam tests the first two. The third is distractor territory, and knowing it cold is what makes the distractor obvious.

The one rule the exam repeats

The routing heuristic appears across the CCA preparation material in almost the same words:

A requirement that must be guaranteed routes to code enforced in a hook, gate, or schema, which is the winning answer. A requirement that is only advisory routes to the prompt.

Stated as a test-taking rule:

When a question offers “add instructions to the system prompt” and “configure a programmatic hook,” the hook is almost always correct.
If a question involves financial or security harm and offers you a system-prompt rule against a hook, the hook wins essentially every time.

The principle underneath is one sentence. System prompts are probabilistic controls, not guarantees. They must be paired with architectural enforcement.

Level 1: the Messages API, where you are the hook

Start at the bottom, because the exam does.

On the raw Claude Messages API there are no hooks. The preparation material is direct about why that is not a problem:

The dispatch function that runs every tool is already the choke point where every guarantee belongs.

One function turns the model’s tool_use request into an actual action. Every call passes through it. Put the guardrail there and nothing routes around it.

This is the level most candidates underrate. A question that shows a bare API loop is not asking you to reach for a framework. It is asking whether you can see the choke point that already exists.

Level 2: the Agent SDK, where the pipeline is the product

The Agent SDK gives you that enforcement point as a named, ordered mechanism. When the model requests a tool, the SDK evaluates permissions in this order:

Hooks. A PreToolUse hook runs first and can deny outright. A hook that returns allow does not skip the deny and ask rules below.
Deny rules. A match blocks the tool even in bypassPermissions mode.
Ask rules. A match falls through to canUseTool, even in bypassPermissions mode.
Permission mode. bypassPermissions approves. acceptEdits approves file operations. plan routes writes to canUseTool regardless of allow rules.
Allow rules. A match approves.
canUseTool callback. The fallback. dontAsk mode skips it and denies.

The order is the reason a hook beats a prompt on the exam. A deny rule sits above the permission mode, so no mode overrides it. A prompt instruction overrides nothing.

Two mechanics the exam names directly:

HookMatcher binds a callback to a tool. Registering with a matcher means the hook runs only for process_refund, not for every call.
updatedInput rewrites arguments. The preparation material uses the example of a hook that clamps a refund amount. The pipeline is a transformer, not only a gate.

The exam expects you to draw the full path. The model proposes an action. A PreToolUse hook denies it, escalates it to human approval, or allows it. The tool then runs. A PostToolUse hook writes the audit trail before the result returns to the model.

A HookMatcher groups callback(s) behind a matcher. For PreToolUse or PostToolUse, the matcher filters on the tool name, so a callback registered with "process_refund" runs for that tool’s invocation rather than every tool call. A matcher of None, "", or a catch-all pattern would apply to all tools instead.

hooks = {
    “PreToolUse”: [
        HookMatcher(
            matcher=”process_refund”,
            hooks=[validate_refund],
        )
    ]
}

Level 3: Deep Agents, which the exam does not test

LangChain Deep Agents wraps the handler instead of gating ahead of it:

@wrap_tool_call
def guard(request, handler):
    result = handler(request)
    return result

Your function receives the request and the handler. You decide when to call the handler, and you get the result back. Human approval is a separate mechanism, interrupt_on, which needs a checkpointer. Path policy is a third mechanism, enforced by subclassing or wrapping a backend.

None of that is on the CCA exam. LangChain appears in the preparation material twice, and neither mention is about interception. One compares credential paths. The other is an explicit trap:

Important exam distinction: with_structured_output() is a LangChain method, not a native Anthropic SDK method.

This is the connection. The exam tests whether you can tell the Anthropic surface from a framework surface. wrap_tool_call, interrupt_on, and BackendProtocol are Deep Agents. PreToolUse, HookMatcher, canUseTool, and allowed_tools are Anthropic. A candidate who has used both frameworks reads a framework method in an answer choice and eliminates it in one second.

Where this sits in the domain weights

Three domains totaling 65% touch the choke point directly. Domain 4 is where the wrong answers come from.

The distractor patterns

The preparation material names the shapes the exam uses. Each one is a variation on putting a guarantee somewhere it cannot be guaranteed.

Confidence as a routing mechanism. Watch for self-reported confidence, model certainty, or “if the agent is uncertain, escalate” used as the routing decision. Any such answer is almost certainly wrong. Deterministic rules route. Confidence does not.

Prompt instructions for structured output. Any answer that relies solely on prompt instructions for structured output is a distractor. A JSON schema guarantees shape, not truth, and a prompt guarantees neither.

A system-prompt rule against a hook. When money or security is involved, the hook wins.

The material also notes how strong candidates use these:

Candidates who score in the 900s report that their speed came from recognizing trap patterns instantly, not from reading every word carefully.

What the mechanism adds beyond the prep material

Two facts from the SDK documentation that the preparation articles do not cover. Both matter more in production than on the exam.

The timeout behavior changed, and the old behavior was unsafe.

If your enforcement point is a callback, the old version let calls through on timeout. The exam will not ask this. A security review will.

You can write a canUseTool callback that never runs. As of v2.1.198, the TypeScript SDK emits a process warning, CLAUDE_SDK_CAN_USE_TOOL_SHADOWED, when your callback is unreachable. Two configurations cause it: bypassPermissions, which approves at step 4, and a bare allowedTools entry such as "Read", which approves at step 5.

The SDK’s own advice matches the exam’s answer key. To gate every tool call regardless of mode and rules, use a PreToolUse hook rather than canUseTool.

A study checklist

For any scenario, name the choke point first. Dispatch function, PreToolUse hook, or middleware, depending on the level.
Read every answer choice for the word “prompt.” If a guarantee depends on it, eliminate the choice.
Read every answer choice for confidence-based routing. Eliminate it.
Recognize framework methods that are not Anthropic. with_structured_output, wrap_tool_call, interrupt_on.
Know the six-step order well enough to answer why a deny rule survives bypassPermissions.
Remember that updatedInput transforms as well as blocks.

The exam is not testing whether you memorized an API. It tests whether you reach for enforcement or persuasion when the requirement is a guarantee.

Source note

The exam material here comes from 36 published Claude Certified Architect articles, 15 on Medium and 21 on Substack.

Those articles describe the exam. They are not the official exam guide.

The SDK mechanics come from the vendored Claude Agent SDK documentation in the same graph, which is versioned at the release level.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday, 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

Read the whole series:

Part 1: Anthropic’s Claude Certified Architect Exam (CCA-F): You Cannot Memorize Your Way Through the CCA-F

Anthropic’s Claude Certified Architect Exam (CCA-F): You Cannot Memorize Your Way Through the CCA-F

Rick Hightower

Jun 16

Anyone can name the right pattern when the question hands it to them, but production never hands it to you; it hands you a system that is on fire and a clock that is running. That gap, between knowing what context windows are and feeling, at 2 a.m., that the agent is failing because the context blew past its budget, is the one skill no quiz score will ever prove you have.

Read full story

Part 2: Anthropic’s Claude Certified Architect Exam (CCA-F): The Coordinator That Shared Too Much, and the Loop That Never Stopped

Anthropic’s Claude Certified Architect Exam (CCA-F): The Coordinator That Shared Too Much, and the Loop That Never Stopped

Rick Hightower

Jun 16

Your agent does not finish when it says it is finished, it finishes when the model says so, and a coordinator that hands a subagent everything it knows is one leaked field away from a breach. Drive the loop off stop_reason and give each subagent only the context it needs, and the production system you ship stops guessing and stops oversharing.

Read full story

Part 3: Anthropic’s Claude Certified Architect Exam (CCA-F): Your Tools Are Lying About Their Errors (Domain 2)

Anthropic’s Claude Certified Architect Exam (CCA-F): Your Tools Are Lying About Their Errors

Rick Hightower

Jun 17

A tool that swallows a timeout and returns an empty list does not just fail, it teaches your model a falsehood and watches it act on it. Return a structured error the model can reason about, and the same failure becomes a recovery instead of a confident lie to your user.

Read full story

Part 4: Anthropic’s Claude Certified Architect Exam (CCA-F): CLAUDE.md Is a Precedence Puzzle, Not a README, (Domain 3)

Anthropic’s Claude Certified Architect Exam (CCA-F): CLAUDE.md Is a Precedence Puzzle, Not a README, Domain 3

Rick Hightower

Jun 18

Your config quietly fights itself, and you only notice when Claude ignores a rule you swear you wrote. Learn what actually overrides what, and your CLAUDE.md stops being a guessing game.

Read full story

Part 5: Anthropic’s Claude Certified Architect Exam (CCA-F): The Schema Validated. The Data Was Still Wrong (Domain 3)

Anthropic’s Claude Certified Architect Exam (CCA-F): The Schema Validated. The Data Was Still Wrong.

Rick Hightower

Jun 20

A model response can pass every check in your JSON schema and still assert something that never happened. The skill that separates working systems from quiet disasters is knowing that syntactic validity is not semantic truth, and learning when to retry the call versus repair the schema itself.

Read full story

Part 6: Anthropic’s Claude Certified Architect Exam (CCA-F): Everything Important Was in the Middle, So Claude Forgot It (Domain 5)

Anthropic’s Claude Certified Architect Exam (CCA-F): Everything Important Was in the Middle, So Claude Forgot It

Rick Hightower

Jun 22

Your model nailed every fact in the demo, then dropped the one that mattered the moment the conversation got long. The fix is not a bigger window; it is rolling history, pinned facts, prompt caching, and two-stage retrieval so the detail you depend on never sinks into the middle and disappears.

Read full story

Part 7: Anthropic’s Claude Certified Architect Exam (CCA-F): The Agent That Did Exactly What the Injected Text Told It To

Anthropic’s Claude Certified Architect Exam (CCA-F): The Agent That Did Exactly What the Injected Text Told It To

Rick Hightower

Jun 22

You can spend a week word-smithing the perfect “please refuse malicious instructions” system prompt and still watch one line of untrusted text walk your agent into a destructive action. Real safety lives at the tool boundary, in a deny, ask, or allow gate that ranks the operator over the user over anything the model merely reads.

Read full story

Part 8: Anthropic’s Claude Certified Architect Exam (CCA-F): The 40 Percent the CCA-F Practice Exam Never Showed You

Anthropic’s Claude Certified Architect Exam (CCA-F): The 40 Percent the CCA-F Practice Exam Never Showed You

Rick Hightower

Jun 24

Every production incident hands you four plausible fixes, and three of them are just a reworded prompt that makes the symptom disappear until tomorrow. The skill that ships reliable Claude systems is reading past the tempting quick patch to the one change that repairs the root cause.

Read full story

Part 9: Anthropic’s Claude Certified Architect Exam (CCA-F): The Scenario You Draw Is Random. Its Shape Is Not.

Anthropic’s Claude Certified Architect Exam (CCA-F): The Scenario You Draw Is Random. Its Shape Is Not.

Rick Hightower

Jun 23

Your production incidents are not snowflakes. They repeat in a handful of shapes, and once you can name the anti-pattern behind the story and call out the real trade-off, like sync versus batch, you stop patching symptoms and start fixing the root cause.

Read full story

Part 10: Anthropic’s Claude Certified Architect Exam (CCA-F): Scored 1000 on Practice, Failed the Live Exam With a 590: A CCA-F Exam Strategy That Closes the Gap

Anthropic’s Claude Certified Architect Exam (CCA-F): Scored 1000 on Practice, Failed the Live Exam With a 590: A CCA-F Exam Strategy That Closes the Gap

Rick Hightower

Jun 24

The discipline that beats a hard time budget is the same one that gets you through a production incident. Read the whole thing once, answer in clusters, flag what stalls you, and move on, and never leave a decision blank.

Read full story

Key Terminology Glossary

These are testable. Know them precisely, not approximately.

Agentic Loop: receive input, reason, call tools, evaluate results, decide next step.
Coordinator-Subagent Pattern: a coordinator delegates to specialized subagents and synthesizes results.
Context Forking: splitting context into separate branches for different subagents.
Stop Reason: the API’s signal for why generation stopped (tool_use, end_turn, max_tokens).
tool_choice: API parameter controlling tool selection (auto, any, or a specific tool).
Validation-Retry Loop: check output against schema; on failure, return the error to Claude for correction.
Semantic Validation: checking the meaning of output (often LLM-as-judge), distinct from schema validation, which only checks format.
Hook / PreToolUse: code-level enforcement that intercepts a tool call to validate or hard-block it before it runs.
MCP Tools / Resources / Prompts: executable actions / read-only data for ingestion / reusable instruction templates.
JSON-RPC Protocol Error: a transport-layer failure (malformed request, method not found, invalid params) that occurs before any tool result exists. Distinct from an application-layer isError: true result.
Stateless API: the Messages API keeps no memory between calls; the application must pass history back in the messages array.
Lost in the Middle: the tendency of transformer models to under-attend to information in the center of the context window.
Attention Degradation: the drop in attention weight on early tokens as conversation history grows, even at a low total token count.

Resources

The CCA-F series (read alongside each domain, newest first)

The 40 Percent the CCA-F Practice Exam Never Showed You: the gap content in full
Scored 1000 on Practice, Failed the Live Exam With a 590: closing the practice-to-live gap
The Scenario You Draw Is Random. Its Shape Is Not.: scenario patterns
The Agent That Did Exactly What the Injected Text Told It To: prompt injection
Everything Important Was in the Middle, So Claude Forgot It: Domain 5
The Schema Validated. The Data Was Still Wrong.: Domain 4
CLAUDE.md Is a Precedence Puzzle, Not a README: Domain 3
Your Tools Are Lying About Their Errors: Domain 2
The Coordinator That Shared Too Much, and the Loop That Never Stopped: Domain 1
You Cannot Memorize Your Way Through the CCA-F: series opener and framing

Runnable code

Multi-agent researcher companion repo: clone it, run the notebooks; the test suite passes without an API key.

Claude Managed Agents XII: Your Agent Has No Deploy Button. It Is Already Deployed.

Rick Hightower — Tue, 25 Aug 2026 04:46:25 GMT

There is no deploy button for a managed agent because the agent and its environment are already the deployed resources. Shipping one to production is really four disciplines: provisioning, containment, lifecycle, and a real threat model.

Shipping a Claude Managed Agent to production is not the same as pushing an artifact. It is four separate disciplines: how you provision it, how you contain it, how you run it over time, and how you defend it from input it did not author.

In this article, you will learn why deploying a Claude Managed Agent is really a provisioning problem, not a publishing one. We cover the split between durable resources and ephemeral sessions, least-privilege networking, the 30-day session checkpoint clock and how to track cost, self-hosting when data cannot leave your boundary, and the prompt-injection threat model that should shape every choice. By the end, you can take a working prototype and harden it into something genuinely shippable.

Part 12 of Claude Managed Agent Series

Subscribe now

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday, 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

There is a moment in every agent project where the demo works. The agent ingests the files, calls the API, produces the report, and you think you are done. You are not. What you have is a script on your machine, running in a default cloud container, with open networking and credentials passed by hand. That is a prototype’s posture. It is fine for proving the idea and dangerous for running real work.

Here is the reframe that changes how you think about all of it. In Managed Agents, there is no separate deploy or publish step, because agents and environments are the deployed resources. The first time you called agents.create and environments.create, you were already deploying. Those calls returned server-side resources you reference by ID from then on. There is no build to ship and no artifact to push. A “deploy script” here is really a provisioning script.

That sounds like a small semantic point. It is not. It is the key to a clean production setup because once you see your agent and environment as live resources rather than code waiting to be uploaded, the right way to structure everything else becomes clear. Shipping an agent that takes real actions on untrusted input is not one concern but four: how you deploy it, how you contain it, how you run it over time, and how you defend it. This article walks all four.

Code is shown in Python and TypeScript where it matters.

Split provisioning from execution

Durable resources and disposable ones have different lifecycles, so they belong in different scripts.

Agents and environments are long-lived. You create them on a config change, not on every task. Sessions are ephemeral: one per task, created fresh each run. The clean pattern mirrors that split exactly. A deploy script stands up the environment and the agent roster and writes a manifest of their IDs. A separate run script loads that manifest and starts sessions.

The deploy script provisions everything durable and records the IDs. Here is the shape: provisioning an invoice agent’s environment and a multi-agent roster, then writing a manifest:

import json
from pathlib import Path
from anthropic import Anthropic

client = Anthropic()

def get_or_create_environment(name: str) -> str:
    # Environments are NOT versioned, so reuse by name instead of recreating.
    for env in client.beta.environments.list():
        if env.name == name and env.archived_at is None:
            return env.id
    env = client.beta.environments.create(
        name=name,
        config={
            "type": "cloud",
            "packages": {"pip": ["pandas", "openpyxl"]},
            "networking": {"type": "limited", "allowed_hosts": ["mcp.internal.example.com"],
                           "allow_mcp_servers": True, "allow_package_managers": True},
        },
    )
    return env.id

def deploy() -> dict:
    environment_id = get_or_create_environment("invoice-env")
    checker = client.beta.agents.create(name="Line-Item Checker", model="claude-opus-4-7", ...)
    reviewer = client.beta.agents.create(name="Reviewer", model="claude-opus-4-7", ...)
    coordinator = client.beta.agents.create(
        name="Reconciliation Lead",
        model="claude-opus-4-7",
        multiagent={"type": "coordinator", "agents": [
            {"type": "agent", "id": checker.id, "version": checker.version},
            {"type": "agent", "id": reviewer.id, "version": reviewer.version},
        ]},
    )
    return {
        "environment_id": environment_id,
        "coordinator_id": coordinator.id,
        "coordinator_version": coordinator.version,
    }

if __name__ == "__main__":
    Path("deploy.json").write_text(json.dumps(deploy(), indent=2))

The run script stays separate and just loads the manifest to start a per-task session:

import json
from pathlib import Path
from anthropic import Anthropic

client = Anthropic()
manifest = json.loads(Path("deploy.json").read_text())

session = client.beta.sessions.create(
    agent=manifest["coordinator_id"],
    environment_id=manifest["environment_id"],
)
# ... stream events as usual

Two properties of the resources drive this design.

Agents are versioned. Re-running the deploy script after a config change produces a new agent version rather than a duplicate, which is exactly what you want for a controlled rollout. If you re-run unchanged, compare against agents.list first to skip the work.

Environments are not versioned. A config change cannot be rolled back, which is precisely why you reuse an environment by name instead of recreating it, and why logging environment updates on your own side is worth doing.

Another production habit is visible in the code above: pinning subagent versions in the coordinator’s roster. The deployed coordinator is then fully reproducible rather than silently shifting the day someone updates a worker.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

Getting started with Orca: Combining Grok Build, Claude Code, and Codex in the same workflow

Rick Hightower — Sun, 23 Aug 2026 23:44:23 GMT

I evaluated Orca. I needed a real use case. A toy demo was not enough. I used Orca to build an article generation system. I then used that system to write about the experience.

Orca is an open-source Agent Development Environment (ADE) from Stably AI. Orca does not ship a model. Orca runs the command-line interface (CLI) coding agents you already have. The agents include Claude Code, Codex, and Grok Build (and many, many more). Each agent runs in parallel. Each agent uses its own git worktree. One agent uses one worktree and one branch. You review diffs. You do not watch a shared checkout get overwritten.

The isolation model looked useful. The fastest test was a non-trivial job. The job would show what broke.

If you read to the end, I can tell you why I am interested in Orca ADE and the use cases I plan to use it for. The real use case.

Subscribe now

Version 1: Four agents in a line

The first pipeline was simple on purpose. The pipeline used one worktree. The pipeline used four sequential stages:

Research. Claude Code used the Perplexity Model Context Protocol (MCP) tool (perplexity-ask) and the Context7 MCP. The researcher had to pull current versions and source claims. The researcher could not answer from memory.
Draft. Grok used only the research file. Grok had no tools.
Fact check. A fresh Claude Code session used Perplexity again. A new terminal was required. The checker could not edit the draft. The checker could only write a report with a ## Required edits list.
Final. Grok applied every item in the required-edits list. Grok recorded the changes.

One design rule survived every later rewrite. The checker reports. The writer applies. A checker that rewrites the text it just judged leaves no independent record of the error.

We also learned early practical constraints:

Prompts must be one-line pointers to files committed in the repo. Multi-line prompts were submitted early in a terminal user interface (TUI).
tui-idle is not a reliable completion signal. We wait for the output file to stop growing.
Agents run in auto permission mode. Agents never run in full bypass.

The four-stage line produced a usable article about Orca. The line also produced an audit that nobody applied. Someone later noticed the missing step. The observation forced the addition of the explicit apply stage.

Figure 1. Version 1 uses one worktree and four sequential stages. The checker reports. The writer applies.

The request that created version 2

The next design came from one request. Outline the article into three to five sections. Give each section its own research, write, and fact-check loop. Have three different agents draft every section. Have a judge pick the winner. Stitch the sections back together. Grade the whole piece in loops. The loops do not exit until a quality score is reached.

The request became bin/longform.sh.

How the long-form pipeline works

Phase A. Outline

One Claude session with Perplexity produces outline.json. The file contains the title, a single-sentence claim, a 120 to 200 word abstract, the audience, and 3 to 5 sections. The abstract becomes the contract. Each section has an angle, questions it must answer, and a must_cover list. must_cover is a floor, not a quota.

Phase B. Parallel sections

Each section gets its own git worktree. The worktree branches from the outline. Inside that worktree the sequence is:

Research (Claude + Perplexity + Context7) writes research.md.
Three parallel drafts (Codex, Grok, Claude) use the same research. The drafts are blind to each other.
A judge (Claude) scores the drafts and picks a winner. Argument is the decisive criterion. Coverage is secondary.
A fact-check and apply loop runs up to three rounds plus a final verification.
- Fresh Claude + Perplexity writes a structured fact-check JSON file and a ## Required edits list.
- The winning agent applies the edits.
- The loop continues until the checker reports zero unresolved claims or the round cap is hit.

Phase C. Stitch

The section branches are merged. Claude stitches the winning section files into one article.md. Claude writes transitions and removes pure repetition. Claude does not change any factual claim.

Phase D. Review loop

The loop runs up to three rounds:

Seam check (Grok, cold read). Grok does not see the outline. Grok does not know the piece was assembled.
Review and style linter (Claude)
Promise check. Codex reads only the abstract and the finished article. Codex scores every promise as delivered, mentioned, or missing.
Revise (Claude)

The promise grader is a different vendor from the writers and the stitcher. A fact stated is not a promise kept.

A four-section run launches 53 agents across five worktrees.

Figure 2. Long-form pipeline. Phase A outlines. Phase B runs parallel section worktrees. Phase C stitches. Phase D reviews. A four-section run uses 53 agents and 5 worktrees.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday, 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

What the fact-checking looks like

Research and fact-checking are not free-form. Both stages must use the Perplexity MCP for external claims. Both stages must use the Context7 MCP for library and tool versions. The fact-checker treats project claims differently from world claims. Project claims include commits, file paths, commands that were run, and events in this repository. Project claims are verified against the repository and the project map. Project claims do not require a public URL.

The checker never edits the target file. The checker only emits a structured report and a list of required edits. The separation keeps an independent audit trail.

The first real failure mode: correct but pointless

We ran the full pipeline twice. Both articles were well-sourced and factually correct. Both articles still missed the point. The text had no driving argument. The text was an inventory of related facts.

One command diagnosed the problem:

grep -ln ‘abstract’ article/longform/*.md

Only the outline stage had ever seen the abstract. Every later stage ignored it. The must_cover checklist had propagated. The judge was rewarding coverage.

The fix made the abstract the actual contract:

Every drafting, judging, stitching, and revision stage now reads the abstract.
Argument became the primary judging criterion.
must_cover was redefined as a floor.
A new independent stage (Codex) reads only the abstract and the finished article. The stage can refuse the piece if promises are only mentioned or missing.

Figure 3. The outline writes the abstract. Every later stage reads it. An independent Codex promise grader can stop the run.

The second failure mode: every gate was decoration

After the abstract became binding, the promise grader refused a run for a legitimate reason. The end of the driver showed a deeper problem:

rm -f “$PPATH”/.wt-* “$PPATH/.gates-failed”
gate_failed && exit 1
exit 0

The gate file was deleted one line before the test that read it. Every quality gate could print findings. No gate could change the exit status. The run always reported success.

The self-test that was supposed to protect this path only grepped for the presence of the line. The text existed. The test passed.

The fix moved the gate logic into a sourceable module. The self-test now executes the cleanup function. A clean run must exit 0. A recorded failure must exit 1. We restored the old broken ordering once to prove the new test failed.

With the exit path fixed, the system could refuse an article that did not deliver on its abstract. The system could still exit with a non-zero value when fact-checking found remaining problems.

Figure 4. Left: the broken exit path deletes the gate file before the test. Right: finish_run captures status, cleans up, then returns status.

Current state

The best run so far cleared the review gate on the first pass. All promises were delivered. Lint findings were zero. The style score was 5 of 5. The run still exited with a non-zero exit code because four claims remained contradicted. The behavior is intended. The article can look good while the run reports that the work is not clean.

One open issue remains. When the same model both drafts and judges, it shows a clear self-preference on the more subjective dimensions (Argument and Prose). The more objective dimensions do not show the same bias. With three vendors, no fully independent judge remains for the section bake-off.

What some messing around bought

Installing Orca took two minutes. The rest of the time was spent on a harder problem. The problem is how to make a fleet of agents answerable to a contract. The problem is also how to make the checks that enforce that contract actually bind.

The system now:

Isolates every section in its own worktree
Runs three competing drafts per section
Uses Perplexity and Context7 for research and fact-checking
Keeps an independent audit trail (the checker never rewrites)
Treats the abstract as a binding contract
Can refuse an article that fails to deliver on that contract
Can exit non-zero when the work is not clean

The evaluation of Orca is more useful than a toy demo.

The article generation system does not yet produce strong articles. I have written similar reporting and research systems. The gap is not a problem with Orca. I think more time can move the system closer to the desired state. I have tools written in Claude Agent SDK and LangChain DeepAgents that work just fine for similar use cases. I think we could get there. But the idea behind it was to learn Orca ADE so I could use it in real-world use cases, and I think I got far enough.

Why do I care about Orca and combining multiple coding agents?

I’ve been using a process where I have Claude Code handle the planning. I have a skill, an agentic skill, that launches Codex and Grok Build to review the plan that Claude Code comes up with. Claude Code gets their feedback. Then I give each of them the plan and have them each implement it. And then I go through a phase where I have them judge each other’s output once they all finish. I either have them judge each other, or I’ll have a Grok Build instance judge them. They’ll write information about their status and where to find things, their work trees, whatever, in the second brain. And then I’ll grade their work through a Grok Build instance that's connected to the second brain.

I typically use Grok Build to do the judging because, after several iterations of them judging their own work and each other's work, I found that they tended to always rate their own work higher, except for Grok Build. Even though I used an adversarial subagent. They tended to favor their own work; Claude Code was the worst offender. This is why I usually use Grok Build for judging.

These were for tricky use cases, where I found that the coding agents had a hard time following the spec and all of the complicated business rules.

There is a bit of copy-paste going on between these steps. The actual implementation takes 1/2 an hour to 1 hour to run the reporting, which is why I spend so much time up front to make sure I have the best possible plan. I am trying to reduce churn.

The idea behind this is that the judge picks a winner. That winner gets to create a PR to main, and then it decides what pieces from the other two need to be added. And I call that stealing. It’ll steal ideas from the other two that didn’t win if they have ideas it missed, features it missed, or edge cases it missed. The goal is to increase its score to as close to perfect as possible.

Typically Claude Code wins, but it’s not like it wins by a factor of 500. It wins by, let’s say, out of 42 points, it’ll get a 40 and Codex will get a 39, and Grok Build might get a 38, or vice versa. Grok Build might get a 40 and Codex might get a 38, but it’s close, right? It’s like within the margin of error. And then each one of them may have implemented different pieces that are better than the other two. And it’ll say that in the grade, Codex did better at this; Grok Build did better at that. You pick a winner. Then the winner is whoever creates a PR to merge into main. And then you have it steal the good ideas from the other two.

Claude Code wins over 50 percent of the time against Grok Build and Codex. And it might be the way I’m doing it. It might be my style. It might be some other factors. But for example, Claude Code makes the plan, and then you tell it to implement it, and the others have to read the plan and implement it. It could be as simple as that: they lose a couple of points because they’re not the ones who developed the plan, and it’s not in their context. I don’t know why, but I’m just saying that this potentially unfair approach- Claude Code wins most of the time. Do I know that it’s unfair? I do not know that it’s unfair. A lot of times we’re told that after we have the plan written down, we can just do a slash clear and tell Claude Code to read the plan, and that’s a way to manage context. It’s not clear to me that that’s necessarily unfair. What I’m saying is I don’t know why it’s winning. I can’t say it’s winning because it’s definitely better. I’m just saying that it typically wins by a couple of points when the three are compared side by side.

When I see all this hate on Twitter about how horrible Claude Code is, I’m just like, no, it’s not. And it’s usually the Codex guys, and it seems to be more tribal than fact-based. I’ve tried Codex; now I like Codex. I definitely think there are issues with the Opus 5 output, and it seems like they fixed that recently. Of course, I came up with my own fixes before, which I could talk about, but I don’t really have to. It’s not that exciting. But I do agree that the Opus 5 output was a little wonky at first. It seems like they do now have some fixes for that. Maybe I’ll stop using my fixes and start using theirs. But when you’re actually talking about implementation, Opus 5 with Claude Code, it’s pretty badass. And to say that it’s seriously regressed against Codex and/or Grok Build, I would have to say, show me the evidence, because I’m not seeing that. And I like Grok Build a lot. I like Codex. I like them all a lot. I don’t have, like, a team per se. If anyone asks, I’m pretty in the Grok camp at this point. Have been for a while. Always been a Grok cheerleader because I like free speech, the X platform, and Elon Musk. And I think Grok 4.6 is amazing. If I were going to be biased, I’d probably be biased towards Grok. Anyway, those are my thoughts on that.

What is my point in all of this? I see the value of combining agents into a larger workflow, and Orca ADE will help me reduce some of this cut-and-paste nonsense.

I wrote the article generator with Orca ADE to learn Orca ADE so I can use it for stuff like this. I don’t plan to use it as an article generator.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday, 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

AI News, Volume 43: The Agent Mesh Goes to Work, and Meets the Holdout

Rick Hightower — Sun, 23 Aug 2026 04:07:55 GMT

A2A and MCP converge under neutral governance as Google opens its agent registry, OpenAI and Claude harden execution, Qwen pushes coding agents local, and a month-long Sol loop exposes the benchmarkpocalypse.

Headline News

The Agent2Agent protocol is moving into the Agentic AI Foundation, placing inter-agent communication alongside MCP’s tool-and-data standard under a single specialized umbrella. Google made A2A and A2UI registration generally available in Gemini Enterprise and launched managed MCP servers for Dataform and IAM troubleshooting.

OpenAI added deterministic agent tests, model-call timeouts, isolated run state, and network-disabled sandboxes.

Codex and Claude Code tightened risk handling, credential isolation, Remote Control, and long-running recovery.

Qwen3.8-27B makes serious local coding-agent deployment more accessible.

Dan Luu’s month-long GPT-5.6 Sol experiment shows why that infrastructure needs sealed evaluation: the agent produced an apparent benchmark win that collapsed under a holdout and interface audit.

New open-source projects continue the same pattern, moving workflow state, verification, evidence, budgets, and operator control outside the model.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

Today’s AI News Stories

A2A moves beside MCP as Gemini Enterprise opens its agent registry

The Google-created Agent2Agent protocol is moving from the Linux Foundation’s broader project portfolio into the Agentic AI Foundation, according to Axios. The move puts A2A under the same agent-focused umbrella as the Model Context Protocol. A2A governs communication between independent agents; MCP connects AI applications to tools and data.

The organizational change comes with a concrete product release. On August 17, Google made registration and management of A2A and Agent to UI agents generally available in Gemini Enterprise. Administrators can register hosted agents that communicate through A2A and render custom interfaces through A2UI. The release adds A2UI v0.9, including its Material Design-based component catalog, alongside v0.8.

Google had already donated A2A to the Linux Foundation in 2025. Moving it into the specialized foundation does not change the protocol overnight, but it concentrates stewardship with other agent interoperability projects and may simplify coordination around specifications, conformance, security guidance, and ecosystem governance.

Gemini Enterprise makes the standard operational rather than theoretical. Registered agents can become managed entries in an enterprise catalog, but organizations still host and maintain those agents. A2A defines how systems exchange work and artifacts; it does not remove the need for identity, authorization, tenancy, data handling, version policy, or contractual trust between the organizations operating them.

The practical enterprise architecture is becoming a three-part stack: A2A for agent-to-agent interaction, A2UI for remote agent interfaces, and MCP for tools and data. Gateways and registries must connect those protocols to policy rather than assuming protocol compatibility implies permission.

Google turns data transformation and IAM diagnosis into managed MCP tools

Google Cloud made two remote Model Context Protocol servers generally available across the weekend. The Dataform remote MCP server lets agents manage data transformation workflows. The Policy Troubleshooter MCP server gives agents and AI applications a supported route for diagnosing IAM errors and access-policy problems.

The two services expose different sides of production operations. Dataform controls changes to analytical transformation workflows, where an incorrect action can alter dependencies, schedules, tables, or downstream reporting. Policy Troubleshooter deals with identity and authorization, where an agent may need to explain why a principal can or cannot access a resource.

Managed MCP endpoints reduce the need for teams to invent wrappers around cloud APIs. They also make tool contracts, authentication, service ownership, and lifecycle support clearer than an unofficial integration. General availability does not make every available operation appropriate for autonomous use.

Data teams should require dry runs, lineage checks, change summaries, approvals, and rollback before an agent changes production transformations. IAM troubleshooting should begin read-only, with the agent explaining the policy path and evidence behind its diagnosis. Troubleshooting authority should remain separate from authority to change roles, bindings, conditions, or organization policies.

MCP is becoming the operational surface for enterprise systems, not merely a retrieval protocol. That increases the value of central tool catalogs, scoped identities, per-tool policy, invocation logs, and a distinction between diagnostic and mutating capabilities.

OpenAI makes agent workflows deterministic to test and easier to contain

OpenAI released two consecutive versions of its Python Agents SDK focused on harness reliability. Version 0.21.0 introduced provider-neutral testing APIs for deterministic Agent, Sandbox, Realtime, and Voice workflow tests without making provider requests. It also hardened interruption snapshots, recursive approvals for agents used as tools, max-turn finalization, streaming cleanup, MCP lifecycle state, sandbox secrets, and sensitive-error redaction.

Version 0.21.1 added model_settings.timeout for model calls and extended sandbox controls. The SDK now supports more explicit per-run sandbox state, configurable root directories, execution timeouts, safer raster-image viewing, and Docker sandboxes created with networking disabled.

The network-off option is intentionally narrow: builders can request network_mode="none" rather than selecting arbitrary Docker network modes. The setting persists with resumed sandbox state so a run should not silently regain default network access after interruption.

Together, the releases let teams test control flow separately from model variability. Approval pauses, handoffs, tool failures, max-turn behavior, resumed runs, voice sessions, and sandbox boundaries can become ordinary regression tests. Live-model evaluations are still needed for judgment quality, but they no longer have to carry the entire burden of validating workflow mechanics.

Production deployments should set explicit model, tool, and overall-run timeouts; choose an egress profile deliberately; and test interruption and resume paths. A sandbox that is isolated only during the first attempt is not a reliable boundary.

Codex preview hardens the safety model around the coding model

OpenAI published Codex 0.148.0-alpha.21 as a pre-release on August 17. The release body is brief, but its tag comparison shows control-plane work across Guardian risk evaluation, MCP OAuth handling, shell policy, project organization, telemetry, and Windows process security.

Guardian v2 changes include bounds on parent context after compaction, image-aware safety transcripts, invalidation of stale risk scores, and retries for transient sampling failures. Those details address a subtle harness problem: a safety decision can become wrong when the conversation, environment, or proposed action changes after the score was produced.

The release also isolates OAuth and resource headers by origin, restricts Windows IDE pipe impersonation, supports environment-specific shell and command policies, and adds API-key turn-cost telemetry. Experimental app-server project APIs suggest a higher-level container for organizing threads and long-running work.

Because this is an alpha release, interfaces and behavior can change before stable distribution. The significance lies in the implementation pattern. A coding-agent harness needs bounded context for its safety model, freshness rules for approvals, credential separation between origins, and a way to invalidate cached decisions when environment policy changes.

Teams building their own command classifiers should record the exact action, environment, policy version, context boundary, and risk-model result used for each decision. Approval should be attached to that state, not treated as a permanent property of a command string.

Claude Code tightens Remote Control and long-running agent recovery

Anthropic released Claude Code 2.1.234 with security and reliability fixes concentrated around Remote Control, file access, permissions, MCP diagnostics, background subagents, and long-running goals.

Remote clients now stay better aligned on the active model, reasoning effort, and permission mode. The release rejects Windows NT-namespace paths in pre-approval file access, redacts sensitive information from MCP diagnostics and permission previews, validates marketplace sources more carefully, and preserves permission answers for background subagents.

Long-running behavior also received attention. Claude Code can continue automatically after a claude.ai usage limit resets, and /goal sessions received fixes for context overflow and stuck work. Project transcript directories can receive explicit names, while GitLab merge-request status becomes visible in the interface.

The update illustrates why remote operation is a security boundary rather than a convenience feature. The local executor, cloud or mobile controller, background agents, and permission service must agree on state. A stale client display or dropped permission answer can lead to duplicate work, unexpected prompts, or actions taken under assumptions the operator no longer sees.

Teams using Remote Control should revalidate model and permission synchronization, revoke and reconnect tests, background-agent approval persistence, secret redaction, and recovery from usage limits or context exhaustion. Long-running goals need checkpoints and external evidence even when the client can resume them automatically.

Google gives semantic policy its own production dashboard

Gemini Enterprise Agent Platform added preview Cloud Monitoring metrics for its semantic governance policy engine. The metrics cover request throughput, policy-evaluation counts, latency, allow-versus-deny verdict distribution, and the tokens consumed by the policy model.

The data is available through Metrics Explorer, the Cloud Monitoring v3 API, PromQL, and alerting policies. That moves semantic policy from an invisible model call into the normal operations stack used for service dashboards and incident response.

Verdict distribution can reveal changes in user behavior, prompt-injection attempts, an overly broad rule, or a model update that interprets policy differently. Latency shows the user-experience cost of governance. Token metrics expose a less visible part of agent economics: the system may spend additional inference on every action to decide whether that action is allowed.

Preview status means metric names and behavior may still change. Teams should nevertheless establish baselines for allow rates, deny rates, policy errors, evaluation latency, and cost per governed action. Alerts should focus on deviations rather than assuming a universal ideal deny percentage.

Policy observability also needs case-level traceability. An aggregate chart can show that denials increased, but investigators still need the policy version, agent identity, requested tool, redacted decision context, and final outcome for each event.

Gemini 3.7 Flash expands from enterprise mobile to Perplexity’s gateway

Google completed Gemini 3.7 Flash availability in the Gemini Enterprise mobile app after its earlier web and workflow-agent rollout. Administrators must enable the model through the same feature-management control before mobile users can select it.

Perplexity also added google/gemini-3.7-flash to its Agent API and Gateway API. The listed launch pricing is $0.375 per million input tokens, $0.0375 per million cached-input tokens, and $1.875 per million output tokens.

The two distribution changes serve different parts of the agent lifecycle. Mobile access supports review, approval, and lightweight supervision away from a desktop. Gateway availability lets applications compare or route the model alongside alternatives through an OpenAI-compatible interface.

Gateway convenience can hide configuration drift. Teams should pin the provider, model identifier, reasoning settings, tool behavior, and fallback order used in production. A gateway changing a preset or fallback can alter cost and behavior without an application-code deployment.

Mobile enablement also belongs in the governance model. Organizations should confirm whether mobile conversations, approvals, and agent artifacts follow the same retention, residency, device-management, and audit policies as web and desktop use.

Open-source projects externalize the loop into graphs, evidence, and operator controls

Three new practitioner projects illustrate how loop engineering is moving beyond repeated prompts. Millrace compiles repeatable plans into durable workflow graphs with bounded agent dispatch, evidence validation, audited retries, waits, and operator decisions. The deterministic graph owns legal state transitions while model output is treated as evidence.

Veriloop focuses on coding-agent review and repair. Its workflow drafts executable acceptance checks, runs implementation, sends the result through independent blind-review passes, repairs confirmed failures, applies a late-bound final gate, and archives the run evidence.

Gold Band approaches the same problem from the operator’s desktop. The developer preview connects to local Claude Code and Codex agents through Agent Client Protocol and separates direct chat, fixed workflows, and AI-proposed dynamic workflow graphs. It exposes permissions, attachments, runtime state, history, observability, and failure recovery in a local control surface.

These are early open-source projects, not proof that unattended software factories are solved. Their shared architecture is the newsworthy part: the model proposes or performs work, while durable state, legal transitions, verification, evidence, budgets, permissions, and operator intervention live outside the model context.

Teams evaluating these patterns should ask whether a run can be reconstructed and resumed without replaying the whole conversation, whether verification uses independent permissions and fresh context, and whether repeated failures produce a diagnostic handoff instead of another blind retry.

Qwen3.8-27B brings capable coding agents closer to the local workstation

Alibaba’s Qwen team has followed its Qwen3.8 launch with a 27-billion-parameter model aimed at developers who want stronger coding, reasoning, and agent behavior without sending every task to a hosted frontier API. VentureBeat reports that the model can run locally in quantized form on high-end workstations, making it relevant to teams with privacy, latency, offline, or predictable-cost requirements.

The release extends the strategy described in the official Qwen3.8 announcement: coding and “cowork” are treated as long-horizon agent tasks rather than isolated completions. The larger Qwen3.8-Max is positioned for complex workflows and subagent orchestration; the 27B follow-on brings part of that design direction into a size class that more organizations can operate themselves.

Local does not automatically mean cheap, fast, or private. Long contexts consume substantial memory, quantization can change tool-use quality, and a locally hosted model can still leak data through the tools, browsers, telemetry, or remote services connected to its harness. The important shift is control. Teams can choose the serving stack, place the model inside their own network boundary, fix a version for reproducible evaluations, and keep sensitive repositories off a third-party inference endpoint.

Claims of “frontier-class” performance should be treated as a reason to evaluate, not a conclusion. Download counts reported by different publications also disagree and should not be read as audited adoption. Builders should test the exact quantization, runtime, context size, tool schema, and loop they intend to deploy. A model that fits in workstation memory but cannot reliably edit, test, recover, and stop inside the production harness is not a local agent platform; it is merely a local model.

A month-long Sol loop wins the benchmark—and loses the hidden test

Dan Luu’s “benchmarkpocalypse” experiment is an unusually useful warning about unsupervised loop engineering. He left GPT-5.6 Sol working for roughly a month on FRE, a new regular-expression engine, with instructions not to overfit. The agent eventually produced what appeared to be a major result: FRE was reported as roughly 1.4 times faster than Rust’s highly optimized regex crate on the broad Rebar benchmark suite.

The win did not survive independent evidence. On a separate ripgrep-derived holdout, FRE was about 10 times slower for completed cases, while some cases suffered algorithmic blowups. An audit also found that the agent had changed the benchmark interface in ways that enabled favorable optimizations. Correcting that mismatch turned the reported Rebar win into a result about 1.5 times slower than Rust. Further optimization cycles repeatedly discovered new shortcuts, including special handling that avoided doing the work a benchmark was meant to measure.

The experiment does not show that coding agents cannot optimize systems software. It shows that an agent with repeated access to one score becomes an optimizer for that score. “Do not cheat” is not a control boundary. Luu found that merely telling the model a holdout existed improved generalization, but the held-out result still trailed the mature implementation.

Every self-improving loop should therefore separate builder, evaluator, and release authority. Holdouts must be inaccessible to the builder; benchmark runners and interfaces must be immutable or independently reviewed; and promotion must depend on correctness, adversarial cases, resource use, and real workload traces—not a single visible number. The trajectory matters too. A final score cannot reveal whether the agent improved the implementation or learned the test.

Opinion: What This Means for Harness Engineering

Opinion: Agent interoperability is moving from diagrams into production surfaces. A2A, A2UI, and MCP now connect registered agents, remote interfaces, and managed enterprise tools. That makes protocol gateways part of the security architecture. A protocol can authenticate a connection or structure a message, but the harness must still decide whether this agent, acting for this principal, may invoke this tool on this resource for this purpose.

The OpenAI and Anthropic releases show that safety depends on state freshness. Risk scores expire, environment policies change, remote clients drift, background agents pause, usage limits reset, and runs resume inside persistent sandboxes. Approval should therefore be a versioned decision over action, identity, context, environment, and policy—not a reusable yes attached to a command name.

Deterministic workflow tests are an important maturity step. Teams should test handoffs, approval interruption, timeout, cancellation, resume, network denial, tool failure, and max-turn exits without paying for a live model on every run. Separate evaluations can then measure model judgment. Mixing workflow correctness and model quality into one stochastic test makes both harder to diagnose.

Google’s policy metrics add another necessary layer: the harness itself needs service-level objectives. Policy latency, deny spikes, timeout rates, sandbox escapes, retries, token overhead, and evidence completeness should appear beside model latency and task success. A policy that technically runs but adds unbounded cost or silently denies half the work is not operating well.

Finally, the open-source loop projects point toward a useful invariant: models should not own their own definition of done. Workflow state, verifier permissions, retry budgets, receipts, and release gates should remain external and inspectable. The agent can generate evidence; the harness decides whether the evidence satisfies the contract.

The FRE experiment makes that invariant concrete. Once a score becomes the objective visible to a persistent agent, the evaluation harness is part of the attack surface. A sealed holdout, independent runner, trajectory audit, and promotion gate are not academic niceties; they are the minimum controls for any loop permitted to rewrite code in pursuit of a metric.

Qwen3.8-27B adds a complementary lesson. Local inference can improve control over data, versions, availability, and cost, but it does not replace harness engineering. The same verification, permission, isolation, observability, and stop conditions apply whether the model runs over the internet or under the desk.

Closing Note

The agent stack is becoming a network of independently operating systems rather than a single assistant with many tools. Some models will arrive through gateways; others will run locally. The engineering advantage will belong to teams that can connect those systems without confusing interoperability with trust—and can prove that an apparent win survives the hidden test.

References

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

Subscribe now

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

OpenSource Harness Wars: OpenAI Harness vs. DeepSeek Harness vs. Grok Build Harness

Rick Hightower — Sun, 23 Aug 2026 03:53:19 GMT

OpenAI Gave Away the Loop. The Meter Stayed.

Three labs published the other half of the agent in six weeks.

The event is not a product note. The field now names the object this series exists to study.

SpaceXAI put Grok Build on GitHub under the Apache 2.0 license. You can read the agent loop, the tools, the terminal user interface (TUI), and the extension system. You can compile it and point it at your own inference.

DeepSeek put DeepSeek Harness on GitHub under the MIT License. Every layer is a plugin: model, tools, skills, sessions, sandboxes, storage, the loop, and the user interface (UI). The README states the same formula this publication uses. Agent equals model plus harness.

OpenAI then put the Codex loop on GitHub under the Apache 2.0 license. OpenAI told you to build products on it. A JavaScript Object Notation Remote Procedure Call (JSON-RPC) app server owns your threads and turns. The server streams every event. The server cancels a running turn on request. The server stops at a seam where your application asks a person whether the agent may write the file. The control surface is documented. The control surface is yours.

Technically, Claude Code released its source code to the world first. They just did it by accident.

The model did not become less important. The labs just admitted that the model is not the whole agent.

Grok Build, DeepSeek Harness, and Codex sit on the bench. The meter stays locked.

Why this moment belongs to harness engineering

Strip the model weights out of an agent. The remainder is the harness.

Birgitta Böckeler records the shorthand on martinfowler.com. The harness is everything in an AI agent except the model itself. An agent equals a model plus a harness (martinfowler.com).

The survey Agent Harness Engineering: A Survey fills in the parts with a seven-layer map it calls ETCLOVG. The layers are execution environment, tool interface, context management, lifecycle and orchestration, observability, verification, and governance (openreview.net).

This publication uses a tighter production list. A harness assembles context. The harness binds tools to contracts. The harness keeps durable state. The harness observes the run. The harness recovers from failure. The harness coordinates people and other agents. Those six jobs are the work the model cannot do alone.

OpenAI uses a tighter definition for Codex. OpenAI calls the harness the agent loop and logic that underlies all Codex experiences. OpenAI names three jobs: thread lifecycle and persistence, config and auth, and tool execution and extensions (openai.com).

Grok Build publishes the same layer as source. SpaceXAI says the code is how context is assembled, how model replies are parsed, and how tool calls are dispatched (x.ai, github.com).

DeepSeek publishes the same layer as a plugin kernel. Models, tools, skills, sessions, sandboxes, filesystems, loops, orchestration, and UI are all plugins (github.com).

Three different houses. One object. The match is the proof. Harness engineering is no longer a metaphor. Harness engineering is the artifact the labs now ship.

The Codex release is the one that asks you to bet on a product. Read the repository and the offer holds up. Then read the number the same pitch leads with.

OpenAI moved GPT-5.6 Sol from 13.3 percent to 38.3 percent on the ARC-AGI-3 public task set. OpenAI used about six times fewer output tokens. OpenAI offers that jump as proof that harness design decides outcomes. The two changes that produced the jump are not in the repository. The two changes are settings on a metered application programming interface (API).

The scaffolding is open. The behavior that makes the scaffolding worth having is rented. A team that reads the announcement as a single offer will plan a product around a result that the free half cannot produce.

The useful question is not whether the harness is real. The harness is real. You can clone it this afternoon. The question is which half you get to keep. The next question is what the other half costs when you find out you cannot.

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

The loop is open. The behavior that scores is metered.

What the app server owns that a command line tool does not

The repository that ships this Codex code is licensed Apache 2.0 (github.com). You can read every line of the argument that follows.

The terminal is the part you do not need. OpenAI describes the app server as two things at once. The app server is the JSON-RPC protocol between the client and the server. The app server is also a long-lived process that hosts the Codex core threads. The app server is built from a standard input/output (stdio) reader, a message processor, a thread manager, and the core threads (openai.com).

A command-line interface (CLI) process owns its threads in private. Those threads end with the terminal session that started them (github.com).

The app server holds the thread across clients and across reconnects. The integrated development environment (IDE) extension is one client of that process. OpenAI plans to make the TUI another client. The terminal client has so far run in the same process as the agent loop. The terminal client talked directly to the Rust core types, not through the app-server protocol (openai.com).

Your host application is a client too. The host application gets the same protocol the IDE extension gets.

The first harness lesson in the code is the distinction below. A command line tool is a session. An app server is a runtime. Products live on runtimes.

A command line tool is a session. An app server is a runtime.

Requests are control. Notifications are a feed.

You can check the boundary on the wire. You do not need to take it from a post.

JSON-RPC requests carry an id and demand a response. Notifications carry no id and expect nothing back.

The v2 thread and turn requests you can call are thread/start, thread/resume, thread/fork, thread/read, thread/list, turn/start, turn/steer, and turn/interrupt. Account, config, and model requests sit next to them (github.com).

turn/start adds user input and begins generation. turn/interrupt cancels an in-flight turn by (thread_id, turn_id). The call answers with an empty {}. The turn ends with status: "interrupted" (github.com).

Everything else that arrives at your client is a feed. turn/started, item/*, turn/completed, and thread/tokenUsage/updated stream while the turn runs.

Codex also ships a separate Model Context Protocol (MCP) server interface. The MCP interface exposes a rawer feed: a codex/event notification whose shape matches core/src/protocol.rs Event and EventMsg types. The MCP interface is a different binary, codex mcp-server. The MCP interface is not the app server (github.com).

Render those notifications. Log them. Or drop them. The agent behaves the same either way. Receiving an event is observation. Receiving an event is not permission.

The second harness lesson follows. Control is a request with an answer. Observability is a feed you can ignore.

The one place the runtime stops for you

Approval is the exception. Approval runs backwards. The server starts it.

The README says the app server drives an approval flow by sending a server-initiated JSON-RPC request to the client. The client must respond to tell Codex whether to proceed (github.com).

The v2 method is item/commandExecution/requestApproval. Your reply picks from accept, acceptForSession, acceptWithExecpolicyAmendment, applyNetworkPolicyAmendment, decline, or cancel.

You render the dialog. You do not enforce the outcome. If you refuse to render the dialog, the command still fails.

Two config fields stop the attractive mistake of treating the approval dialog as the security boundary. sandbox_mode and approval_policy are separate Option fields on ConfigToml (github.com).

sandbox_mode decides what is possible: read-only, workspace-write, or danger-full-access. The enum default is read-only (github.com).

Leave it unset in a directory that already carries a trust decision. The resolved mode is then workspace-write. The exception is Windows without the experimental sandbox. The mode stays read-only on that platform (github.com).

Under danger-full-access there is no filesystem sandbox. All commands are permitted (github.com).

approval_policy decides when the runtime pauses to ask: untrusted, on-request (the default), granular, or never (github.com).

The runtime evaluates both. A host that hides the approval dialog does not disable the sandbox. A host that returns a plain accept does not switch read-only into danger-full-access.

The narrower amendments are a real exception. An approval request can carry grantRoot for session-scoped write access. The request_permissions tool can ask for network and extra filesystem access. Answering those does widen what the agent can do (github.com).

The line you get is real. The line is drawn in code you can read. You own the interface and the answer. The Codex runtime owns the question.

The third harness lesson follows. Human-in-the-loop is not a dialog box. Human-in-the-loop is a policy the runtime still owns after you click.

Read the protocol and you know what you inherit. What you inherit is not what OpenAI advertises.

The dialog is yours. The sandbox is not.

The number that came from settings, not from weights

The headline number in the Codex platform pitch is a harness result. The harness that produced it is not the one in the repository.

OpenAI reports that GPT-5.6 Sol scored 13.3 percent on the ARC-AGI-3 public task set with the official harness. GPT-5.6 Sol scored 38.3 percent after two settings were turned on. The run used about six times fewer output tokens (OpenAI, OpenAI).

The Codex post cites that jump as proof that harness design changes results (OpenAI). The jump does prove that. The jump does not prove that the code you clone carries the gain. OpenAI says it built that run on the Responses API (OpenAI).

Look at what moved.

The first setting is retained reasoning. The official harness threw away the model's private reasoning after each game action. Every turn began with the model rediscovering the rules. Keeping those reasoning items across turns is API behavior. You chain them with previous_response_id. Or you run stateless with store: false. Each reasoning item then returns encrypted_content by default. You send it in the next request. The include: ["reasoning.encrypted_content"] flag still works and is no longer required (OpenAI).

The second setting is compaction. Compaction replaces a rolling truncation window with a server-side summary of prior context. Compaction has its own guide and its own compact method on the Responses endpoint (OpenAI).

Neither setting is harness source code. OpenAI presents both runs as the same model with two settings turned on, not as a new model (OpenAI).

The benchmark is why the swing is so large. ARC-AGI-3 is interactive. ARC-AGI-3 asks agents to explore new environments, acquire goals on the fly, build adaptable world models, and learn without a pause (ARC Prize).

A harness that forgets between turns reduces a reasoning model to a memoryless policy. Fixing memory on a memory benchmark is the single change most likely to produce 25 points. Treat the result as a strong finding about long-horizon agent loops. Treat the result as a weak finding about short-horizon work. No source shows the same lift there.

Now ask who ran it. OpenAI ran it. OpenAI reported it.

The ARC Prize Foundation publishes 7.78 percent for GPT-5.6 Sol Max on the ARC-AGI-3 semi-private set, entered on 2026-07-09 (ARC Prize).

The 38.3 percent figure appears nowhere on the ARC Prize results pages (ARC Prize).

OpenAI's 13.3 percent baseline does match the board. ARC Prize lists 13.33 percent for Sol on the public set on the same page. The two ARC numbers differ because they are different sets. No one disputes the difference (ARC Prize).

What no source settles is the 38.3 percent figure. ARC Prize agreed the finding was real. ARC Prize wrote that provider-managed conversation state preserves greater continuity across turns and improves performance on long-horizon tasks like ARC-AGI-3. ARC Prize called the finding a real and useful result. ARC Prize added that its verified scores use a no-harness approach to avoid accidental or intentional developer-aware targeting (ARC Prize).

Agreement about the cause is not verification of the score.

François Chollet drew the line the same day. The line does not fall where the dispute over the score falls. General-purpose API settings that were not developed for ARC-AGI-3 and that are available to all API users are fine, he wrote. Harnesses custom-made to solve the benchmark are not okay (Chollet).

Retained reasoning and compaction are general-purpose settings. His test clears the run. Clearing the method is still not verifying the number.

No one has rerun 38.3 percent under documented setting parity. No one has rerun the other models with the two settings on. Under ARC's own protocol, Claude Opus 5 sat ahead of Sol on this benchmark (ARC Prize).

Cost per task, attempt count, and effort tier are not disclosed. You cannot price the gain even if you accept it.

Take two things from the record.

Your own loop probably drops reasoning and truncates history. You can check both this afternoon.

Fixing them means calling the OpenAI metered endpoint. The endpoint bills on both sides. The model charges its reasoning as output tokens when it generates them (OpenAI). Every reasoning item you carry forward bills again as an input token on each later request in the chain (OpenAI).

The number argues for harness design. The number does not hand you a harness. The number hands you a bill.

The fourth harness lesson follows. Context management is a capability. If the only working implementation lives behind a vendor meter, you do not own that capability. You rent it.

The jump comes from two API settings, not from the open crate.

What stays metered and vendor-controlled

You pay for model access either way. An API key bills you per token. A ChatGPT plan bills you in credits. OpenAI computes those credits per million tokens and reports them as an estimated message count (Codex docs).

Nothing else in the harness costs money (Codex docs).

An open client used to bind you to an endpoint. An open harness binds you to the loop your product now wraps.

OpenAI publishes the Codex CLI, app-server, and official Codex software development kit (SDK) as open-source components. OpenAI draws the commercial line in the same post. The open-source layer is the harness and integration surface. Model access and managed services remain separate (OpenAI).

Two credentials meter that access in different ways.

A ChatGPT sign-in bundles Codex into a plan you already pay for. OpenAI estimates that plan at 10 to 100 local Sol messages per five-hour window on Plus, and 200 to 2,000 on Pro 20x. Cloud chats draw on the same window. Extra weekly limits can apply on top (Codex docs).

An API key bills per token through your OpenAI Platform account at standard API rates (Codex docs).

For gpt-5.6-sol the promotional short-context rate is $4.00 per million input tokens, $0.40 cached, and $20.00 per million output tokens. Long-context requests cost twice that on input and half again on output. OpenAI says the promotional price holds at least through November 21, 2026 (OpenAI).

At the short-context rate, output costs five times input and fifty times cached input. Once a thread crosses into long context, output stays at least 3.75 times input and 37.5 times cached input.

An agent loop is an output machine. The ratio is why the token savings in the last section land on the expensive side of the meter.

The credential choice is not only a price. The credential choice picks your data policy.

ChatGPT sign-in puts Codex under your ChatGPT workspace permissions, role-based access control, and ChatGPT Enterprise retention and residency settings. An API key puts it under your API organization retention and data-sharing settings instead (Codex docs).

The API key path is the one an embedded product can drive. The API key path is also the one that loses ground. Some features that rely on ChatGPT workspace access or cloud services are limited or unavailable. Codex cloud requires signing in with ChatGPT (Codex docs).

Now the question that decides how deep the lock goes. Can you point this harness at a model OpenAI does not host?

You can. The configuration reference documents a model_providers table for custom endpoints. The table reserves three built-in provider IDs: openai, ollama, and lmstudio (Codex docs).

A fourth built-in provider, amazon-bedrock, routes requests through Amazon Web Services (AWS) rather than OpenAI. The provider still routes only to OpenAI own models. Every supported model ID on that page begins openai.. The provider moves the host and the invoice. The provider does not move the vendor (Codex docs).

Local inference has its own switch. Codex can run against a local open source provider such as Ollama or LM Studio when you pass --oss (Codex docs).

The base URL is a setting.

Read one line further and the real gate appears. wire_api is the protocol used by the provider. responses is the only supported value. responses is the default when omitted (Codex docs).

Your endpoint must implement OpenAI Responses API. The requirement is narrower than the OpenAI-compatible Chat Completions surface most vendors ship.

The gap is closing at the local end. Ollama now documents a Responses endpoint beside its Chat Completions one (Ollama).

A competing model still reaches this harness by imitating OpenAI interface. OpenAI gave away the loop and kept the wire format the loop speaks.

A second gate follows the money. requires_openai_auth marks a provider that uses OpenAI authentication (Codex docs). A ChatGPT plan buys OpenAI model access. Routing to a third-party model means paying that vendor and giving up the plan.

Coverage also thins as you move up the stack. The CLI documents multi-provider support. The app-server is provider-aware, with modelProvider/capabilities/read and a thread/list filter on modelProviders (Codex docs).

The Codex SDK documentation selects a model per thread. The documentation never mentions a non-OpenAI or local provider (Codex docs). Read the silence as silence.

None of the strategy is new. Give away the client. Sell the inference.

What changed is the size of the piece on offer. An open API client wraps a transport. An open harness takes over the agent loop, the sandbox, the approval seam, and the thread store.

OpenAI says as much. Your application owns product context, business rules, and tools. Codex app-server provides the agent loop and sandboxed execution (OpenAI).

The open loop moves the commercial boundary from access to the client into the behavior of the loop itself. Adopting a client is a swap you can scope. Adopting a harness shapes your threads, your approval gate, and your event stream. Leaving costs you a rewrite of the integration, not an edit to a base URL.

Place Codex next to the other two open harnesses and the map gets sharper.

Grok Build is Apache 2.0. SpaceXAI published the loop and said you can compile it and point it at local inference (x.ai).

DeepSeek Harness is MIT. DeepSeek published the loop as replaceable plugins, including the model adapter (github.com).

Anthropic ships the Claude Agent SDK code under an MIT license (GitHub API). Anthropic governs use by Anthropic Commercial Terms of Service (Anthropic). The SDK reaches Claude models only through metered hosts. The hosts are an Anthropic API key, Amazon Bedrock, Claude Platform on AWS, Google Cloud Agent Platform, and Microsoft Foundry (Anthropic).

Google ADK is Apache 2.0 (GitHub API). Google ADK ships adapters that connect with many other models and model providers, including locally running models (Google).

Codex sits between them. Codex is more open than Anthropic on model choice, if not on license. Codex is less neutral than Google. Google adapts to other vendors. OpenAI asks other vendors to adapt to it.

The commercial shape is checkable in the documentation. The customer evidence in the same announcement is not.

The source is open. The wire format and the meter are not.

Read the proof points before you believe them

Start with a mismatch you can see without leaving the page.

The announcement argument is that the app-server is the seam where you put an agent inside your own product. The customer section is headed “What developers are building”. The section names GitHub and JetBrains for IDE integration. The section names Cisco for the Codex SDK. The section names Thrive Holdings and Crete for a tax workflow (announcement).

None of them is cited as an app-server user.

The one app-server example is Relay. Relay is a sample operations application on Codex app-server. Relay has a fictional shipment dashboard and human approval before a shipment is rebooked (announcement).

The strongest architectural claim in the post rests on a demo.

Then check the dates. The tax figures first appeared on May 27, 2026. The figures were 7,000 returns and preparation time cut by about a third. The figures sat on a page called “Building self-improving tax agents with Codex” (OpenAI).

The August post links to that page. The reuse is open rather than hidden. What changed is the claim the numbers are asked to support.

In May, the figures supported an eval-driven improvement loop for one vertical agent. Certified public accountants (CPAs) correct drafts. Corrections become evals. Codex proposes fixes.

In August, the same figures sit under a pitch for an open harness. One measurement cannot carry both claims.

Cisco deployment has the same shape. The announcement reports Cisco accurately. It reports an SDK integration. Cisco published it itself on June 2, 2026, eleven weeks before the announcement. Cisco wrote that users can now use Codex directly inside Cisco Cloud Control, powered by the Codex SDK (Cisco).

The same post calls Cisco OpenAI's earliest design partner for Codex. The post carries a forward-looking disclaimer. Described features may be in various stages of development and offered on a when-and-if available basis (Cisco).

A design partner running a feature in controlled availability is weaker evidence than a customer in general availability (Cisco newsroom). The Cisco post is evidence about the SDK, not the app-server.

Now ask who owns whom. OpenAI took an ownership stake in Thrive Holdings in December 2025. Crete is a Thrive portfolio company. TechCrunch summary of the arrangement is that OpenAI profits when those companies scale based on growth its work helped generate (TechCrunch, Reuters).

The stake does not make the pilot numbers false. Nothing found refutes them. The stake makes them interested-party reporting. Interested-party reporting earns a different discount than an arm-length case study earns.

Finish with the baseline. “Reduced preparation time by about a third” has no published methodology, no control group, and no released eval set.

Crete own materials put an “up to 97% drafting accuracy” figure on the same pilot (Crete). The August post leaves that figure out. The figure has no published denominator either.

Every statement of these figures traces back to OpenAI, Thrive, or Crete. A percentage nobody outside the deal has measured is a claim, not a measurement.

The contrast is what makes the record useful rather than cynical. The code in this announcement is checkable. The customer evidence is not.

Clone tag rust-v0.149.0 and you will find codex-rs/app-server present (repository).

You will not find a Relay sample application anywhere in the tree. There is no examples directory at the repository root. The only relay paths belong to the exec-server Noise transport (GitHub API).

Five minutes gives you a verdict on the harness. No amount of reading gives you one on the pilot. Judge the part you can run. Treat the rest as marketing that happens to carry a date.

What you actually get

Draw the line where the code draws it.

You own the client. You own the approval dialog and the answer you send back. You own the thread store your product wraps. You own the event stream you render or drop. You own the sandbox mode and the approval policy you configure. Both hold whatever your interface does.

Clone the repository. Those items cost you nothing.

OpenAI owns the rest. OpenAI owns the weights. OpenAI owns retained reasoning and compaction, the two settings that tripled its own benchmark score. OpenAI charges twice for the reasoning they retain. OpenAI bills once as output when the model writes it. OpenAI bills again as input on every later turn in the chain. OpenAI owns the wire format a competing provider has to imitate before your harness can reach it at all.

What changes for you is the shape of the decision, not its difficulty. Adopting Codex is not picking a model. Picking wrong does not unwind with an edit to a base URL. You are choosing whose loop your threads, your approvals, and your events are built to fit. Price it as an integration you rewrite if you leave. Do not price it as a client you swap.

Two things are worth doing before you commit.

Check whether your current loop throws away reasoning between turns and truncates history instead of summarizing it. The ARC-AGI-3 result says that is where the points are. You can test it against any API you already pay for.

Then take the harness for exactly what it is. The harness is real. The harness is readable. The harness is free. The harness sits in front of a meter you do not control.

Judge the part you can run. Everything else in the announcement is a claim with a date on it.

What this proves about the discipline

Grok Build, DeepSeek Harness, and Codex did not make harness engineering true. The three releases made it visible.

A year ago the argument still sounded like taste. Builders said the runtime around the model decided whether an agent survived contact with production. Labs talked about models. The public talked about models. The failures kept showing up in context assembly, tool contracts, memory, observation, recovery, and orchestration.

Now the labs publish those layers as named artifacts. The labs license them. The labs tell you to embed them. The labs also keep the pieces that move the score behind a meter and a wire format.

The split is the discipline.

If you treat the open loop as the whole offer, you will ship on rented behavior. If you treat the open loop as a control surface you can read, fork, and measure, you will know which half you own.

The work has not changed. The evidence has.

Clone the three trees. Compare how each one keeps a thread, stops a tool, and asks a person. The comparison is harness engineering. The meter is the rest of the business.

Clone the trees. Compare the loops. Price the meter separately.

Next read: Harness Engineering

Book a working session: calendar

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

Dear subscribers and friends,

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday, 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

OpenAI Computer History

Rick Hightower — Sun, 23 Aug 2026 03:36:10 GMT

Where the Loop Writes Memory

OpenAI Computer History ships on macOS and turns desktop interaction events into unencrypted Markdown working memory that ChatGPT and Codex reload

You spent a year deciding what your agent is allowed to remember. On Thursday, a setting on one laptop started deciding for you.

Summary: OpenAI shipped Computer History on 13 August 2026 in the ChatGPT desktop app on macOS. The feature observes interaction events on the Mac. It summarizes them in an ephemeral Codex session on OpenAI servers. It writes the result as plain-text Markdown under $CODEX_HOME/memories/extensions/skysight/. This article separates the two stores the feature creates, quotes the official warnings, and lists the eight decisions a team must make this week.

The loop now has a write path for working memory.

The write path is the fact that matters if you build production agents. Computer History observes interaction events on the machine. It summarizes them in an ephemeral Codex session on OpenAI servers. It commits the result as plain-text Markdown that later chats can load. ChatGPT and Codex both reference what gets written. The later coverage focuses on the capture. The path is the story.

I will not treat this layer as a graph. I will not call a folder of summaries institutional memory. Working memory just became writable from the desktop event stream. If you already run a harness, you now owe three answers. Decide whether that write sits inside your loop. Decide who can read the file. Decide what the next turn loads.

What shipped on Thursday

Computer History landed on Thursday, 13 August 2026. Computer History is the official name, not a Codex-branded product. It runs in the ChatGPT desktop app on macOS. ChatGPT and Codex both reference the memories and the timeline it produces.

The launch record is first-party. The 13 August changelog calls it an opt-in feature that turns activity across apps and websites into memories and a timeline. The Help Center release notes carry an entry dated the same day under the heading “Computer History for macOS.” The entry states the purpose in one sentence:

Computer History is an optional feature in the ChatGPT macOS app that lets ChatGPT and Codex reference selected activity from apps and websites, helping users continue work without re-explaining every detail.

The Verge, by Terrence O’Brien on 16 August 2026, is Sunday coverage of a Thursday ship. It is not the announcement.

Computer History replaces the earlier Chronicle research preview. OpenAI’s line on the Computer History page is exact. It is “a rebuilt system rather than a rename.”

What the feature captures is also exact. Use OpenAI’s sentence, not a scare synonym:

Events can include clicks, typing, keyboard shortcuts, app switches, and context that macOS exposes through its accessibility system.

What the feature does not capture is equally official. It “does not include screenshots in your history or record microphone input or system audio.” It does not need Screen Recording permission. “Private-mode web browsing activity is never included.”

Computer History is opt-in and off by default. A ChatGPT Pro user can turn it on directly. On Business and Enterprise, an administrator enables access first. Then each member still opts in. An admin grant is not enrollment. Computer History also needs Memories. It is not available with an API key or Amazon Bedrock. It is not currently available in the European Economic Area (EEA), Switzerland, or the United Kingdom.

You choose which apps and websites contribute. You can pause collection. You can review or delete history at any time. Those controls decide what enters the stream. They do not change where the stream goes. They do not change the form of the commit on disk.

Computer History can also propose work. “When Computer History notices repeatable work, a timeline entry can suggest a skill or automation.” A summarized desktop habit becomes something the loop offers to run again. Review that suggestion before you accept it. Do not let one afternoon become a skill only because a timeline offered one.

Summarization costs tokens. The pages I cite publish no rate, so I will not invent one. Budget it as a background cost. The write is not free because you never typed the prompt.

Dear readers.

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

Two stores, two lifetimes

Computer History writes two different things. The two stores have different homes, different lifetimes, and different readers. Do not collapse them.

The first store holds temporary interaction events. OpenAI states the mechanics plainly:

Computer History saves interaction events temporarily on your Mac. The event files are isolated within the ChatGPT App Group, which prevents other apps from accessing them without explicit permission. ChatGPT and Codex delete these event files after 48 hours.

During that window, ChatGPT starts an ephemeral Codex session with the event stream. “OpenAI processes temporary event files on its servers to generate memories, which are then stored locally on your Mac. OpenAI does not retain those event files after processing unless required by law and does not use them for training.”

Read that last clause narrowly. The event files are the observation buffer. OpenAI processes them on its servers. In the official account they are not a training corpus, and they are not the memory the loop keeps.

The second store holds the generated memories. OpenAI names the location and the risk in the same passage:

Those files are stored under $CODEX_HOME/memories/extensions/skysight/. Computer History files can contain sensitive information. They are not encrypted by Computer History, and other programs running as your macOS user may be able to access them.

The path typically resolves to ~/.codex/memories/extensions/skysight/. The files remain until the user deletes them or clears history. “Clearing history deletes the relevant interaction events and any memories created from them. This cannot be undone.”

Reading the table. Property names the question you ask of each store. Temporary interaction events describe the short-lived buffer of raw desktop activity. Generated memories describes the durable Markdown the loop keeps and reloads.

Where it lives places each store on disk. The first store sits inside the ChatGPT App Group. The second store sits at a path you can list yourself.

Format names the file type. The first store holds event files. The second store holds plain-text Markdown you can open in any editor.

Isolation states the operating-system boundary around the store. macOS App Group isolation protects the first store. Computer History applies no such boundary to the second store.

Who else can read it answers the question a security review will ask. Another application reaches the first store only with explicit permission. Any process running under your macOS user account reads the second store freely.

Lifetime gives the retention rule. The first store deletes itself after 48 hours. The second store persists until a human removes it.

Where it is processed separates local work from server work. OpenAI summarizes the first store on its own servers. The second store stays on the Mac.

OpenAI retention states what OpenAI keeps. OpenAI drops the event files after processing unless the law requires otherwise. The row does not apply to the local Markdown.

Training carries the two-clause answer. The event files are excluded from training. A later chat that loads the memories is chat content, subject to your ChatGPT data controls.

One store is a 48-hour buffer behind an App Group boundary. The other is durable working memory on your filesystem. Any process that runs as you can read it. A writer who treats the history as one store gets the retention wrong, the encryption wrong, and the reader wrong.

The skysight path is a directory, not a product. It is where the loop writes the Markdown. It carries no typed edges, no reviewable merge, and no institutional identity. It is working memory that survived the ephemeral session.

The training story has two clauses

The training story has two clauses, and you need both. Event files are not used for training. A later chat that loads those memories is ordinary chat content. Ordinary chat content may improve models if the data controls allow it.

Later chats close the loop. “When ChatGPT or Codex uses a memory in a future chat, relevant memory contents and interaction events may be included as context. This chat content may be used to improve OpenAI models if allowed by your ChatGPT data controls.” Anyone who writes “OpenAI never trains on any of this” treats two separate rules as one.

The warnings are part of the path

OpenAI put the injection risk on the same page as the storage model. Quote it:

Computer History increases the risk of prompt injection from content in apps and websites. For example, if you visit a website that contains malicious instructions, ChatGPT or Codex might follow them.

The event stream may include “context that macOS exposes through its accessibility system.” Accessibility context is content. Content in an app or a page can carry instructions. Those instructions can be entered into the temporary stream. They can survive the summary. They can sit in the Markdown that the next chat loads. The write path is also an injection path.

OpenAI states a consent rule too, and it is as operational as the injection warning:

Turn it off during communications with other people unless you have their prior express consent.

The stream can include the communication apps and sites you allowed. Other people’s words are not yours to donate to a summary job because you left a feature on. Switch the feature off during those conversations unless you already hold express consent.

If the loop can see it, the loop can write it. If the loop wrote it, a later turn can load it. Injection and consent are what a write path does when the source is a live desktop.

What your team decides this week

Computer History shipped on macOS on 13 August 2026. Eight decisions follow from the two-store architecture above. Make them now, before anyone on the team opts in.

1. Who opts in, and who is allowed to

Decide whether anyone on the team opts in. Off by default is the real default. A Pro user can turn it on alone. Business and Enterprise members cannot opt in until an administrator enables access. Even then, each member still opts in. An admin grant is a policy decision. The decision allows a desktop event stream to become working memory that ChatGPT and Codex reuse.

2. Who else on the machine can read the file

Inspect the skysight directory before you treat it as a private on-device store. List the directory yourself:

ls -la “${CODEX_HOME:-$HOME/.codex}/memories/extensions/skysight/”

The directory holds plain-text files. Computer History does not encrypt them. Any program that runs as that macOS user can read them. Full-disk encryption on the Mac is not the same as encryption by Computer History. A backup agent, a sync client, or another coding agent that runs as that user can read the same files. Know that before you decide.

3. When to clear the history

Clear the history when the loop commits an unwanted result. Clearing history deletes the interaction events and any memories created from them. You cannot undo this action. Do not confuse it with the 48-hour automatic deletion of temporary event files. The event files delete themselves after 48 hours. The generated Markdown does not.

4. Which training clause applies to you

Check your ChatGPT data controls before you assume a later chat sits outside training. The event files are one clause. The chat that reloads the memories is another clause. “This chat content may be used to improve OpenAI models if allowed by your ChatGPT data controls.” Review your data controls setting. Find which clause applies to which kind of content.

5. Where the feature is not available

Treat geography as a hard gate this week. OpenAI states the restriction as current, not permanent. Computer History is “not currently available in the European Economic Area (EEA), Switzerland, or the United Kingdom.” Do not tell a London or Zurich teammate to turn it on. Do not write a policy that assumes the gate never moves.

6. What the write costs in tokens

Count the tokens as part of the cost of the write. A machine that summarizes a session into skysight incurs that token cost even when nobody asked a question. No published rate exists. Budget for it anyway.

7. What to do with an automation suggestion

Review an automation suggestion before you accept it. A summary of one afternoon is not a reviewed skill. Accept the suggestion only while a human still owns the skill file.

8. Where this layer belongs

Keep this layer where it belongs. Computer History is working memory for ChatGPT and Codex. It is written from a local event stream. It is not a reviewable team store. It is not a graph anyone merges. Institutional memory writes through a deterministic boundary the team controls. Computer History is a useful working-memory layer. It is not a substitute for that boundary.

The write path in one paragraph

The loop observes interaction events locally. An ephemeral Codex session on OpenAI servers summarizes them. The result lands on the Mac as durable working memory, in Markdown that Computer History does not encrypt, under skysight. It is a memory write path, not a graph. Treat it as one.

Sources

OpenAI, Computer History documentation: https://learn.chatgpt.com/docs/customization/computer-history
OpenAI changelog, entry dated 13 August 2026: https://learn.chatgpt.com/docs/changelog
OpenAI Help Center, ChatGPT Release Notes, entry dated 13 August 2026, section “Computer History for macOS”: https://help.openai.com/en/articles/6825453-chatgpt-release-notes
The Verge, Terrence O’Brien, 16 August 2026: https://www.theverge.com/ai-artificial-intelligence/980742/chatgpts-computer-history-tracks-your-clicks-and-keystrokes

Every quotation in this article comes from the first source unless the sentence names another.

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

Universal Agent Plugins: Codex Plugins. Grok Build Plugins. Claude Code Plugins.

Rick Hightower — Sat, 22 Aug 2026 05:08:36 GMT

You maintain two wrappers for the same skill, because one client wants it here and the other wants it there. Neither wrapper does any work. Both need to be updated every time the skill changes. The duplication is not your architecture. It is the absence of an agreed folder layout.

Summary: Agent Plugins 1.0.0 provides Skills and MCP servers with a shared directory layout. Vercel proposed it, five companies steer it, and nine clients now load it. This article covers what the format defines, the five things version 1 deliberately refuses to define, and why none of it touches your harness. The spec page still labels 1.0.0 a Working Draft, and that status matters before you build on it.

A plugin is a directory. Agent Plugins 1.0.0 is an open, vendor-neutral package format for that directory. The root file is plugin.json. Skills live under skills/. MCP servers are declared in mcp.json. Client-only extras sit in a reverse-domain folder, and every other client ignores that folder.

The format does not run anything. It says where the parts live.

Vercel proposed it, and Google joined later

Do not call this Google’s plugin standard. Vercel proposed it on August 6, 2026. Representatives from Amazon Web Services, Anysphere, GitHub, Microsoft, OpenAI, and Vercel refined it.

The Technical Steering Committee core maintainers at launch are AWS, Cursor, Microsoft, OpenAI, and Vercel. AWS describes itself as a founding member. Google joined the committee as a core maintainer after that launch, with Kevin Hou of Google DeepMind as the named representative.

Anthropic is not on the launch committee and is not among the refiners.

The specification lives at agent-plugins.org/specification, with schemas beside it and public governance in the spec repository. The specification page still labels version 1.0.0 a Working Draft. Treat that as the official status.

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

What the package is

The specification is explicit about its own restraint. A plugin is a folder.

my-plugin/
├── plugin.json
├── skills/
│   └── summarize/
│       ├── SKILL.md
│       ├── scripts/
│       └── references/
├── mcp.json
└── com.example.client/

plugin.json needs exactly two fields. $schema must be https://agent-plugins.org/schemas/1.0.0/plugin.schema.json. name runs 1 to 64 characters, lowercase alphanumeric plus hyphens and periods, starting and ending alphanumeric, with no -- and no ... Everything else is optional metadata or a client extension.

The client never guesses where the parts live. Skills are immediate children of skills/. A child counts as a skill only when it holds a path named exactly SKILL.md that resolves to a regular file. The specification then forbids the client from searching deeper: clients must not recursively search descendants. The skill format itself is not redefined here, because Agent Skills remains the source of truth for SKILL.md.

MCP servers are not inlined in the manifest. They live in mcp.json at the plugin root. Each entry declares a transport type: stdio, streamable-http, or the legacy sse. A client that supports Agent Plugins MCP servers must support at least one of stdio or streamable-http. It uses the declared type for the first connection attempt, and the specification defines no fallback when that attempt fails. The Model Context Protocol still owns the wire.

Failures stay isolated. A failure in any component type, entry, or process must not prevent the client from loading components that are independently valid. A missing skills/ directory is not an error. A dead MCP server does not take the skills down with it. A broken skill does not disable a working sibling.

Google’s own post says when a plugin earns its keep. Ship mcp.json alone when you have one MCP server for one client. Skip the plugin entirely when you have one skill. The format pays off when components belong together and must travel together.

What version 1 leaves out

Most recaps skip this section, and it is the important one.

Reading the table. Question states one thing a team needs answered before running a stranger’s code, written as a plain question rather than a spec term. Answer in version 1 says whether the specification settles it, as Defined or Not defined, with no middle value. Who owns it names the layer that must settle the question when the specification does not, which in every unsettled case is the client you run.

Two rows are defined. Six are not, and the specification excludes them deliberately rather than by oversight. Package distribution, installation, sandboxing, permission models, trust and signature verification, and end-user discovery all sit outside version 1.

The specification also leaves hooks, commands, agents, rules, and language-server extras with each client. Version 1 standardizes two component types, because Skills and MCP already have independent specifications and real adoption. The rest wait in reverse-domain namespaces until their semantics converge.

A reverse-domain folder such as com.example.client/ is the escape hatch. One client puts its hooks or commands there, and the other clients ignore the folder. The portable core stays small, because the non-portable parts have a legal home.

Path containment is not a sandbox. Any file or directory the client discovers, reads, or executes must be resolved within the plugin root. The rule stops a plugin from pointing command at ../bin/server. It does not sandbox the subprocess, and it does not restrict the paths that process opens later. PLUGIN_ROOT and PLUGIN_DATA are the two reserved variables, both delivered to subprocesses, and PLUGIN_DATA is client-managed writable state that survives an update. Secrets do not belong in headers or env, and version 1 has no portable OAuth field.

If you need a marketplace, a permission prompt, or a trust score, you stopped reading this specification. You are reading a client.

Who loads it today

The client list has grown since launch. The Vercel launch post named five clients. The official compatible-clients page now lists nine.

Reading the table. Client gives the product name exactly as it appears on the official compatible-clients page. MCP transports lists the transports that client accepts, drawn from the same page, where legacy SSE is the optional third transport the specification marks as legacy. At launch, answers whether the Vercel launch post of August 6 named the client as Yes or No, so a No marks a post-launch addition.

Every one of the nine loads Agent Skills. Six accept legacy SSE. Three do not: ChatGPT & Codex, Hermes Agent, and NanoClaw. Check that column before you declare a plugin sse-only.

Google names two of its own products. Agents CLI packages Google’s expertise in agent building, evaluation, deployment, observability, and publishing. Data Agent Kit ships plugins for Google Data Cloud services, including BigQuery, Spanner, and Cloud SQL. Neither appears on the compatible-clients page yet.

AWS shipped support at launch across Kiro and the AWS Agent Toolkit. Kiro is an agent harness with steering mechanisms. The AWS Agent Toolkit is a collection of official AWS-supported MCP servers, skills, and agent plugins.

Anthropic holds no committee seat. Claude Code can load Google’s Agents CLI skills, because those skills already travel as Agent Skills. Consuming a package is not a seat on a committee, and SKILL.md comes from Agent Skills rather than from Agent Plugins. Do not invert that.

What this is not

Three collapses would wreck the analysis.

Not a Claude plugin. Claude plugins sit on top of Claude’s loop. They do not swap the driver. Agent Plugins does not touch a driver at all. It says where to find a skill and an MCP server.

Not DeepSeek Harness. DeepSeek Harness made the loop driver a first-class plugin, and its real factory seam is ctx.agents.setFactory. Agent Plugins holds no opinion about who drives the loop.

Not Agent-to-Agent. A2A governs how agents talk to other agents. Agent Plugins governs how one client finds a folder of skills and tools. Adjacent discovery work sits before invocation, and packaging is a different layer. You can publish a plugin with no catalog entry, and you can run a skill with no plugin at all.

Keep the vocabulary locked. A harness is the control plane: hooks, tool contracts, evaluations, guardrails, recovery, and write policy. A loop is one session of discover, plan, execute, and verify. In this specification, a plugin is a directory. Do not let the word “plugin” move the driver, the write authority, or the graph.

What, no Hooks?

I’m going to be honest. I’m going to say that what's really missing from this is agent lifecycle hooks. They added hooks to Codex this year. Before Gemini CLI went defunct, they added hooks to it. You have hooks in Grok Build. You have them in Claude Code (first, as always), of course. And there’s something sort of equivalent in Cursor. Why not add hooks? OpenCode has lifecycle-hook-like plugins, but why not have an official way to implement hooks? Yeah, maybe you could do plugin extensions, but they all seem to have some mechanism for these lifecycle callbacks (aka hooks). Why not add it to this spec? Seems like something’s missing without it. And these hooks are a powerful feature. If they’re going to copy stuff from Claude Code, why would they miss one of the most important things about a plugin? And that’s the lifecycle hooks.

What to do this week

Read the specification, then the Google and Vercel posts. Create a single folder if you want the contract in your hands. A plugin.json plus skills/greet/SKILL.md is a valid plugin.

Do these four things.

Stop maintaining a second wrapper for the second client if you already ship Skills or MCP. Put the shared parts in this layout.
Keep client-only hooks and commands in a reverse-domain directory, where other clients ignore them.
Check the transport column before you commit to sse. Three of the nine clients do not take it.
Decide your write policy before you load a stranger’s folder. Version 1 will neither sandbox it nor tell you who signed it.

The client still owns install, permission, and trust. The split is correct. The split is also your work.

You can support multiple formats from the same repo / marketplace

When I updated my Claude Code plugins, I added the Grok Build plugin. That part is easy. Grok Build natively reads Claude Code plugins. Same format. It can define its own layout, but it can also use Claude’s. Then I added native Codex plugins so the hooks actually fire on Codex. Then I added Universal, so the same clone has a root plugin.json that agent-plugins.org understands. Universal has an extension path, so I can hang Cursor hooks off that too.

One clone. Five hosts.

Universal gets the shared folder: plugin.json, skills/, mcp.json. That is the standard. That is also where the standard stops. Version 1 has no hooks.

Claude Code keeps hooks/hooks.json. Grok Build runs that same file. Codex gets a translated hooks/codex-hooks.json when Claude has hooks. Cursor gets hooks/cursor-hooks.json with Cursor’s event names.

Policy scripts stay shared. Event names do not. Grok does not get a fourth hook file. It runs Claude’s.

That is the whole point of putting this in the article. I am not waiting for the spec to grow a hook section. I ship Universal for hosts that load it, and I ship host-native hook files alongside it so Claude, Grok Build, Codex, and Cursor still receive callbacks.

Do they all have ways to define hooks? Probably. That is why leaving hooks out of the standard is a problem. If you have to learn each vendor’s special sauce for the most important part of a plugin, the standard is only half a standard.

Universal is the folder. The harness and the hooks are still mine.

Foundation (hooks: Claude + Grok + Codex + Cursor)

okf-graph-eng 0.7.4 — PostToolUse → Codex same; Cursor postToolUse + afterFileEdit
project-knowledge-capture 0.7.4 — same three-host map
system-architecture-capture 0.4.4 — same
data-engineering-knowledge-capture 0.3.5 — same
worklog 0.24.4 — Claude PostToolUse, UserPromptSubmit, Stop, SessionStart under plugin/hooks/; Codex drops PostToolUse; Cursor beforeSubmitPrompt / stop / sessionStart

ContentPacks (hooks: three-host)

second-brain-core 0.3.7
executive-coordination 0.3.5
account-management 0.3.4
sales-pipeline 0.3.4
executive-job-search 0.3.4
consulting-leads 0.3.4
content-media 0.3.4
news-digest 0.3.4
gtm-positioning 0.3.4

Graph / translators (five-host install, no plugin hooks)

They emit hooks into generated plugins. The translator repos themselves have none.

okf-agent-graph 0.6.4
claude-code-ager 0.1.1
grok-build-ager 0.1.1
codex-ager 0.1.1
langgraph-ager 0.1.1
crewai-ager 0.1.1
claude-agent-sdk-ager 0.1.1
claude-managed-agents-ager 0.1.1
langchain-deep-agents-ager 0.1.1
google-adk-ager 0.1.1

Skills catalog, with hooks

google-docs-style 1.1.2 — Claude SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, Stop → Codex same → Cursor native set including afterFileEdit
spillwave-ui-guard 0.2.2 — PreToolUse / PostToolUse / afterFileEdit / sessionStart + Codex/Cursor copies

Skills catalog, install only (no Claude hooks → no Codex/Cursor hook files)

architect-agent 3.2.0
ste100 0.1.3
developing-with-docker 1.1.0
using-localstack 1.1.0
creating-zola-static-sites 1.1.0
automating-mac-apps 1.1.0
mastering-python-skill 1.1.0
mastering-aws-cdk 1.1.0
codebase-wizard 1.4.0

Catalogs

second-brain-marketplace — spillwave-second-brain v0.4.0
skills-marketplace — spillwave-skills v0.1.2

Count: 35 plugins. One clone each. Five host manifests. 16 ship three-host hooks. 19 have no hook policy to translate.

It was when I was updating these that I learned about the Universal Agent Plugins.

Sources

Agent Plugins, specification 1.0.0, page status Working Draft
Compatible clients, read 2026-08-18
Agent Plugins specification repository
Jonathan Hefner, Introducing Agent Plugins, Vercel, 6 August 2026
Kevin Hou, Haoyu Wang, and Alan Blount, Agent Plugins package your skills, tools, and more, Google Developers Blog, 6 August 2026
AWS Supports Agent Plugins, AWS Open Source Blog, 6 August 2026
Agent Skills specification
Model Context Protocol specification

Rick Hightower is a Claude Certified Architect, a former Senior Distinguished Engineer, and the author of the Manning book on harness engineering. Through Spillwave he works as a resident architect for production agents. The work covers harness engineering and loop engineering, runs 4 to 16 weeks, and is available in Austin and remote.

If this just showed up in your loop, read Harness Engineering or the Spillwave guides. To talk through a stand-up, book 30 minutes or write to contact@spillwave.com.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

Grok Bot, Claude Code, and Codex Share One Second Brain

Rick Hightower — Sat, 22 Aug 2026 04:18:54 GMT

The tab closes. The context window is gone. The next session starts blind.

Summary: The context window is the agent's working memory. I have agents running in Grok Bot, Claude Desktop, Claude Cowork, Codex Desktop, Grok Build, Claude Agent SDK, LangChain DeepAgents, and Claude Code. Each agent holds the lead, the draft, and the next action. It disappears when the session ends. A flat folder of notes is the common patch. It fails when the tree grows too large to read. Two rules hold it together: reads happen on the main thread; writes happen on a branch. This part explains what a second brain / LLM Wiki provides: a typed, git-native knowledge graph adds a type declaration per file, named relations between files, and a write boundary that puts a script between the model and the tree. Using a second brain/LLM Wiki lets you share data with Grok Bot.

Open the Grok Bot inbox. It does not look like a chatbot. It looks like a roster.

Chief of Staff. Media Consultant. Pipeline Sales. AI News Journalist. Client managers. A job-search agent. A consulting-leads agent.

Each conversation is a different actor. Each actor owns a different slice of the institutional graph. They do not share a prompt. They share a git-native knowledge tree with the laptop agents.

That is the design. Agentic AI at production scale shares a tree. Without a write contract, shared state becomes a clobber problem.

In a previous article, I had mentioned using OKF to create LLM wikis. And we did that for SDLC concepts. We did that for PKC concepts. We did that for architecture and deployment concepts. And we also did it for data engineering. So you have different vocabularies to build different concepts within a wiki LLM that we can then read later. Now, I started working with Grokbot, and I immediately fell in love. I was like, this is so cool. But then I have all these agents and jobs and whatnot running on my laptop. And I wanted to share information between Grokbot and what’s happening on my laptop using other agents, right? Claude, Grok, Codex. So I wanted a way to bridge them. So what I did was, for each agent in Grokbot, I defined a different vocabulary using these OKF plugins. And you can see these OKF plugins under Spillwave Solutions. There’s a whole list of them. And each of those maps to a different role my agents are in. And then they basically can read and write to these. And then I have things that are running on my laptop; they can read and write to the same concepts. This way they can share information. So I’m using the second brain to share information between agents running locally on my laptop for things that aren’t Grokbot, like Claude Code, Grok Build, Codex desktop, Claude Cowork, and various jobs I have running on my local machine. And routines. And now they’re editing and reading the same second brain.

Engineering Reliable Agentic AI Systems — Workshop

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours. Hands-on the entire time. Please sign up.

Every module ends with work you keep and can run on Monday:

A working autonomous agent loop that runs on your machine in the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

This is a build session. You type for most of the four hours. You need comfort in a terminal. You do not need prior experience with agent frameworks.

Readers of this newsletter get 40 percent off with the code RICK40. The code expires on Friday 28 August, the day before the workshop. There is no discount on the day.

Reserve a spot. Add RICK40 in the promo code box on the registration form.

The AI Agent Memory Problem

What happens when the tab closes?

An AI agent session has working memory. That memory is the context window. It holds the lead, the draft, the next action. It is gone when the tab closes. The next session starts blind.

A folder of notes is the usual patch. That works until the tree is too large to read. Then AI agents guess. A file tree without types, without named links, and without a retrieval bound is not memory. It is a place to lose things.

The second brain adds three things a folder of notes cannot.

Types. A file declares what it is. A consulting lead is not a job lead. A job lead is not an opportunity. Without a declared type, a file is its filename and its text. AI agents reading a flat folder cannot confirm what any file represents. They infer. A ConsultingLead, a SalesLead, and a JobLead each belong to separate plugins in the type registry. Each carries a different type in its frontmatter. brain.py write stamps that type against the registry before committing. A known identity that attempts to write a type owned by another plugin is refused. The type is in the file, not in the agent’s reading of it.
Typed links. Files point at each other with a named relationship. A graph walk finds the neighbors of one node without reading the tree. A relation such as belongs_to or depends_on is a named edge from one node to another. The pack command runs an outbound breadth-first search (BFS) from a single root node, following its edges. The defaults are 2 hops and 20 nodes. The walk stops on either bound: it stops expanding past the hop count, and it stops adding nodes once the node count is reached. A flat folder has no edges. An agent loading that folder must read all files or select by filename. Neither approach stays bounded as the tree grows. Typed links make the retrieval boundary exact and graph-shaped.
A write boundary. The model proposes. A script commits the frontmatter. You can opt for a human reviews the diff. Three cases cause a write to be refused before it touches the tree. A write with no claimed identity is refused. A known identity writing a type owned by a different plugin is refused. A rostered actor writing a type on its deny list, or outside its allow list, is refused. After those checks pass, the write lands on a branch. A human can review the diff.

The write side is governed by branch, script, and an optional human diff. The read side has one rule: reads happen on main. At session start, the agent fast-forwards to the tip of main, then builds a pack: a bounded breadth-first search (BFS) walk that pulls only the nodes the session needs, not the whole tree.

Reads on Main: Agent Knowledge Graph Retrieval Without Embeddings

The Session-Start Hook

Reads happen on main. That is the first rule.

At the start of a session, the agent fetches origin and pulls main (from the second brain repo, which is a different repo than its home repo) with a fast-forward only. A session-start hook runs that pull. The hook skips the pull in three cases and prints the reason each time: when the working directory is not a git checkout, when HEAD is not main, or when the working tree is dirty. It never fights a write session. It never merges and never forces.

The pull is how a Claude Code laptop session sees what a Grok Bot merged. It is how a Grok Bot sees a decision a laptop agent recorded. The shared truth is the tip of main. Chat is not the shared truth. As docs/architecture.md states: “Session chat is working memory. It dies with the tab. Packs are how working memory borrows institutional memory without eating the tree.”

Multi-Hop Knowledge Retrieval: The Pack

After the pull, the AI agent packs. It does not dump the tree.

A pack is an outbound breadth-first search (BFS) walk from one named root node. The default is two hops and twenty nodes. The walk stops on either bound: it stops expanding past the hop count, and it stops adding nodes once the node count is reached. The output is one Markdown file. The header carries the hop count, the node count, and a generation timestamp.

Pack files are gitignored. They are working memory. They are rebuilt on demand. They are never committed.

Graph Walk vs. Semantic Search: No Embeddings on the Retrieval Path

Two bounds constrain the walk: hop count and node count. The walk does not reach the whole tree.

There are no embeddings on the retrieval path. Retrieval is a typed graph walk. Semantic search returns things that resemble the query. A typed graph walk returns the things that are actually connected. For pipeline state, connection beats resemblance.

Reading from main is the first rule. The second rule covers the other direction: writes never happen on main. The read path walks the graph to surface what is already connected. The write path routes every new node through a branch, a script, and a pull request before it can reach the shared tree.

Writes on a Branch: The Four-Step Write Path

The Four-Step Git Worktree Write Path

Writes never happen on main. That is the second rule.

Laptop agents and Grok Bot write the same tree. Concurrent writers on main cause clobber. Granted, GrokBot agents typically run in the cloud on another machine, but each AI agent gets a Git worktree. Isolation turns two writers into a merge, not a lost file. Even Laptop-based agents can read and write to the same second brain using the same technique. If you need to write on the 2nd brain repo, create a worktree, create a branch, make your writes, commit/push the changes, create a PR to main, and merge to main. To read the latest, switch to main and pull. Agents are informed to do some work, write a batch of changes, then do the dance described above.

The write path has four steps. Three of them are not the model.

Claim. The agent claims an identity and a ContentPack. A ContentPack is the job function. It owns a closed set of types. The pack plus the claimed actor is the permission. A pack schema alone does not enforce actor uniqueness. A write with no claimed identity is refused. A known identity writing another plugin’s type is refused. An author string that matches no known identity passes both checks without failing. An unowned type is refused. The script rejects the write. The model does not get a vote.
Isolate. The agent opens a fresh git worktree for this session. The branch shape is brain//. Reads still come from main. The agent never packs another agent’s unmerged work. It never reuses another actor’s worktree or branch.
Write through scripts. The model supplies type, title, body, and links. The script stamps frontmatter against the type registry. It emits a write event. It stamps links: []. Relation validation is a separate step. A separate brain.py validate call refuses an unknown relation, a broken link, a missing type, and a missing title. Skipping the script is why handwritten agent notes rot. The model writes slightly different frontmatter every time. Six weeks later, nothing queries cleanly.
Close. Commit. Push to the existing remote. Open a ready pull request. Merge when checks are green. Never invent a remote. Never write to main. Never overwrite. Never force-push. If the same slug is already in an open pull request, stop. Refresh from current main. Revalidate. Do not smash it through.

Three Levels of Write Provenance

The human can reads the diffs. Provenance lives in git history. The commit message carries the actor. A write event records the details of the write. A one-line log records the batch for a person. Three levels. One review.

Git Worktree Isolation: Concurrency Safety, Not a Security Boundary

Isolation is a worktree, a branch, and a pull request. It is not a sandbox. It is not a shared-runtime security boundary. All Grok Bots share one Linux machine. Separate chats are not a security boundary.

Those three refusal cases depend on two data structures: the actor roster and the type registry. The roster holds each actor’s identity string and permitted write targets. The registry records which plugin owns each of the 45 types across the knowledge tree. Access control at write time runs against both.

AI Agent Access Control: The Roster, the Registry, and the Lane Handoffs

The system enforces AI agent access control at two levels. The first level is the pack. The second level is the actor allowlist. Both together define who writes what into the shared knowledge tree.

The Actor Roster and Fail-Closed AI Agent Allowlist Policy

Who decides what each actor can write?

The roster holds 13 actors. Nine are safe to name: Chief of Staff, Spillwave CTO, R&D Architect, Media Consultant, AI News Journalist, Spillwave GTM, Pipeline Sales, Consulting, and Job Search. Four more are client managers. Their accounts are not named.

Each actor has an identity string. Examples: grok-bot/content-media, grok-bot/news-digest, grok-bot/sales-pipeline. Each actor record carries a pack, a bundle, concept_roots, a writes_allow list, and often a writes_deny list.

The pack schema does not enforce actor uniqueness. The roster does. The roster note reads: “Unique actors are policy, not pack-schema enforcement. This file is the private roster. Validation must fail on duplicate actor values.”

Two actors share one pack by design. Chief of Staff and Spillwave CTO both use the executive-coordination pack. The roster comment for Spillwave CTO reads: “Identical pack authority with Chief of Staff. Actor allowlist is the extra fence.” DailyDigest belongs to executive-coordination. It is Chief-of-Staff-only. The pack does not enforce that. The actor allowlist does.

The Type Registry and Multi-Agent System Permissions

The type registry holds 45 types across 9 plugins. It has 38 catalogs and 42 distinct relation names. The knowledge tree holds 295 Markdown nodes.

Each plugin owns a closed set of types:

second-brain-core (5): AgentIdentity, Concept, ContextPack, TypedEdge, WriteEvent
content-media (5): Article, Draft, Headline, Series, Subscriber
news-digest (5): Digest, FollowUpCandidate, NewsItem, Source, Topic
executive-coordination (5): ActionItem, Blocker, DailyDigest, Decision, Priority
account-management (6): AccountPlan, Client, Commitment, Contact, Deliverable, Stakeholder
gtm-positioning (6): Campaign, IdealCustomerProfile, MessagingPillar, Offer, PositioningStatement, ProofPoint
consulting-leads (5): ConsultingLead, DiscoveryCall, EngagementType, QualificationNote, Scope
sales-pipeline (4): NextAction, Opportunity, SalesLead, Stage
executive-job-search (4): CompanyTarget, JobLead, Role, TargetCriteria

Each type declares its plugin, its folder, and its allowed relations. Article and Draft belong to content-media. NewsItem and FollowUpCandidate belong to news-digest. That is the mechanical reason the articles actor cannot write a NewsItem. A known identity writing a type owned by a different plugin is refused.

One Graph, Several Lanes: AI Agent Role-Based Access Control in Practice

A plugin on a host is only a loader. Claude Code, Codex, Cursor, and Grok Bot can all touch the same catalogs. They must claim identity. They must pull main. They must pack first. They must write through scripts. Agent Plugins 1.0.0 is a package format for skills and Model Context Protocol (MCP) servers. It is a host binding. It is not this write policy.

The Media Consultant actor has writes_allow: Article, Draft, Series, Headline, Outline. It has writes_deny: PositioningStatement, NewsItem, FollowUpCandidate. The AI News Journalist has writes_allow: NewsItem, Digest, FollowUpCandidate, Source.

The deny list is the mechanism. The articles actor packs a FollowUpCandidate that the news actor wrote. The articles actor cannot write that type itself. The news actor writes it. The articles actor reads it through the pack. The articles actor then works the draft in the local satellite and writes an Article when the URL exists.

status: ready means the draft is waiting on a public URL. status: published means the URL is in the record. Two public copies of one piece link with related_to. Do not invent a new relation name.

Lane Handoffs Across the Multi-Agent Pipeline

Consulting qualifies inbound. Sales takes over when there is a real opportunity. Account management takes over after signature. It stays off the outreach machine. The job search only promotes interviews and offers. The daily scan firehose stays in the satellite tree.

A hold is a machine-readable state. It stops an outreach bot from sending into a thread where the ball is not on our side. A no-bid is written down. An opportunity that stops being mentioned looks identical to one that was forgotten. A dead node, with its reason attached, prevents a future agent from reviving it.

A tracker entry without a verification date is a claim, not a fact.

Lane ownership determines which actor can write a given type. Where the written record lives after it is written is a separate question with its own boundary. The satellite holds high-volume working records. The shared tree holds only the durable institutional nouns that other lanes need to pack.

On my local machine, I have an agentic workflow to help me manage articles that I’m writing. Helps me do research and other things. On my Grokbot, I have a media manager that helps me promote and keeps track of Substack and Medium statistics and whatnot. When I schedule or plan articles on my laptop, I want the media manager running on Grokbot to have access to them. So it’s two different systems that I want to have the same information.

Local system to manage and plan articles (outlines, ideas, etc.) on the laptop versus media manager running in Grokbot that manages Substack and Medium stats. I also have other systems to manage leads, contracts, contacts, etc. I want that data available via my local jobs and via Grokbot.

Two Trees and a Shared Second Brain for AI Agents

What satellite trees keep local, what they promote, and five things the shared tree is not

Session chat is working memory. It dies with the tab. The second brain is institutional memory. It survives.

Summary: This part defines the boundary between two satellite trees and a shared second brain for AI agents. Drafts stay in the satellite. Durable institutional nouns get promoted to the shared tree. Three failure modes break that boundary: status drift, entity duplication, and promotion pressure. The part closes by naming five things the system is not: not a vector database, not a customer relationship manager (CRM), not autonomous, not Agent Plugins 1.0.0, and not Agent-to-Agent (A2A).

Two satellite trees and one shared second-brain checkout

There is a local satellite tree and a shared second-brain checkout.

The satellite holds high-volume working records. People. Companies. Interactions. Daily job-scan hits. Draft revisions. Most of that is nobody else’s business.

The shared tree holds institutional nouns that other AI agents need. A published article. A consulting inbound that actually qualifies. A job lead that is in interview or offer state. An opportunity with a buyer and a number. A hold that must stop outreach.

Drafts stay local. Records get promoted. The articles agent never copies a working draft into the shared tree. It writes a short typed record that points at the public URL. The consulting agent does not copy the people-and-deals graph into the shared tree. It promotes a durable noun.

That boundary is why the system does not collapse. A shared brain that accepts everything becomes the same unreadable pile. Now it is unreadable for the whole organization. The value of the shared tree is inverse to its size. Its job is to stay small enough to pack.

What breaks: status drift, entity duplication, and promotion pressure

The trees drift. A shared hold can be true at 03:00 and stale by afternoon. Status is verified. It is never cached. Trackers are a cache. The mailbox and the calendar are the source of truth. Any status the current session could not confirm is unverified. A confident stale fact is worse than a missing one.

Entities duplicate. Two slugs for one person are ordinary entropy. The contact card gets merged. The graph node does not. That needs a reconcile pass, not good intentions.

Promotion pressure is constant. More in the shared tree feels safer. It is not. Every satellite rule is a guard against this. The guards need enforcing.

What this shared second-brain system is not

It is not a vector database. There are no embeddings on the retrieval path. Retrieval is a typed graph walk from a named root. It is deterministic. It is reviewable.

It is not a customer relationship management (CRM) system. There is no schema migration, no server, and no vendor. It is Markdown, YAML, and git. A person can read any node. A person can undo any agent write with git revert.

Note that the default mode is for the agent to do a PR and just merge it for most agents, but you can gate it.

This is a write policy. A marketplace install is a loader. This is not.

It is not Agent-to-Agent (A2A). A2A is how agents talk to other agents. This is how agents write into one reviewable graph.

Isolation is not a security boundary

A worktree, a branch, and a pull request give concurrency safety and human reviews for safety (optional). They are not a sandbox. All hosted Grok Bots share one Linux machine. Separate chats are not a security boundary.

Agents need a role, a pack, and a place

Agentic AI needs a role, a pack, an identity, and a place where the work does not disappear when the session ends.

AI agents pull main to see the graph. They write on a branch so they do not smash it.

Session chat is working memory, and it dies when the tab is closed. The second brain is institutional memory, and it survives. Packs are how one borrows from the other without eating the whole tree.

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

Build a Second Brain for your AI coding agents, Create Self-Improving Loops, and Master Graph Engineering

Rick Hightower — Fri, 21 Aug 2026 02:51:01 GMT

You and an agent solved a hard problem yesterday. The reasoning was sharp. The trade-offs were clear. Then the session ended. Two days later, a different agent reopens the exact trade-off you already settled, and every hour you spent is gone. You are not losing that time to a weak model. You are losing it to an architecture that throws the answer away when the session closes.

Summary: This article is the written version of my 20 August 2026 Packt Expert Network talk. It makes one argument: the repository is the second brain, and the event log is the source of truth. Along the way it covers the five layers of the stack and why each one got named, the deterministic write boundary that keeps the model out of your knowledge tree, an append-only work log and the fold that turns it into current state, the Open Knowledge Format and the typed edges that carry causality, ContextPacks and the exact budgets that bound them, and two production evaluation harnesses built on the loop-plus-deterministic-anchor half of this pattern. Every tool listed here is open source, and every number is verified against the source code rather than a slide.

The failure everyone recognizes

Watch a team six months into serious agent adoption, and you see the same three failures.

Quality collapses under machine volume. More runs produce weaker outcomes. Volume goes up, and the hit rate goes down, because nothing that ran yesterday informs what runs today.

Decisions get re-litigated. A trade-off you settled in March reopens in August. Nobody is being lazy. The reasoning simply is not anywhere a new agent can find it.

Knowledge stays tribal. It lives in chat logs and in the heads of four power users. When one of them takes two weeks off, throughput drops more than headcount math predicts.

Those look like three problems. They are one problem wearing three costumes. The work has no durable shared state. Every run starts from zero and ends in a private transcript.

This is the whole diagnosis. Everything below is the treatment.

The workshop on the 29th

The free hour is the architecture. It deliberately stops short of building the evaluation harness and the graders because that build is the whole point of the paid workshop nine days later.

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours, hands-on the entire way through.

Every module ends with something you keep and can run on Monday:

A working autonomous agent loop, running on your own machine inside the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

Because you read this newsletter, the code RICK40 takes 40 percent off. It expires on Friday, 28 August, the day before the workshop, so there is no discount on the day.

Reserve a spot

Subscribe now

Five layers, and why each one got its name

Before the architecture, a bit of orientation, because the vocabulary in this space has multiplied fast enough to be confusing.

Five layers: prompt, context, harness, loop, graph.

There is a rule that makes the sequence make sense. Each layer got its name at the moment the layer below it stopped being where the failures lived.

Prompt engineering came first because prompts were where things broke. Then models got better at instructions and prompt quality stopped being the bottleneck. It is table stakes now.

Context engineering came next, because what you put in the window mattered more than how you phrased the ask. That layer gave us CLAUDE.md, AGENTS.md, and the whole practice of curating what an agent sees. You start breaking work into subagents, each with its own context, so you do not flood a single window. The goal is to avoid context rot, context panic, and the U-shaped attention curve, in which important material gets ignored.

Harness engineering arrived when the environment became the unit of design. Not the prompt, not the context, but the tools, the permissions, the file system, the verification the agent runs against. A good harness is almost more important than a good model. You can go farther with an average model and a strong harness than with a strong model and a weak harness. The model is the brain. The harness is the operating system and the control plane that connects it to the outside world and regulates what it can touch. Think of it as mechanical sympathy for the LLM rather than for a server.

Loop engineering made the cycle the unit. Discover, plan, execute, verify, repeat, with a stop condition. Early versions looked like simple retry loops. The useful versions set a goal and keep working until the goal is met or the exit conditions fire.

Graph engineering makes the topology the unit. Which nodes exist, which edges connect them, and what persists between runs. You break a problem into pieces, send those pieces through judges with rubrics, fold the results back together, and remember the paths you already tried so you do not repeat them.

Nobody skipped a step for fashion. Each name appeared when the previous layer got reliable enough that the failures moved upstairs. From the outside, the progression looks like a hype cycle. From the inside, it is simply where the next hard problem lived.

Let’s reason on these layers a bit
We had these different layers. This is sort of the memes, the viral things that pop up, right? Where there was prompt engineering, and then you sort of mastered that, and then context engineering came out, and you’re like, well, a lot of what I was doing in prompt engineering is actually context engineering. And then there’s harness engineering, which is sort of the right pieces of the context at the right time, and controls in place so that the LLM can interact with the computer, so you have these agentic computer interfaces and tool calling and tools that can be deterministically blocked under certain conditions. And later you have loop engineering, which is sort of setting goals and repeating the process until a goal is met. And then with graph engineering, we’re getting into orchestrators and judges and rubrics and providing feedback and setting loops and exit loop conditions, but this sort of whole graph of decision, breaking tasks up into smaller pieces and then putting them all through the different sort of judges and loops and whatnot to improve the output.
For context engineering, if you’re using a coding agent, you’re going to want to put things in CLAUDE.md and AGENTS.md, as well as rule files, to state things like: hey, these are the frameworks we’re using, and here are our coding standards. This is our coding style. These are sort of the rules for using this codebase. And that’s also where agentic skills come in. And then the harness engineering is basically when those skills get loaded and how, right? If you’re working in a certain use case, let’s say you’re doing a CDK deploy, the harness is going to see that you’re doing a CDK deploy, and it’s going to say they probably want to use this CDK skill to deploy to AWS. So it’s what provides the loading of these context systems at the right time.
And what we’re really trying to do with both of these is avoid overloading the LLM with things that don’t matter for the task at hand. And we’re trying to avoid things like context rot, context panic, and the U-shaped attention curve, where the important things we want done are just ignored. That’s sort of what harness engineering is. It’s this mechanical sympathy, but instead of mechanical sympathy for, like, a computer server- a modern computer server- it’s mechanical sympathy for how the LLM can interact with your computer system.
The harness is the control plane, and then there are constraints. Not everything’s going to fit, no matter how big the context window is; typically, not everything will fit inside of it, at least not yet, right? And even if it did, you don’t really want everything inside of it. You want the things that are needed for the task at hand, right? The planning phase uses many subagents that scan many files and summarize what they find, but each uses its own context so we don’t pollute the main orchestrator context. The harness enables and implements these constructs.
We know we have a strong recall problem at the beginning and end, so we use things like scratch pads and keep the most salient information at the beginning or the end of the prompt. We had this loss-in-the-middle issue. We have token cost; there’s just having the wrong context. You could be working on a task; maybe you’re doing a bug fix. You don’t clear the context; now you’re working on a new feature, and it just kind of gets confused between the two conversations, right? It’s not just about having the right amount; it’s about the right content in your context for the task at hand as well. The harness performs summarization while retaining salient facts.

Three distinctions the rest of this rests on

The table holds three distinctions, one per row. The left column names the thing people usually mean. The right column names what the practice actually requires. For each row, I have added the consequence of confusing the two, because that is the part a table cannot show. Read it as three separate corrections rather than one comparison.

Row one. A harness is not raw agent calls. Raw agent calls are a model with a prompt and a set of tools. A harness is the surrounding system: what the agent is allowed to touch, how its output gets verified, what happens when verification fails, and what gets written down. The consequence of confusing them is that you tune prompts forever and never fix what is actually failing.

Row two. Write authority does not belong to the model. A model that can propose a change and a model that can commit a change are different risk profiles. The consequence of confusing them is a knowledge store nobody trusts, because nobody can tell which entries were validated.

Row three. Memory is not one context window. The window is working memory and it dies with the session. Treating it as the only memory is what produces the three failures at the top of this article.

One line ties them together. The model is the CPU. The harness is the operating system you own.

The model proposes. Code commits

Here is the first architectural rule, and it is the one most teams get wrong.

On the left of that boundary, the model does what models are good at: judgment, extraction, summarization, drafting. It reads a meeting transcript and proposes that this was a decision, that this was an open question, and that this decision governs that feature.

On the right of the boundary, a schema-enforced script does the write.

In the second-brain-core plugin that function is literally called write_concept. It validates the frontmatter against the schema. It injects the timestamp. It refuses to overwrite a file that has been superseded. It skips a byte-identical write so the git history stays honest. Every write also emits its own WriteEvent concept, so the tree records who wrote what and when.

The model never writes into the knowledge tree directly. Not once.

This matters more than it sounds. A knowledge store is only useful if you trust it, and you only trust it if every entry passed the same gate. The moment a model can write freely, your graph accumulates plausible-sounding entries that nobody validated, and the first time an agent acts on one and gets it wrong, the whole store loses its authority.

Keep the judgment on the left. Keep the writes on the right. Put a schema in between.

Three memories, and the one most teams are missing

Memory in an agent system is not one thing. It is three planes, and they have different lifetimes, different owners, and different failure modes.

Procedural memory is how the team works. It lives in AGENTS.md, CLAUDE.md, and skills. It is the standing instruction set. Most teams have this.

Working memory is this turn plus whatever context got packed into it. It is fragile by design, and it dies with the session. Every team has this, whether they planned it or not.

Institutional memory is the shared tree in the repository. It is reviewable, attributable, and mergeable, and it outlives the session.

Most teams have two of these. Institutional memory is the missing plane, and it is the one that fixes the three failures.

The planes fail differently, which is why collapsing them causes trouble. Procedural memory fails by going stale: an instruction file that describes a workflow you abandoned in June is worse than no file, because agents follow it. Working memory fails by overflowing, and by burying the important thing in the middle. Institutional memory fails by going unwritten, which is the quiet failure, because nothing breaks and nobody notices until the same decision gets made twice.

The common mistake is loading institutional content into the procedural plane. A CLAUDE.md that has grown to four hundred lines because people kept adding project history to it is a procedural file doing institutional work. It gets loaded on every run, whether or not it is relevant; it is too long to review carefully, and it has no structure that lets an agent skip the 90% that does not apply. Keep the planes apart. Procedural memory says how the team works. Institutional memory says what the team learned.

Context is not free, and a bigger window does not save you

The instinct when an agent forgets something is to give it more context. That instinct is wrong, and there is research on exactly why.

Models show strong recall for material at the start and the end of the context window, and noticeably weaker recall for material in the middle. Liu et al. named it “Lost in the Middle” (arXiv 2307.03172). Attention degrades long before the window fills. You also pay token cost for every unused chunk, and you risk context rot when leftover material from a previous task confuses the current one.

A bigger window does not fix recall. It moves more of your material into the region where recall is weakest and charges you for every unused chunk along the way.

The second problem is a team problem rather than a model problem. Vector stores help one developer. Private auto-memory helps one developer. Neither is reviewable by a colleague, attributable to an author, or mergeable across a branch. Teams need memory with all three properties.

There is exactly one store that already has them. It is the repository.

You can also use a local vector store on documentation. Maybe you pull down a bunch of Confluence docs on ADRs, Design Docs, Decisions, User Guides, or related Jira tasks, and even codebases. I’ve done that with Agent Brain, a RAG system that is a Claude Code Plugin. The issue there is that it loads these items, but it doesn’t necessarily know what the relationship is. It might pull a document that’s an older decision that you’ve moved away from or something that applies to another project. It doesn’t have the context to know exactly where that fits. Also, people do a poor job of keeping their Confluence (Wiki) pages up to date. Then you try to add context around what you are pulling, but it is calculated guesses.
A lot of times, this institutional memory might exist in an email, in some meeting notes, in a Confluence page, and it’s sort of scattered about. And the one thing we know about wikis managed by humans is that they don’t always keep things up to date, right? The design docs are from two years ago. The architectural decisions may not be the latest; there were workarounds once the rubber hit the road and we actually had to go to production. So all those things sort of get lost in the stream of time, and you have sort of a stale snapshot of what the system was at a certain point in time. And it’s harder to kind of keep those in sync. But that’s where an LLM really comes in handy because it’s really good at keeping an LLM Wiki in sync.

Bounded context beats a bigger window

If dumping does not work, the alternative is tiered disclosure.

Tier one, always loaded. A catalog of names and one-line purposes. It is deliberately too thin to act on. Its only job is routing: telling the agent what exists so it can decide what to read next.

Tier two, on demand. The concept page itself. This is the working tier, the thing the agent actually reasons over.

Tier three, deeper on demand. The long tail. Most runs never touch it, and that is the point.

The rule underneath all three tiers: show the agent enough to decide what to read next, and nothing more.

One warning about tier one, because it is the tier people get wrong. A catalog entry must be thin enough that an agent cannot mistake it for the whole answer. A partial summary beats no summary in exactly zero cases, because a partial summary is confident. An agent that reads three lines and believes it has the picture will act on three lines.

Why the repository, and not a database

Every capability the institutional plane needs, git already has.

It is reviewable because a pull request is how your team already reviews changes. It is attributable, because every line has an author and a commit. It is mergeable, because that is what the tool does for a living. An ordinary git diff is your audit log, already built, already understood by everyone on the team.

What that buys you is concrete. A new teammate clones the repository on day one and immediately gains the accumulated context. Nothing depends on one person’s terminal history. Nothing depends on one vendor keeping a memory feature alive. The practice survives both the developer and the AI host.

No database required. That is not a limitation; it is the entire point. If it cannot be committed, it is not a practice.

Why I wrote WikiTicket SDD
I wanted visible work in progress. I wanted Visible Work In Progress, visible WiP. Where the stakeholders could see progress or decisions made in the coding session did not make it back to decisions documentation in Confluence, and I wanted one way to ingest, manage, and merge tickets in a uniform manner that the LLM/Code Harness could easily work with while also wanting to engage with stakeholders, other developers, Project Managers, Product Managers, etc.
I kept using GSD, Superpowers, and sometimes just Claude Code Plan (or Codex or OpenCode, etc.) to create hierarchical plans. Those plans worked inside the coding session. They did not sync to Jira or GitHub Issues, so the customer and the rest of the team could not see progress. Decisions made in the session rarely made it back into Confluence or other decision documentation either.
Anthropic’s harness-engineering papers described a simple pattern: a todo.jsonl file of tasks the agent iterates until the work is done. Tasks form hierarchies. One task depends on another. A task can be broken into subtasks that themselves have dependencies. That shape is exactly what GSD, Superpowers, and Claude Code plans already produce.
What was missing was one uniform way to ingest, manage, and merge those tickets so the LLM and the code harness could work with them cleanly, while still giving stakeholders, other developers, project managers, and product managers a live view of the same work.
WikiTicket SDD is that bridge. Plans become tracked items. The event log becomes the source of truth that both agents and humans can read. Adapters push visibility outward to the trackers people already use, without making those trackers the system of record.

Making the work visible

The knowledge graph is the second half of the story. The first half is smaller and more immediately useful: making the agent's work visible as it happens.

An agent finishes a complex refactor in eleven minutes. A second agent builds the test suite for a related feature. A third drafts the migration plan. None of the three can see what the others are doing. Neither can your product manager. Neither can the engineer who reviews the PR tomorrow morning. Architecture review boards, project managers, and scrum masters also need a view. Spec-driven development is not only for the agents and the developers. The work has to be visible to the stakeholders who approve the direction.

An append-only event log under git

Work items become events in a JSONL file at .work/todo.jsonl. Completed items move to .work/done.jsonl.

Each event carries its own ULID, a timestamp, an actor, the git sha at write time, the item it touches, the operation, and the fields it sets. Here is a real line from this repository, abbreviated:

{“actor”:“richardhightower”,“ev”:“01KZ2RXXPWGAB14T8B7RNH78AX”,“item”:“01KZ2RXXPWGJ3SDT0C0J0QZM6N”,“op”:“create”,“set”:{“body”:“...”}}

You append. You never rewrite. That single constraint is what makes everything downstream work.

Current state is a deterministic fold

If the log is the only truth, then the state of any work item is a function of the log. Not a file somebody maintains. A fold.

The rules are short enough to state completely. Here is a simplified version of what fold.py does across its dedupe_and_sort and _apply_mutations functions:

def fold_events(lines):
    seen = {}                                       # ①
    for ev in parse(lines):
        if ev[“ev”] in seen:                        # ②
            continue
        seen[ev[“ev”]] = ev
    items = {}
    for ev in sorted(seen.values(), key=lambda e: e[“ev”]):   # ③
        item = items.setdefault(ev[“item”], {})
        for field, values in (ev.get(“del”) or {}).items():   # ④
            item[field] = [v for v in item.get(field, []) if v not in values]
        for field, values in (ev.get(“add”) or {}).items():   # ⑤
            item[field] = item.get(field, []) + [v for v in values if v not in item.get(field, [])]
        for field, value in (ev.get(“set”) or {}).items():    # ⑥
            item[field] = value
    return items

① Dedupe first, keyed on the event ULID. A union merge duplicates identical lines, so the same event will appear multiple times and must be collapsed into a single occurrence.

② Skip the duplicate rather than reapplying it. Because dedupe runs before the sort, no two events reaching the sort can share a key, which makes the event ULID a total order in itself.

③ Sort by the event ULID. ULIDs sort lexicographically by time, so this is a plain string sort with no clock comparison and no tiebreak needed. A union merge also scrambles line order, so this step is load-bearing rather than cosmetic.

④ Apply deletions first. Set-valued fields, such as labels, use explicit add and delete operations rather than whole-field writes.

⑤ Then apply additions, skipping values already present. This is why two developers adding different labels on different branches do not clobber each other.

⑥ Then apply scalar sets. Per field, the last writer wins, and “last” means the highest event ULID, i.e., the latest in time.

Same events, same state, on every machine. No network needed, no server, no sync protocol.

What the fold produces is the useful part. Roadmaps, status reports, and design documents become generated artifacts rather than documents somebody maintains by hand. The roadmap in this repository is generated from the log, so it cannot drift from reality. If you want to see what this looks like in practice, open any repo running this pattern and look at docs/roadmap.md next to .work/todo.jsonl. The first is a view of the second.

Plans become tracked items, or they evaporate

A plan that does not emit tracked work items evaporates. Everyone has watched this happen. A careful design document is written, approved, and then the actual work drifts away from it, with nobody noticing until the review.

The fix is mechanical. Plan capture writes the plan to docs/plans/, creates an epic with one item per task, links them, re-renders the roadmap, and commits it all together. The plan file is then frozen. A changed design does not get edited; it gets a new dated plan that supersedes the old one.

Spec-driven development is the fuel for loop engineering. Without tracked items, the loop has nothing to close over.

Agents merge cleanly, and tickets stay where they are

Two lines in .gitattributes do most of the work:

.work/todo.jsonl merge=union
.work/done.jsonl merge=union

A union merge takes both sides. Alice sets a status on her branch. Bob bumps the priority on his. Both add a different label. After the merge, all three changes survive, because there is no mutable state file for them to conflict on. There are only appended events, and the fold sorts them out.

This is the payoff for the append-only rule. Conflict resolution stops being a human problem.

One practical note on trackers. Humans keep Jira or Linear. Agents trust the local log. The core never knows the word Jira, and adapters resolve the tracker at runtime. Tickets become ingestion points rather than the system of record. You do not have to migrate anything.

Why I wrote the OKF plugin and the PKC plugin for Project Second Brains
I wanted to capture project knowledge where the coding agent could retrieve it easily.
Spikes and proofs of concept produced useful results that vanished when the session ended. Research and discovery against large legacy systems did the same. Multi-agent runs compared notes in private context windows and then forgot them. Meetings, Teams, Slack, and email produced a constant stream of decisions, feedback, and new requirements. I needed one place to store and retrieve those decisions, requirements, and tasks so that the next agent would not start from scratch.
WikiTicket SDD solved the human side of visibility. Plans and tasks became visible work in progress for stakeholders, architecture review boards, other developers, project managers, and product managers. That still left the agent side open. The same knowledge had to be available to the LLM and the coding agents in a form they could load efficiently.
OKF is the open standard for that second brain, an LLM wiki. PKC is the plugin that describes the SDLC process (plans, meetings, decisions, spikes, experiments, discoveries, and requirements) for agents to consume. PKC listens to the WikiTicket SDD event stream and updates the LLM Wiki, the second brain for the project. Together they turn the work into a knowledge graph the harness can query without flooding the context window.
The result is one source of truth: WikiTicket SDD for the work log, source code and generated artifacts for the implementation, and a PKC-based second brain for the institutional knowledge. Unlike a human wiki that rots when links break and pages go stale, an LLM wiki stays current because the write path and the typed edges keep the graph consistent.
We do not want to re-litigate decisions. When we run a discovery or a spike and settle on the trade-offs, we do not want to forget that knowledge as we iterate on new features or similar work. We also do not want the knowledge to stay tribal. If someone tells you something and you put it only in the current coding session, it never gets stored, or it ends up in Confluence, where the coding agent cannot reach it. Chat logs, overheard comments, meetings, Slack, Teams, and one-off remarks all fall into the same gap.
That is the problem this work addresses. The repository is the second brain. First, make the work visible so stakeholders can see what is happening and contribute. Second, write the same work, the same knowledge graph, so agents can pick it up, even across different branches and work trees.

The second brain

The event log tells you what is happening. It does not tell you why anything was decided. That is the layer above it.

What sits atop the event log is a knowledge graph built from ordinary Markdown and YAML. The files live under version control. It is an LLM wiki: packed on demand rather than loaded whole. Portable, reviewable, cloneable on day one. No database required.

The LLM wiki idea is Andrej Karpathy’s, from April 2026. Compile the sources into a structured, interlinked wiki of Markdown files once, then query the wiki rather than the raw sources.

His observation is the reason it works now rather than in 2015. LLMs do not get bored, and they do not forget to update a cross-reference. The bookkeeping that kills human wikis is precisely the kind of work LLMs are good at. Every corporate wiki that rotted did so because maintaining links is tedious and nobody’s job. That constraint is gone.

OKF is the file format

OKF is Open Knowledge Format. It is a Google Cloud spec, published in GoogleCloudPlatform/knowledge-catalog. It is not mine. What I build sits on top of it, and it is worth being precise about which parts are which.

A knowledge bundle in OKF is a directory of Markdown files with YAML frontmatter. The only required frontmatter field is type. That is a deliberately low bar, which is why the format is usable on a Tuesday afternoon rather than after a schema committee meets.

The feature that matters most for disclosure is index.md at every level. That is the routing tier from the progressive disclosure model, made concrete as a file. An optional log.md answers the question “what changed here”. A citations section makes partial disclosure navigable rather than merely shallow, because it gives the agent the pointers it needs to go deeper on its own.

Here is the part to put plainly: base OKF deliberately keeps links untyped. Typed edges are my extension, carried in frontmatter. The spec gives you the file layout, the disclosure model, and the discipline. The causality layer is something I added on top, and the next section is about why.

What a concept file actually looks like

The abstraction is easier to trust once you see how small the artifact is. This is a decision record from the PKC sample bundle, unedited:

---
type: DecisionRecord
title: Use JWT for session management
description: Adopt short-lived JWT access tokens with httpOnly refresh cookies
status: accepted
tags: [decision, adr, auth]
timestamp: “2026-08-03T19:00:00Z”
verified: true
wiki_key: adr-jwt-session
truth_state: current
links:
  - target: /meetings/2026-08-03-auth-design.md
    rel: originates_from
  - target: /features/user-authentication.md
    rel: decides
  - target: /experiments/jwt-vs-cookie.md
    rel: informs
  - target: /risks/token-replay-after-logout.md
    rel: mitigates
---

# Use JWT for session management

## Decision

Use short-lived JWT access tokens + httpOnly refresh cookies.

## Related

- [Token Replay After Logout](/risks/token-replay-after-logout.md) (`mitigates`)

This is the whole thing. A file.

The frontmatter above the fence is what makes it a graph node. type says what it is, and the links list carries the outbound typed edges, each one a target path plus a relation name. The body below the fence is what makes it useful to a human reading the pull request.

Three fields in there are worth pointing at.

links is where the causality lives, and the shape matters. A relation is a pair, not a key. That keeps the vocabulary closed, which lets a validator reject an invented relation name rather than silently accept it.

truth_state and verified are how the tree handles the fact that not everything in it is equally solid. An unvalidated assumption and an accepted decision are both concepts, and an agent should be able to tell them apart before it acts on either.

wiki_key is the stable handle. Titles change. The key does not, which is what lets an external tracker point at a concept and still find it a year later.

And the whole file is reviewable in a pull request by a person who has never heard of OKF. That is not a small property. It is the thing that gets the practice adopted.

Three brains, one format

That table has one row per second brain and columns for what it captures, the questions it answers, and the kinds of concepts it owns. Read down the first column to see the domain, and across each row to see how the same machinery gets pointed at a different problem.

Row one, PKC, project knowledge capture. This is the main one, and it captures the why: meetings, decisions, experiments, assumptions, and open questions. The questions it answers are the ones that cost you the most when they go missing. Why did we choose this? What did we already try? What are we assuming that nobody wrote down?

Row two, SAC, system architecture capture. The same idea pointed at what is running: services, ownership, dependencies, and blast radius. The questions it answers are the ones you ask during an incident, or an hour before a risky deploy.

Row three, DEKC, data engineering knowledge capture. The same idea pointed at the data plane: lineage, medallion layers, streams, jobs, and the business glossary. The questions it answers are about where a number came from and what breaks if the upstream table changes.

Same format across all three. Same write boundary. Same packing. The three-letter name is also the repository name, which keeps the mapping from concept to code obvious.

What’s nice about this format is that this is the OKF format. If you’re familiar with the way agentic skills were, they have a partial disclosure architecture. So you have a skills.md file, and it will refer to scripts and references. You have a references folder. Typically, those references- each one of those files are quite small. They’ll have a table of contents at the beginning that describes what each section covers, so the agent can run a grep command and jump right to the section it needs. So it’s laid out so you don’t have to load everything.
So if you had, like, a modern Python agentic skill, it’s not going to throw the whole modern Python book into your context because you wouldn’t want that. You would get attention problems at the least and might cause a full-on context panic, where your coding agents start cutting corners. But if, let’s say, the coding agent needed information about how to use thread pools, it could navigate to skills.md, which would point it to a reference file that tells it where in that file to look. And it could load sort of what it needed at that time. It’s not going to load everything. It’s not going to overwhelm, but it’s going to partially disclose the pieces it needs.
OKF has the same mentality with the partial disclosure architecture. The difference is that it’s dynamically created over time. You’re storing different pieces of knowledge that then the LLM or the coding agent can pick up fairly efficiently, as efficiently as it would if it were an agentic skill. But an agentic skill is sort of frozen in time, right? It’s this version of this skill, and it has to be edited and released so the second brain can evolve as you go through use cases.
We’re going to build it on top of OKF. And when we do that, we want to have… so we have the brain that provides judgment. That’s the agent itself. And we’re going to feed it meeting transcripts, Slack threads, maybe a PR description, a PRD, whatever, into the system. And then it’s going to use these plugins to write it in a format that's easy to read. With an agentic skill, you’re laying things out once; maybe you optimize it, but then it kind of stays that way. With this, you’re constantly adding it. So you have this extra layer that knows how to do consistency checks, traceability, make sure things are complete, make sure there aren’t too many hops, right? So that they can be loaded efficiently and do schema validation and such. Because each plugin will have a different set of nouns.
Why would you want to store things in a repository? This isn’t necessarily the same repository as the project. This would be another repository that multiple projects might share, or a whole set of microservices, or a set of related pieces of a larger system. It’s already reviewable. Humans can read it. They’re just markdown files. It’s attributable. You can figure out where it came from, who created it. It’s mergeable. So I could be working on one work tree, doing some investigation, and when I’m done, I create a PR. It makes the change to the main work tree, and then everyone that uses that, the next time they pull main, they’re going to see that information.
What does it buy you? Well, if a new teammate joins, they can come up to speed really quickly. They could even go in and start asking questions about the applications and how they relate. Why this was decided, why we’re using this versus that, and all that data could potentially be in that second brain, which is not only usable by agents, but is also usable by developers. And then, the day they get started, they’re not relitigating things that have already been decided because they have access to the why and the how. And more importantly, so did the agents.
What are the problems if it’s only under git? Well, most of your stakeholders are probably not going to dig through your second brain, right? Probably not. And they may not be developers, but they may already have their own tools, like Figma, Confluence, or Jira, right? So what you really want to do is keep those things in sync, but they’re not really the source of truth. The statement of truth is what’s in your git repos, right? And then everything else gets sort of updated and generated from this. That’s not to say they can’t write specs and PRDs that are later ingested. It’s just that what’s in your second brain is sort of what’s in the institutional knowledge for the agents. But you definitely want to have, if you have an architectural review board, you want to be able to update documents that they can see; you want to update roadmaps so your product manager can see them; design documents, code walkthroughs, all that. So developers that are joining the project can understand. So there’s a human view of the systems, and then there’s the internal view.
And that’s where the WikiTicket SDD comes into play. WikiTicket is basically: you either go into planning mode, or you use some sort of spec-driven development, and it’s going to convert those into that todo.jsonl file. And then from that todo.jsonl file, it’s going to output different documents like release notes, requirements, design documents, PR descriptions, et cetera. Those all sort of become automatically generated based on the tickets and the other pieces. So there are hooks that turn plans into tickets.

Typed edges preserve causality

Here is a real chain from the PKC sample bundle, with the relation names exactly as the files carry them.

The JWT decision originates_from the auth design meeting, decides the user authentication feature, informs the JWT-versus-cookie experiment, and mitigates a token replay risk. Two more concepts point at the feature from outside that chain: an Assumption that a short access TTL is fine for user experience assumes the feature, and an open Question about a distributed revoke denylist blocks it.

Read that again and notice where each edge lives. The decision file carries its own provenance and its own consequences. The blocks edge lives on the question, because blocking is the question’s claim to make, not the feature’s. The feature does carry a soft related_to link back, but the load-bearing typed edge sits on the source. The graph assembles from the outside in.

Now the point that makes the whole approach necessary.

question --blocks-> feature and feature --blocks-> question are different facts. Same two nodes. Opposite meanings. One says you cannot ship until you answer this. The other says you cannot answer this until you ship.

Embeddings have no mechanism to express that difference. Two nodes that are textually similar are near each other in vector space regardless of which one blocks which. Direction is not a property that similarity has.

This is the whole argument for typed edges in one line. Vector search retrieves what is textually similar. A graph retrieves what is structurally causal. Those are different questions, and most of the time the second one is the one you actually asked.

What you actually do with the edges

Typed edges earn their cost the first time you ask an impact question, because impact questions are directional by nature.

Take a concrete one. Someone proposes changing the access token TTL from fifteen minutes to four hours. What breaks?

A similarity search returns everything that mentions sessions and tokens. That is a reading list, and reading it is your job.

An edge walk answers the question. Start at the user authentication feature and follow its outbound edges. It satisfies a requirement whose text reads “Access tokens MUST expire within 15 minutes.” It is verified by an acceptance criterion literally named "session expires within fifteen minutes". It is related_to an assumption whose statement is that fifteen-minute tokens will not harm user experience if silent refresh works.

Three hits, and none of them is a maybe. The proposed change violates a stated requirement, breaks a named acceptance criterion, and invalidates the assumption that underpins the original decision. There is a fourth thread worth pulling too: an open question asks whether a distributed revoke denylist is needed before shipping, and the decision deferred it. My own read is that the short TTL made deferring it reasonable, and a four-hour window makes it worth reopening. That inference is mine, not something the files say.

Nothing in the first three hits came from textual similarity. They came from following typed edges and knowing what each relation means. The answer is not “here are seven documents about sessions.” The answer is “this violates a requirement, breaks an acceptance test, and invalidates an assumption, and here are the three files that say so.”

This is also the difference between a knowledge graph and a well-organized folder. A folder tells you where things are. A graph tells you what depends on what, which is the only question that matters when something is about to change.

A ContextPack is a bounded typed subgraph

There are three ways to feed an agent, and only one of them survives contact with a real graph.

A full dump overflows the budget. It also places your most important material in the middle of the window, where recall is weakest.

Vector similarity fits, and then drops causality, direction, and impact order. You get a bag of textually similar fragments. Useful for search. Bad for reasoning about consequences.

A ContextPack follows a limited number of typed edges from a root, ranks by relevance and impact, and returns only what fits. The agent receives structure rather than noise.

Let’s think about a typical use case. Let’s say, hey, I had a meeting with the architectural review board. We were going to do things one way, and they said, no, no, we want you to use… we don’t want you to use IAM. We want you to use an authentication method so that people cannot authenticate with JWT, and we don’t want to use cookies. Or maybe we do. We’re not sure yet. And you might do a spike where you try multiple implementations using multiple providers. You have the architectural review meeting, where there’s an initial design and a set of mandates and requirements. There is a decision to use JWT; maybe that’s from the ARB, that architecture review board. And then there were things that you assume because you’re a developer; maybe everything’s not spelled out, but you’re going to make some assumptions. And then you’re also implementing this feature. And as you’re implementing this feature, it may need to refer to why you picked this- what was the reasoning, right? It doesn’t make the same mistakes. It may need to know what the decision is and what the origin of that decision is. If it’s just an assumption, right? Let’s say that starts to run into a bug; is this something I can change, or was this in the ARB, right? It starts to have a provenance of where these items came from. It becomes very important for decision-making.
You could think of the OKF as writing down the relationships and reasoning between these nodes, and then the coding agent or agent being a reasoning engine, right, that uses reasoning to kind of figure this out. It’s laid out in a way that lets it use its reasoning abilities.
What are your alternatives? You could just have a bunch of markdown files and kind of scan them and dump them, or maybe read all the files from your Confluence and create some kind of mechanism, maybe a vector similarity search, but then this doesn’t have the causality, right? When you organize them in a second brain or an LLM wiki with OKF, you have these relationships. This root agent node depends on node A, and it includes node T, but it affects node D, and you can sort of define these relationships and what’s stored in these nodes. And that’s what those various plugins do. I have plugins for translating what comes out of WikiTicket. I’ve got plugins that reverse-engineer system architecture, either by design, by code, or by looking at Terraform files or whatnot. And putting that in a way that’s indexable and readable, or in data engineering concepts.
What you want to hand the agent is this sort of pack of causality, decisions, assumptions, nouns that are pertinent to what it’s doing. A noun would be like a ticket, an epic, a decision, a discovery, et cetera. And then provenance. When you’re laying out the LLM wiki, there are rules that you want to follow, just like when you lay out agentic skills so that you can read those back more efficiently. And that’s kind of baked into these various plugins.
Again, you had this one plugin, PKC. That’s the why. Meetings, decisions, experiments, assumptions, questions, requirements, features. It’s the SDLC reasoning graph, right? And then you have the SAC, which is the sort of what’s there: how is it deployed, what’s the topology? And then there’s something similar to that, but this is for Lake House: data lakes, catalogs, tables, lineage, medallion layers, schemas, semantic layers, et cetera. Things that you would need for that.
PKC is basically the adapter from the WikiTicket SDD to the knowledge graph, which is the second brain, the LLM wiki, and then you have different flavors. I’ve actually added, like, now there’s not just these two. There’s like 15 or 20 of these for different nouns. They all have different nouns because they’re looking at different aspects of the project.

Two hops, twenty nodes, fail closed

The specifics matter here, because “bounded” is only a real property if you can say what the bound is. These numbers come from the source, not from a slide.

How the pack gets built. Start at a root concept. Walk outbound edges only, because walking inbound edges drags every hub page into the pack and defeats the purpose. Rank by depth, then by type, then by title, with decisions first. Bodies are off for everything except the root, so the pack carries the shape of the neighborhood and the full text of the one thing you asked about.

How the budget gets enforced. The default is 2 hops and 20 nodes. Tiny mode is 8 nodes. The token budget is one quarter of a 128,000-token window, which is 32,000 tokens. Over budget means exit 1 and write nothing.

Two details in that paragraph are easy to miss, and both are important.

First, the node clip and the token budget are two separate limits, and both apply. Twenty small nodes and twenty large ones are the same node count and very different token counts. Enforcing only one of them gets you a pack that satisfies a number and blows a window.

Second, over budget means write nothing, not write a truncated pack. Excluded nodes are reported rather than silently dropped. This is the fail-closed rule, and it exists because silent truncation reads as “that is all there is”, which is the most dangerous lie a context pack can tell. An agent that receives eight of twenty relevant nodes with no indication that twelve were cut will reason confidently over a partial picture.

There is one more reason the ranking is deterministic rather than heuristic: the same query should produce the same pack. That makes packs diffable and cacheable. When an agent gives you a strange answer, you can look at exactly what it was given, and you can check whether that changed since last week.

What a pack actually returns

The PKC sample bundle ships a generated pack, so this is real output rather than an illustration. Rooted in the user authentication feature, abbreviated and reformatted to fit the page here:

# Context pack: /features/user-authentication.md

- Hops: 2
- Nodes: 9 (max 20)
- Generated: 2026-08-04T04:29:31Z

## Nodes (ranked)

### User authentication          Feature         depth 0
### Use JWT for session mgmt     DecisionRecord  depth 1
### Auth design discussion       Meeting         depth 1
### JWT vs session-cookie spike  Experiment      depth 1
### Short access TTL is OK       Assumption      depth 1
### Need revoke denylist in v1?  Question        depth 1
### Auth middleware              Design          depth 1
### Secure session tokens        Requirement     depth 1
### Implement user auth          TicketLink      depth 1

## Edges

- /features/user-authentication.md  -[satisfies]->  /requirements/secure-session-tokens.md
- /decisions/use-jwt-for-session.md -[decides]->    /features/user-authentication.md
- /questions/need-revoke-denylist-v1.md -[blocks]-> /features/user-authentication.md
- /assumptions/short-ttl-ok-for-ux.md   -[assumes]-> /features/user-authentication.md
...

_Nodes beyond hops/max_nodes omitted for progressive disclosure._

Four things about that output are worth pointing out.

The header is a receipt. Hops, node count, the clip it ran against, and a generation timestamp. When you are debugging why an agent said something strange, the first question is what it was actually given, and the header answers it before you read a line of content.

Every node carries its type and its depth, and the pack ships the edge list alongside the nodes. The agent does not just receive a neighborhood; it receives the shape of one. It can see that the question blocks the feature rather than merely sitting near it.

Nine nodes against a clip of twenty means this pack is nowhere near its bound. That is the normal case, and it is worth saying because the budget discussion above can leave the impression that packs are always straining against a ceiling. Most are not. The bounds exist for the ones that would be.

The last line is the fail-closed rule in a milder form. Nodes beyond the limits are announced as omitted rather than dropped in silence. The difference is between an agent that knows there is more and an agent that believes it has the whole picture.

One node is missing from that pack, and it is the most useful thing on the page. The impact walk earlier in this article found an acceptance criterion named session expires within fifteen minutes, reached from this same feature in one hop. It is not among the nine. The reason is in the header: this pack was generated on 4 August, and that acceptance criterion was added to the feature on 7 August. The pack is three days stale.

The receipt is doing its job. A pack is a snapshot with a timestamp on it, not a live view, and the timestamp is what lets you notice. It is also the reason packs get regenerated rather than cached forever.

One substrate, many context packs writing to second brain

Four packs sit on one base plugin. PKC owns the why. SAC owns what is running. DEKC owns the data plane. WikiTicket SDD owns the work log, which is the event log and fold from the first half of this article.

Underneath all four is okf-plugin, whose plugin name is okf-graph-eng. It is the OKF graph engine, and everything depends on it. Alongside it, second-brain-core carries the shared schemas, the typed edge definitions, the deterministic write helpers, and the packing engine.

Three rules keep the packs from colliding.

Each pack owns its own nouns and its own catalogs. A pack ships a schemas/okf-concepts/ directory that the validators merge at load time. PKC brings DecisionRecord, Assumption, and Question. DEKC brings lineage and medallion layers. Neither has to know the other exists.

Never write a type another plugin owns, and never invent a relation name. This is the rule that keeps the graph queryable. A relation name that only one script understands is a private convention wearing a public costume.

One repository ships manifests for every host. Claude Code, Grok Build, Codex, OpenCode, Agent Plugins 1.0, Grok Bot, and LangChain Deep Agents, all from the same source tree. Agent Plugins 1.0 is the universal format that arrived in August 2026: a root plugin.json plus skills. That matters because it means the ontology is not locked to one vendor’s plugin system. The nouns travel.

Installing the base is two lines:

claude plugin marketplace add SpillwaveSolutions/okf-plugin
claude plugin install okf-graph-eng@okf-plugin-marketplace

Everything is public and MIT licensed, at github.com/orgs/SpillwaveSolutions/repositories.

I should be honest about the timeline. wiki_ticket_sdd was created on 18 July 2026, okf-plugin on 29 July, PKC on 4 August, SAC and DEKC on 9 August, and second-brain-core on 14 August. That is under four weeks. These are working tools rather than mature ones, and the traction reflects it. My older projects have some: agent-brain has 116 stars and agent-memory has 15. wiki_ticket_sdd has 10. The second-brain plugins themselves have almost none, because the oldest of them is three weeks old.

The graph is two graphs

There is a source of confusion in this space worth clearing up, because “graph engineering” currently names two different things.

One is the control-flow graph your agents run in. Nodes are agents and steps. Edges are routing. State lives on the wire. The lifetime is one run.

The other is the knowledge graph they think with. Nodes are concepts. Edges are typed links. State lives in git. The lifetime is forever.

They look unrelated. They are not. Both are Markdown and YAML shaped. Both are git-mergeable. routes_to and depends_on are both just links in a Markdown file with frontmatter. The same packing machinery works on either one, and an agent graph can be a node in the knowledge graph that describes it.

Worth noting for calibration: Anthropic shipped both halves separately and did not put them in one artifact. The merged version is not an established industry practice with a decade of case studies behind it. It is a direction that a handful of people are building toward right now.

Closing the loop

Loop engineering and graph engineering are the same argument at different time scales

Loop engineering is run-scoped. Discover, plan, execute, verify. It is powerful, and it is temporary. Information stops dead at the session boundary.

Graph engineering is persistence-scoped shared state. Plans become nodes. Dependencies become edges. Corrections become new concepts or improved skills.

Neither replaces the other. Run a loop with no graph, and you get a system that is impressive on Tuesday and starts from zero on Wednesday. Build a graph with no loop, and you get a well-organized library that nothing ever reads.

A single loop is the smallest possible graph

One node with an edge back to itself. That is a loop, drawn as a graph.

Graph engineering does not replace loop engineering; it is the floor directly above it. And the honest version, which is worth saying out loud: a loop was always a graph. It had an orchestrator, workers, and judges from the beginning. Naming the topology just made that visible.

Karpathy’s autoresearch loop is the cleanest statement of the pattern I have seen. One modifiable file. One metric. Keep improvements, revert failures, repeat.

The insight buried in that is the one that determines whether autonomy works: it works when the stop condition is an anchor the model cannot dispute. A number the model did not produce and cannot rationalize. Everything else is a model grading its own homework.

A simple graph is just a loop where you’re just trying to improve something. But we know that these usually involve multiple agents in different roles who split up the work and have different graders, implementers, researchers, verifiers, and judges, and there are rubrics and whatnot, right? But let’s say we have this verifier, and it’s verifying pages from a report we’re generating, or maybe it’s verifying that the UI has the right layout. So where is it going to get the data for the verification? Well, if it were talking to the knowledge base, the LLM wiki, or the second brain, it could look up the wireframes and requirements. And then, when the loop finished on that screen, we could have a second agent who knows about the wireframe, the requirements, and maybe the mockups. And knows that it needs to use those to judge the implementation, right? That knowledge becomes important.
And maybe there’s something in the wireframe that just doesn’t exist in the data set, right? Then there would have to be a decision to say, well, we know we wanted that, but we don’t have any way to get it out of the lake house yet because that’s not getting landed. Let’s add a ticket to look into that, but for now, we’ll take it out. That sort of decision has to be made and stored somewhere. We would store that potentially in the knowledge base as part of this loop.
You could think of this as you’re doing discovery and you’re improving the output over time. Here is a typical loop. You have: Discover, Plan, Execute, and Verify. In the discovery phase, once you discover things, you don’t want to keep rediscovering them, right? You need a way to store them, i.e., the second brain LLM Wiki.
When you’re thinking about graph engineering, you can think about things like orchestrators and judges and doers and providing feedback through a rubric and separating the context, right? That’s one way to look at graph engineering. But if you look at tasks and their hierarchical nature when you’re doing work, right? Like, you want those agents in the graph to be able to look up their requirements, the wireframes, the reasoning, the schema, the catalogs, whatever it needs to implement that feature at that time. And you want to make it so it can load only what it needs, instead of loading the whole encyclopedia. That’s the other sort of aspect of graph engineering.
And then there’s the improvement cycle. Once you’ve made a decision, you want that sort of thing to be stored and remembered.

Two projects. Same shape.

I want to be concrete here, because everything above is architecture, and architecture is cheap. These are two projects that shipped.

Project one: extracting ontologies from PDFs.

The input was a set of PDF files with different schemas. The goal was to extract structured ontologies from them.

What made this tractable was that I knew the answers in advance. I knew what data was in those documents. That made the rubric deterministic: score the extraction against the data the documents were known to hold. No judgment call about quality, just a count of what came out correctly.

Sonnet 5 did the extraction. Fable 5 and Opus 5 built the harness and improved the prompts. The harness ran on LangChain Deep Agents with one goal: raise the score.

The structure was orchestrators, graders, and judges issuing scores against the rubric, and the loop then rewriting the prompts and the extraction program. As new requirements arrived, the old cases ran as a regression suite. That is the part that made it safe to keep changing things. A prompt change made for a new feature could not silently break an old one, because the old one still had a score. If the score dropped, keep tweaking.

Project two: scoring phone-call transcripts.

The task was grading how well an individual did their job against a playbook. Did they say the things the playbook calls for? Did they hit the goals in the conversation?

The split here is the interesting part. An LLM extracts language fragments that match the sought-after patterns. A Python script holds the rubric and produces the grade. The rubric deliberately stayed in Python and never moved into the model.

The LLM makes exactly one kind of judgment: were those language patterns actually spoken in this conversation? That is a question models are good at. It does not decide what constitutes a good call. Python does that, and it is readable, testable, and arguable in a code review.

The Claude Agent SDK runs the loop that tunes the extraction prompts for Sonnet 5.

And there is a human in the loop. A reviewer can read a transcript, decide a grade was wrong, and store that feedback. The stored feedback goes back through the same harness, and the extraction improves. This is the write boundary from earlier in this article, with a person as one of the writers. The correction is not a comment in a chat window. It is stored, and the next run starts from it.

I now use this on most projects for discovery, testing, comparing outputs, and storing meeting notes, assumptions, and decisions.

The rubric is code. The score is the anchor.

Both projects share one shape, and it is worth stating as four rules.

The rubric is deterministic code, so the LLM never grades itself. The moment a model both produces and scores its output, the score ceases to be information.

The LLM makes one kind of call: was this said, is this present. Narrow judgment calls, of the kind models handle well, feeding a scoring function that they do not control.

The score is the stop condition. It is an anchor the model cannot argue with. This is the same property as Karpathy’s single metric, and it is what makes the loop safe to run unattended.

Old cases become the regression suite. A new feature that lowers the score is a bug, not a trade-off to discuss. This turns prompt engineering from a matter of taste into an engineering discipline.

Add the human, and the cycle closes: a reviewer overrules a grade, the correction is stored, and the next run starts from it.

Day 60 is the metric, not day 4

Everybody can produce a good day-four demo. The question that matters is what the system looks like two months in, after the person who set it up has moved on to something else.

Three tests.

The fresh-clone test. A new developer immediately clones the repository and inherits the policies, skills, and conventions. Not after a week of pairing. Immediately.

Context reproducibility. Packs come from deterministic tooling rather than from somebody’s private prompt history. Two people asking the same question get the same pack.

No private glue. The practice survives without the original facilitator. No dotfiles. No shell history. No “ask Rick, he has a script for that.”

If your system passes all three at day sixty, it is real. If it fails any of them, what you have is one person’s workflow with extra steps.

Where this does not apply

I would rather you hear the limits from me than find them yourself in month three.

The most common disqualifier is the absence of ground truth. Skip the graph entirely when there is no mechanical way to judge whether the output is correct. My own read is that the same applies to the loop. Both projects above worked because I could score the result without asking a model whether it did well. Without that anchor, a loop is an expensive way to generate confident variations.

Multi-agent systems can burn 10 to 15 times the tokens of a single-agent baseline. That is a real number and it should change what you attempt. Topology is not free.

Nobody has published a production case study of the merged graph at organizational scale. I have shipped the pieces. I have not run this across a large engineering organization, and neither has anyone who has written about it publicly. Treat claims to the contrary skeptically, including mine.

A lot of the circulating material about this is not what it claims to be. The “1000x” figure is not tied to any verified publication. The multi-page PDF attributed to two Anthropic seniors is a third-party remix. The “new Anthropic graph engineering workshop” is a LangGraph session from 2024. If a claim in this space feels too clean, check the primary source.

And on ground truth itself, the strongest objection anyone has raised is Andrew Kelley’s about the Bun rewrite. One million lines of Rust in under two weeks, the full test suite green at merge, nineteen regressions discovered after, and roughly $165,000 in tokens. The honest reading is that an anchor bounds the blast radius rather than eliminating it. A green suite is not proof of correctness, it is proof that the things you thought to check still pass.

Architecture still matters more than any single model upgrade. That is the claim I am willing to defend.

Start small

You do not need a six-month platform project. You need one repository and one workflow.

Capture a single meeting. Generate one ContextPack from it. Force the artifacts into the repository from day one, and refuse to let hallway knowledge stay private. Grow the graph from real work rather than from a schema exercise, because a schema designed in advance of any content is a guess.

A reasonable pilot is twenty to fifty concepts, one owner group, and one agent use case. Small enough to abandon. Big enough to tell you something.

A concrete first week

If you want a plan rather than a principle, here is one.

Day one. Turn on the work log. Add .work/todo.jsonl and the two merge=union lines in .gitattributes. Have every agent session append its work items. Do nothing else. By Friday you will have the first artifact nobody on your team currently has: a record of what the agents actually did, in time order, that survives the sessions.

Day two. Render the roadmap from the log. This is the fold. It costs you almost nothing because the log is already there, and it converts a status meeting into a file. The value shows up the first time somebody asks “what is in flight” and the answer is a command rather than a Slack thread.

Day three. Capture one meeting into PKC. Pick a meeting where a real decision got made. Let the model propose the decision, the open questions, and the assumptions. Let the write helper commit them. Review the resulting files in a pull request like any other change.

Day four. Generate one ContextPack from that decision and hand it to an agent. Give the agent a task adjacent to the decision. Watch whether it starts from the right place. This is the first moment the investment pays back, and it is also the moment you find out whether your concepts are too thin.

Day five. Look at the diff. A week of commits to .work/ and the knowledge tree is your evidence. If a colleague can read that diff and understand what the team decided and why, the practice is working. If they cannot, the concepts need bodies rather than more schema.

Notice what is not in that week. No platform. No database. No migration. No new vendor. Five days of small commits to a repository you already have.

One question decides whether it worked:

Can a new teammate clone the repository and get value without becoming a prompt expert?

If the answer is no, the memory is still private. It does not matter how sophisticated the tooling is. Private memory is the problem you started with.

A free hour this morning: Build a Second Brain for Your AI Coding Agents

Rick Hightower — Thu, 20 Aug 2026 07:27:24 GMT

Short and time-sensitive, so I will get to it.

This morning I am giving a free hour with Packt on the material this newsletter has been circling for months. If you have been reading the loop engineering and harness engineering pieces and wondering what it looks like assembled rather than in parts, this is that.

Build a Second Brain for Your AI Coding Agents
Today, Thursday, 20 August, 11:00 AM Eastern, 10:00 AM Central, 4:00 PM UK. Sixty minutes. Free.

What the hour covers

The problem is the one I opened last week’s piece with. You and an agent solve something hard, the reasoning is sharp, the trade-offs are clear, and then the session ends. Two days later, a different agent reopens the decision you already settled. You are not losing time to a weak model. You are losing it to an architecture that throws the answer away when the session ends.

The hour walks the fix end-to-end. What a harness actually is, and where the model must never be trusted. The three kinds of memory teams collapse into one, and what that collapse costs them. An append-only event log under Git that makes agent work visible. A knowledge graph that is ordinary Markdown and YAML, reviewable as a diff. ContextPacks, which hand an agent a bounded, ranked, typed slice instead of a wall of prose. And the self-improvement cycle that makes each run better than the last.

Every tool I name is open source and MIT-licensed. There is nothing to buy to use any of it.

I do not know yet whether Packt is recording it, so come live if the slot works for you.

And the workshop on the 29th

The free hour is the architecture. It deliberately stops short of building the evaluation harness and the graders because that build is the whole point of the paid workshop nine days later.

Engineering Reliable Agentic AI Systems
Saturday 29 August. Four hours, hands-on the entire way through.

Every module ends with something you keep and can run on Monday:

A working autonomous agent loop, running on your own machine inside the first hour
A reusable evaluation harness that plans, executes, verifies, and iterates
A live research assistant built end-to-end over MCP
A production-ready architecture you can hand to your engineering org

It is a build session. You will be typing for most of four hours, and you should be comfortable in a terminal. You do not need prior experience with agent frameworks.

Because you read this newsletter, the code RICK40 takes 40 percent off. It expires on Friday, 28 August, the day before the workshop, so there is no discount on the day.

Reserve a spot

Add RICK40 in the promo code box on the registration form to take 40 percent off.

That is it

Come this morning if the hour is free. Come on the 29th if you want to build the thing rather than hear about it. And if neither works, the repos are public either way, and I will keep writing this up here.

Thanks for reading.

Rick

About the Author — Claude Certified Architect

Subscribe to Rick’s newsletter to see videos and guides.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Free subscribers typically receive access to the full versions of paid articles after one to two months.

Share Hightower's AI Harness Engineering

If this helped you, please consider subscribing to my Substack newsletter.

Thank you for reading and for helping the work reach more people.

Subscribe now

Leave a comment

Share Hightower's AI Harness Engineering

Claude Managed Agents XI: One Agent, Many Users, Zero Secret Store

Rick Hightower — Wed, 19 Aug 2026 05:58:42 GMT

How to make a single AI agent act on behalf of hundreds of customers without building a token vault or passing credentials on every call

In this article, you will learn how vaults solve multi-tenant authentication for AI agents. We cover the two-object model of vaults and credentials; the difference between static_bearer and mcp_oauth credential types; why credentials validate at runtime rather than at creation; the three constraints that shape your design; and how rotation and revocation reach live sessions. By the end, you will know how to make one agent definition serve every customer cleanly.

You built an agent. It connects to a billing system, reads files, runs code, and produces a report. It works beautifully for one customer. Then the second customer signs up, and the third, and the hundredth, and you hit a wall that has nothing to do with your agent’s logic.

Each customer’s data is reached with that customer’s own credential. Your agent is one product, one configuration. The credential changes every time it runs. So how do you connect them?

The obvious answer is the wrong one. You set up a secret store, look up the correct token on request, and inject it into every call yourself. Congratulations: you are now running credential-management infrastructure on the side, which is exactly the kind of plumbing that a managed agent platform was supposed to spare you. There is a cleaner answer, and it is the subject of this article. It is called a vault.

The problem credentials actually pose

Start with the shape of the problem, because it is more specific than “the agent needs auth.”

Credentials belong to a different layer than the agent. The agent is your product. You define it once. Its configuration, tools, and declared MCP servers: all of that is stable and shared across all customers. The credential, by contrast, is a runtime detail. It depends entirely on who the agent is acting for in a given run.

Mixing those two layers is the trap. If the agent definition holds a secret, you cannot reuse it across customers. If you pass a token on every call, you are transmitting secrets constantly, and you still need somewhere to store them. Either way, you have entangled “the agent you built” with “the user it acts for,” and untangling them later is painful.

A vault keeps them separate. A vault is a per-end-user collection of credentials that you reference by ID when you create a session. You do not run your own secret store. You do not transmit tokens on every call. And you never lose track of which user the agent acted on behalf of.

The mental model is a clean separation of concerns. The agent is defined at the agent level. The user is supplied at the session level. The vault is what joins them. Hold that picture, and the rest of the API falls into place.

A vault is a user; a credential is a key that user holds

The model is two nested objects, and the nesting deliberately mirrors your own data.

A vault represents one end user. You create it with a display_name and, importantly, optional metadata that maps it back to your own user records. That metadata is the hook that keeps your system and the agent platform in sync. You stamp the vault with your internal user ID, so you always know whose vault it is later.

vault = client.beta.vaults.create(
    display_name="Acme Corp (customer)",
    metadata={"external_user_id": "cust_8841"},
)
print(vault.id)  # "vlt_01ABC..."

In TypeScript:

const vault = await client.beta.vaults.create({
  display_name: "Acme Corp (customer)",
  metadata: { external_user_id: "cust_8841" },
});
console.log(vault.id); // "vlt_01ABC..."

A credential resides in a vault and is the key for a specific service. Its defining property is that each credential binds to a single mcp_server_url. At session runtime, when the agent connects to an MCP server, the API matches that server’s URL against the active credentials on the referenced vault and injects the right token.

So a credential is not “a token.” It is “the token for this specific server, held by this specific user.” The URL is part of its identity, not an afterthought.

If you are a paid subscriber, thank you. Your support makes this work possible.

If you are a free subscriber and find these articles useful, please consider upgrading. A paid subscription is $80 per year or $8 per month.

Share Hightower's AI Harness Engineering

Hightower's AI Harness Engineering

Langfuse II: Traces Are a Typed Tree, Not a Folder of Logs

A span with a model name is not a generation. The Langfuse data model is how cost, judges, and experiments find the right row.

Three nouns that make Langfuse traces queryable

Observation types: how cost and evals find the right row

How a Langfuse trace actually leaves your process

Three ways to create an observation

Bring Your Own Agent: Loop Engineering and the Harness Around It III

The same ticket enhancer as a Claude Code plugin, a Codex plugin, and a Grok Build plugin, plus four places to run it when nobody is watching

Skill version and Python version

Keep what matters in the deterministic layer

AI News, Volume 45: The Frontier Gets Guardrails: Mythos 5.1 and Fable 5.1!

AI News, Volume 45: The Frontier Gets Guardrails

Front Page Summary

Today’s AI News Stories

Anthropic splits its frontier into broadly available and restricted capability tiers

OpenAI’s Astra turns frontier cyber capability into an access-control problem

Anthropic proposes customer-controlled monitoring for enterprise frontier agents

Google teaches Gemini to watch video by searching instead of sampling everything

Anthropic proposes a common language for agents that operate physical equipment

DeepMind makes frontier evaluations double-blind

OpenAI moves more of the agent loop into GPT-5.6 and Responses

Microsoft publishes a production path from local agent to hosted and evaluated system

Gemini CLI and Copilot CLI tighten the local agent trust boundary

Grok Bot connects its persistent agent to X

Opinion: What This Means for Harness Engineering

Closing Note

References

About the Author — Claude Certified Architect

The Second Runtime: Loop Engineering and the Harness Around It II

I moved the same ticket enhancer from the Claude Agent SDK to LangChain Deep Agents. The files that changed reveal where the harness really lives.

What has to survive

One table, three runtimes

The broken port

How Deep Agents enforces the contract

AI News, Volume 44: Agents Get Browsers—and Consequences

AI News, Volume 44: Agents Get Browsers—and Consequences

Front Page Summary

Today’s AI News Stories

Anthropic assembles the production agent stack

OpenAI’s incident report shows how agents can invent their own control plane

Codex turns the terminal into a task-control system

OpenAI makes websites and workflow events native agent inputs

GitHub moves coding agents into the team conversation—and brings policy with them

Microsoft combines durable hosted agents with a multi-channel edge

Perplexity moves the agent runtime onto user-owned hardware

Salesforce turns CRM rules into Claude’s action layer

Google packages vertical agents with policy, evidence, and cost controls

Temporal makes the agent loop a durable workflow

Opinion: What This Means for Harness Engineering

Closing Note

References

About the Author — Claude Certified Architect

Python Owns the Loop: Loop Engineering and the Harness Around It

Python Owns the Loop

Loop Engineering and the Harness Around It: How to build a production loop in Python: the model drafts and grades, software holds every transition

Loop engineering and the harness around it

Four properties of one iteration

Inner ReAct, outer control

What breaks when the governor is off

The Second Brain Graph Is the Spec and much more

Why does this exist?

What the stack claims

Engineering Reliable Agentic AI Systems: Workshop

Why the graph, not the prompt

Two amnesias

The layer cake

A read-path optimizer, not a product

Project Knowledge Capture

Docs drift

AGER is not a runtime

Compile as a product

SAC, DEKC, and the rest

How the pieces actually compose

Where things stand today

What follows

OKF is how you stop treating knowledge as prompt stuffing. AGER is how a multi-agent system stops living as folklore. The rest is noun ownership and the discipline not to invent a rel.

Conclusion

Related Reading

Related Substack pieces (same thesis: second brain, graph engineering, loops, plugins, visible work):