Welcome to Edition #39 of a newsletter that 16,000+ people around the world actually look forward to reading.

🎁 Take the 5-minute sample Claude Code Architect exam to test your Claude Code skills.

🎁 Free course on Claude Code

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing. This isn’t that. You’ll find raw, honest, human insight here, the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

🎓 Here’s my Maven’s top rated course on Forward Deployed Engineering

Join today with a special discount, here

TL;DR: In 2026, a voice agent is one of the strongest bets in applied AI, wherever the channel already fits: phone-native support, hands-busy work, high-volume operations. But the build turns on a single constraint.

You have roughly 700 to 800 milliseconds per conversational turn before it stops feeling human, and every architectural decision, cascading versus speech-to-speech, open versus closed, build versus buy, is a choice about where you spend that budget.

Match the architecture to your latency math, and the bet to your use case, not to the hype.

What It Takes to Build Voice AI: The 800-Millisecond Budget Behind Every Voice Agent

Introduction

Voice is having its moment. Call a business today and you may not be sure whether the thing answering is a person. Demos of agents that book appointments, screen calls, and handle support are everywhere, and after a decade of assistants that mostly set timers and played music, it finally feels like the interface has arrived.

That is exactly why the mistakes are getting expensive.

A team we later worked with at Traversaal opened their first voice AI meeting with a confident question: “Should we use ElevenLabs or OpenAI for this?” They had a vendor shortlist and a sprint planned. But look closely at the question. ElevenLabs is a text-to-speech company; OpenAI’s Realtime API is a full speech-to-speech platform. They were weighing one slice of the pipeline against an entire architecture, the tell that they were choosing tools before they understood the problem. What they didn’t have was a latency budget. Six weeks later they had a beautiful-sounding agent that felt like talking to a loading spinner. Needless to say, that version never shipped.

July 2026 is a genuine inflection point, and the capital knows it. Analysts put the broader conversational AI market north of $40 billion by 2030, with voice agents its fastest-growing slice, and most large enterprises now report deploying or piloting voice AI in customer operations. The tooling caught up to the ambition: OpenAI’s Realtime API, Google’s Gemini Live, and a maturing open-source stack around Whisper, Kokoro, and LiveKit have made production-grade voice agents buildable without a dedicated audio-engineering team.

But the money is running ahead of the trust. In consumer surveys tracked from late 2025 into 2026, preference for a human agent climbed to around 85 percent while preference for AI slipped to 5, frustration rose, and nearly a third of people said they would hang up the moment they realized they were talking to a bot. The strange part: in blind tests, most consumers can no longer reliably tell an AI voice from a human one.

The voices are good enough; the trust is not. Add Alexa’s retreat from ambient computing, years of IVR-hell disappointment, and rising regulatory scrutiny of voice biometric data, under GDPR and a wave of new US state laws, and these architectural choices now carry legal and reputational weight, not just UX weight.

Think of a voice turn like a restaurant ticket. The kitchen has a fixed window to plate the dish before the experience degrades. Every pipeline stage is a cook; every millisecond is a resource.

The framework governing all of it is the Turn Budget, and every architectural decision in a voice agent is a latency allocation decision within it.

Key Takeaways

📈 The money is ahead of the trust, Voice AI is a multi-billion-dollar market and most large enterprises are deploying, yet consumer preference for human agents is climbing, not falling. The engineering job is to earn trust, not just sound human.

⏱️ The turn budget is everything, Every architectural decision in a voice agent comes down to how you slice roughly 800 milliseconds across pipeline stages before conversation feels broken.

🔗 Cascading pipelines trade speed for control, Separate STT, LLM, and TTS components give you stage-level visibility and guardrails, but latency accumulates fast at every handoff.

🎙️ Speech-to-speech removes your safety nets, Native audio models like OpenAI’s gpt-realtime and Google’s Gemini Live cut latency but eliminate the checkpoints where you catch errors, enforce compliance, or inspect what the model heard.

🧩 Open source wins some slices, not all, Whisper and Kokoro are competitive for transcription and synthesis, but self-hosting carries real infrastructure costs that closed APIs often beat on total cost of ownership at moderate scale.

📵 Voice earns its keep in specific contexts, Hands-busy workflows, phone-native channels, and accessibility use cases are where voice makes sense; anywhere a user can comfortably type, voice adds friction.

⚠️ Failure modes are the honest story, Noisy-environment transcription errors, barge-in failures, and PII exposure mean the real question in 2026 isn’t whether you can ship a voice agent, but whether the use case justifies the risk.

What is voice AI agent architecture, and why does latency define everything?

A voice AI agent architecture is the ordered sequence of real-time processing stages, detection, transcription, reasoning, and synthesis, that must complete a full cycle within roughly 800 milliseconds before human conversation feels broken.

Read more

Welcome to Edition #38 of a newsletter that 16,000+ people around the world actually look forward to reading.

🎁 Take the 5-minute sample Claude Code Architect exam to test your Claude Code skills.

🎁 Free course on Claude Code

🎓 Looking to deepen your knowledge as a developer and pass the Claude Code Certification? Here’s your opportunity!

Join us on July 23rd for a three-week immersive Claude Code Course and ability to take the certification through our partner network.

Sign up today

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing. This isn’t that. You’ll find raw, honest, human insight here, the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

Subscribe now

If you’ve ever nodded along in an architecture review while having absolutely no idea what just happened, this guide is the one you should have read first.

Introduction: The PM who accidentally made an infrastructure decision

Here’s a scenario that played out at a dozen Series B AI startups in the past year. A product manager uses Claude Code to prototype a customer-facing document summarization feature over a weekend. The demo is clean, the output looks sharp, and by Monday morning it’s sitting in a staging branch.

By Wednesday, the engineering lead is asking uncomfortable questions:

Q. How does it handle PII in uploaded documents?

Q. What happens when the document exceeds the context window?

Q. Why is there no retry logic for when the model API times out?

The PM doesn’t have answers, not because they were careless, but because they didn’t know those were questions they were supposed to ask.

This is the structural reality of full stack for product managers in 2026.

The tools have changed faster than the mental models. Claude Code, OpenAI Codex CLI, and agentic harnesses like LangGraph have collapsed the time it takes to go from idea to running code, but they haven’t collapsed the complexity underneath it. Someone still has to own the architectural choices. Right now, PMs are making those choices without realizing it, one AI-generated file at a time.

The numbers back this up. AI-native teams are shipping three to five times faster than traditional engineering teams, according to the 2025 State of DevEx report from DX (Source: https://getdx.com/research/state-of-devex-2025). But velocity without architectural literacy is how you accumulate technical debt that paralyzes the team six months later. The code ships fast. The consequences arrive slowly.

This guide won’t turn you into an engineer. What it does is give you the mental models to understand what your engineers are actually arguing about, so you can make better decisions, ask smarter questions, and catch the moment when the AI-generated code is quietly creating a problem you’ll own in Q3.

🔑 Key Takeaways

🖥️ Frontend is the contract layer: Every UI decision your team makes creates a promise to users that the backend then has to keep, so PMs who understand the gap between the two ship fewer broken features.

⚙️ APIs are your product’s nervous system: When engineers argue about API design, they’re really arguing about how hard it will be to change your product later, which makes that conversation your business.

🤖 AI backend is not regular backend: LLM-powered features have a completely different failure mode than traditional software, where outputs drift invisibly over time instead of throwing an obvious error.

🗄️ Your data model is a product decision: Choosing how data is structured and stored shapes what you can build next, meaning a database schema choice made in week one can box you in for years.

🚀 “Production” means real consequences: Shipping to production isn’t flipping a switch; it’s exposing real users to real risk, and understanding environments, rollbacks, and where your code actually runs is how you manage that risk without freezing your team.

📡 Observability is now a PM responsibility: If you’re shipping AI features without logging what the model actually does in the wild, you have no way to know when quality silently degrades on your users.

1. The full stack mental model: what it actually is and why the layers matter

1.1 Why “full stack” is a systems metaphor, not just a job title

When engineers say “full stack,” they mean a complete software system, everything from the pixel a user taps on their phone to the database row being written on a server somewhere. For PMs, the value of this metaphor isn’t technical precision; it’s situational awareness. Every product decision you make touches at least one layer of that stack, and understanding which layer tells you how expensive, how risky, and how reversible that decision is.

Think of it as an iceberg. Users see roughly ten percent of the system, the interface, the interactions, the words on the screen. That visible surface depends entirely on the ninety percent they never see: the servers processing their requests, the databases storing their data, the authentication systems validating their identity, and increasingly, the AI pipelines generating their content. When a PM makes a UI decision, they’re also making assumptions about the entire iceberg beneath it. The mental model of the full stack is what lets you hold those assumptions consciously, rather than accidentally.

Technical literacy for product managers has become urgent in 2026 not because PMs suddenly need to write code, but because PMs are now generating code through AI tools and deploying it into systems where the iceberg is large, and largely invisible.

1.2 The three major layers: a PM-readable map

The stack breaks into three practical layers that every PM should be able to navigate without jargon.

Frontend is everything the user directly interacts with, the visual interface, the buttons, the forms, the animations. It runs in the browser or native app on the user’s device. When a user clicks “Generate Report,” the frontend registers that click and makes it look like something is happening.

Backend is where the actual work gets done. Business logic lives here: the rules that decide whether a user has permission to generate that report, the code that fetches the right data, the service that formats the output. The backend runs on a server, not on the user’s device, which is why it can be trusted with sensitive operations.

Infrastructure and data layer is everything that makes the backend possible, the databases that store data persistently, the cloud services that run the servers, the networking that moves data between components, the caches that make retrieval fast.

AI backend is now a fourth emergent layer that sits between backend and infrastructure in agentic systems. It includes the LLM API calls, the prompt engineering layer, the retrieval systems, and the output parsers. We’ll go deep on this in Section 4 because it has a completely different failure profile than the other three layers.

These layers communicate through APIs, which is why API design is so central to every product conversation that feels like it’s “just engineering.”

Illustration: The Production Gap

1.3 Why the boundaries between layers are where product bugs live

Here’s something no introductory tech guide tells PMs: most production bugs don’t live inside a single layer. They live in the contracts between layers, the assumptions each layer makes about what the other will deliver.

A frontend that assumes data will always arrive in a specific format breaks the moment the backend changes its response structure. A backend that returns an HTTP 200 OK status even when it hit an internal error breaks the frontend’s ability to handle failures gracefully. An AI layer that silently returns an empty string instead of throwing a catchable error breaks the entire user experience with no visible alarm. None of these failures is catastrophic inside a single layer, but all of them are catastrophic at the boundary.

PMs who understand these boundaries write better acceptance criteria and catch more problems during spec review, before the bug is ever coded. When your spec says “show the user their report” and doesn’t describe what happens when the backend is slow, the AI returns nothing, or the data is malformed, that’s not an incomplete engineering decision. That’s an incomplete product decision.

2. Frontend explained: what PMs actually need to know

2.1 What frontend is (and what it isn’t)

Frontend is everything that runs in the user’s browser or native application. In a web context, that means three things working together: HTML provides the structure and content (the bones), CSS provides the visual styling (the skin), and JavaScript provides the behavior (the muscles). When a user sees a dashboard, clicks a button, watches a spinner appear, or reads generated text streaming in word by word, all of that is frontend.

The most important distinction for PMs is between static frontend and dynamic frontend.

Static frontend is content that doesn’t change based on who’s looking at it, a marketing landing page, a documentation site, a pricing page. It can be pre-built and served from a global cache, which makes it fast and inexpensive.

Dynamic frontend is content that’s fetched and assembled based on who the user is and what they’re doing, a personalized dashboard, a feed, an AI-generated response. Dynamic pages are powerful but introduce latency, authentication complexity, and a hard dependency on the backend being available and responsive.

This distinction comes up in roadmap decisions constantly. When you’re deciding whether to build a feature into the main application or as a standalone marketing page, you’re making a static-versus-dynamic call with real cost and performance implications.

2.2 Frameworks, rendering, and why your engineering team’s choice affects your roadmap

As of mid-2026, the dominant frontend frameworks are React, Next.js (built on React with additional server-side capabilities), and Vue. You don’t need to know how to write code in any of them. What matters is why the choice affects your product.

Framework choice affects three things PMs care about directly: how fast your team can ship new UI features, how SEO-friendly your product is out of the box, and the size of the engineering talent pool you can hire from.

React remains the most widely adopted, with Next.js capturing the majority of new production projects in 2025–2026 due to its flexibility between rendering modes (Source: https://survey.stackoverflow.co/2025/).

The rendering mode decision is worth understanding.

Server-side rendering (SSR) means the server assembles the complete HTML page before sending it to the browser, like a restaurant that plates your food in the kitchen before it reaches your table.

Client-side rendering (CSR) means the browser receives a minimal HTML shell and a bundle of JavaScript, then assembles the page itself, like a meal kit delivered to your table that you put together.

SSR is generally faster for first load and better for SEO; CSR can be more interactive once loaded. Many modern frameworks, including Next.js, let you mix both. When your engineers argue about which to use for a new feature, they’re making decisions about SEO performance, initial load time, and engineering complexity, all of which affect your product metrics.

Illustration: The Production Gap

Read more

Welcome to Edition #37 of a newsletter that 15,000+ people around the world actually look forward to reading.

🎁 Take the 5-minute sample Claude Code Architect exam to test your skills

🎓 Looking to deepen your knowledge as a developer and pass the Claude Code Certification? Here’s your opportunity!

Join us on June 5th for a three-week immersive Claude Code Course

Sign up today (25% OFF, while seats last)

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing. This isn’t that. You’ll find raw, honest, human insight here — the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

Subscribe now

If your team uses Claude Code, Copilot, or any AI coding tool, you’re already inside a feedback loop where the AI improves based on what happened last time. Most teams don’t know it has a name, a failure mode, or a fix.

Introduction: The Self Improving Architecture (RSI)

Here’s a scenario most teams using AI tools will recognize.

An engineer notices Claude keeps writing overly verbose PR descriptions. They update CLAUDE.md with a new instruction. Fixed. A PM notices the AI coding assistant keeps flagging false positives in security reviews. The team refines the prompt. Better. A developer tweaks their Copilot skill file after a bad refactoring session. The next run goes cleaner.

Each of those is Recursive Self-Improvement (RSI).

The agent ran a task. Someone evaluated the output. A modification was made to how the agent operates. And the next task ran on the updated version. That’s the whole definition: three steps, satisfied regardless of whether there’s code behind it or just a human making a judgment call.

Now zoom out one level. What if instead of a human making that judgment call, the agent does it itself? It runs a task, scores its own output against a rubric, proposes a better version of its instructions, and overwrites them before the next task starts.

That’s Recursive Agent Optimization (RAO): the automated version of what your team is already doing manually. It’s shipping in production today inside Claude Code pipelines, LangGraph agents, and Copilot-powered workflows, usually without anyone on the team recognizing it as a distinct pattern with its own failure modes.

This matters differently depending on where you sit.

If you’re a product manager, the risk is behavioral drift.

An agent silently rewriting its own instructions can change how it behaves on your product (which tools it prioritizes, how it evaluates quality, what tradeoffs it makes) without anything in your deployment pipeline showing a change. No PR, no diff, no change log. The agent your team reviewed last week may not be the agent running this week.

If you’re an engineer, the risk is more specific: the missing third component. Most implementations that approximate RAO have a task loop and an evaluation step. Almost none have a versioned state manager with rollback. Without it, when the self-modification loop produces a bad update (and it will), there’s no recovery path and often no signal that anything went wrong until it surfaces downstream. This guide covers both.

• Sections 1–2 build the taxonomy and define what RAO is — conceptual, any reader.

• Section 3 works through the Gödel Agent as a concrete reference architecture.

• Sections 4–5 cover guardrails and real production use cases.

• The final section on Claude Code is practical for anyone building with it today, no prior framework experience required.

🔑 Article Key Takeaways

🔁 Recursive Self-Improvement (RSI) is already happening on your team — Every time an engineer updates CLAUDE.md, refines a skill file, or tweaks a prompt based on what worked last time, that’s RSI. The question isn’t whether you’re doing it. It’s whether it’s running intentionally or quietly on its own.

🧠 There’s a version that runs automatically — Recursive Agent Optimization (RAO) is when the agent scores its own outputs and rewrites its own instructions between tasks, without a human in the loop. Teams are shipping this today in Claude Code and LangGraph pipelines, usually without recognizing the pattern.

⚠️ The dangerous version is the invisible one — The RSI that dominates safety discourse (AI modifying its own model weights) is visible and rare. The version actually in production — agents silently rewriting their own routing logic and tool priorities — is invisible and common.

🏗️ You already have most of the infrastructure — CLAUDE.md is a mutable policy file. Skills are version-controlled routing targets. Hooks are the evaluator layer. Claude Code users are one step from a proper RAO loop.

🔬 PMs and engineers face different risks — Engineers risk silent degradation when a self-modifying loop optimizes the wrong metric. PMs risk shipping on agents whose behavior has drifted from what was reviewed and approved, with no change log and no diff.

🛠️ Three components determine whether your loop is safe — A performance evaluator, a self-modification generator, and a versioned state manager with rollback. Most teams have the first two. The third is the one that makes recovery possible.

Section 1: The RSI spectrum

1.1 What RSI actually means

RSI enters the technical literature through two distinct lineages.

Jürgen Schmidhuber’s Gödel Machine (2003) proposed a formal framework in which a general problem solver could rewrite any part of its own software, including its learning algorithm, provided it could construct a proof that the modification would improve future performance.

Ben Goertzel’s early AGI writing used RSI to describe the hypothetical inflection point where a system’s self-modifications begin compounding faster than human oversight can track.

Both formulations are weight-centric and capability-explosion-oriented. Source: https://arxiv.org/abs/cs/0309048

That framing has collapsed a rich engineering concept into a single dramatic scenario. A more useful working definition: RSI is any system that

(a) evaluates its own performance on a task or class of tasks,

(b) generates a modification to itself or its operating procedure based on that evaluation, and

(c) applies that modification before or during subsequent task execution such that future task performance is influenced by the self-generated change.

None of these three steps require weight modification. An agent that rewrites its system prompt after scoring its own tool calls satisfies all three criteria.

This definition is broad by design, it needs to capture the engineering reality. RSI is a structural property of a system’s feedback loop, not a claim about the magnitude of the self-modification.

A system that nudges its own evaluation rubric by five percent after each task cycle is doing RSI. A system that overwrites its own reasoning procedure is also doing RSI. Treating these as categorically different, one “just prompt engineering,” the other “dangerous self-improvement”, is the conceptual error that leaves production pipelines unguarded.

1.2 The five-level RSI taxonomy

Five levels. They’re not a risk ranking on their own (reversibility and scope matter more than position), but they give you a common vocabulary when your team is arguing about what you’re building.

Level 0, Static Agents are the baseline: a standard ReAct or tool-calling loop in which the agent selects actions and generates outputs but doesn’t modify any aspect of its own operating procedure. Every call to agent.run() uses the same prompt, tools, and routing logic as the previous call. No RSI occurs.

Level 1, Prompt-Level Recursion is the most common recognized form: an agent that rewrites its own system prompt, few-shot examples, or instruction block based on feedback from a previous run. DSPy’s automatic prompt optimization and Google’s OPRO framework both formalize this pattern. The agent evaluates its outputs, generates a better instruction, and overwrites the previous instruction before the next invocation. Reversibility is high, you can overwrite the prompt again, but prompt injection and instruction drift are real failure modes.

Level 2, Scaffolding-Level Recursion is where most production RSI actually lives, largely unrecognized. The agent modifies not just its instructions but its structural operating logic: tool selection prompts, memory retrieval queries, sub-agent routing rules, or evaluation criteria. A LangGraph orchestrator that scores its own routing decisions after each task cycle and rewrites its routing prompt before the next graph.invoke() call is a Level 2 system. The code can fit in a single Python file. The failure modes are substantially more dangerous than Level 1 because the modification surface is larger and degradation is harder to detect.

Level 3, Training-Data-Level Recursion moves into territory currently occupied by AI labs and advanced research teams. The agent generates its own fine-tuning data, preference pairs, or RLHF signals fed into a downstream training run. AlphaCode 2 and subsequent DeepMind coding pipelines use this: the system generates and evaluates synthetic problem-solution pairs to improve the next model checkpoint. The modification takes effect in a subsequent training run rather than immediately, which makes it lower-urgency but harder to reverse once the new model is deployed. Source: https://deepmind.google/discover/blog/competitive-programming-with-alphacode

Level 4, Weight-Level Recursion is the scenario that dominates safety discourse: an agent directly modifies its own model parameters at runtime, without a separate training pipeline. As of mid-2026, this remains confined to experimental frameworks and frontier lab environments. It requires substantial compute infrastructure to implement, which is precisely why it’s the most visible and, paradoxically, the least imminent threat.

1.3 Why the middle layers are the practitioner blind spot

The engineering community has concentrated its safety attention on Level 4 because it matches the AGI narrative: dramatic, irreversible, compute-intensive. But Levels 2 and 3 are where the actual production action is, and where the safety discourse has the least coverage.

The asymmetry is structural. A Level 4 system requires you to run a training loop, manage model checkpoints, and execute a new deployment, all visible in your infrastructure, all natural audit points. A Level 2 system requires a few hundred lines of Python, a JSON config file to store the current routing prompt, and an LLM call that scores the previous task cycle. It ships in a sprint and looks like normal prompt engineering to a code reviewer who hasn’t framed it as RSI.

Here’s a concrete example running in production at multiple companies right now. A LangGraph agent manages a code review pipeline. After each review cycle, it calls an LLM with the task history and asks: “Which tools did I call? Which contributed useful signal? Rewrite the tool-routing instructions to prioritize the most useful tools.” The output overwrites a prompt variable passed into the next graph.invoke() call. This is Level 2 RSI.

The code looks innocuous:

# What engineers read as: "just dynamic prompting"
routing_prompt = load_routing_prompt()  # reads a JSON config key

result = graph.invoke({"task": task, "routing_prompt": routing_prompt})

# "adaptive step" — what engineers don't recognize as RSI
new_prompt = llm.call(
    f"Previous routing: {routing_prompt}\n"
    f"Task history: {result['history']}\n"
    "Rewrite the routing instructions to improve performance."
)
save_routing_prompt(new_prompt)  # overwrites the config key, no prior version stored

What makes this Level 2 RSI rather than “dynamic prompting”: the modification target is the agent’s structural operating logic (tool routing), not just its natural language output. The modification persists across task boundaries. And the three safety components are all absent: no evaluator quality check before the rewrite, no version history, no rollback.

It’s structurally indistinguishable from “adaptive prompting” to the engineer who wrote it, but it has no rollback logic, no stopping condition, and no audit log of what the routing prompt looked like three cycles ago.

The issue isn’t ignorance, it’s framing. Engineers who would immediately add version control and rollback logic to a system they labeled “self-modifying” are shipping that same system under the label “adaptive prompting” without a second thought. Naming this correctly isn’t semantic pedantry; it’s the precondition for applying the right engineering discipline.

1.4 Level-by-level comparison

Read more

Welcome to Edition #36 of a newsletter that 15,000+ people around the world actually look forward to reading.

🎓 Read my free book: Claude Code: A Practitioner’s Approach

Access free courses

Most developers enable prompt caching and see modest savings, the ones hitting 90% reductions are doing something structurally different that almost nobody talks about.

Why You’re Not Getting Claude’s 90% Cost Savings (And How to Fix It)

Subscribe now

Introduction: the token cost problem that’s now a board-level conversation

By mid-2026, Claude Code is inside CI/CD pipelines, code review systems, and multi-agent workflows at enough companies that token costs are showing up in budget spreadsheets rather than expense reports. Engineers who used to say “it’s not that much” are now getting questions from finance.

The math is real. Claude Sonnet 4 runs ~$3.00 per million input tokens at standard rates. Cache reads are $0.30, one-tenth the price. (Source: https://www.anthropic.com/pricing) A team burning five million input tokens per day on Sonnet spends $15,000 at standard rates. Shift 80% of that to cache reads: $3,500. That’s an $11,500 daily difference, or roughly $4M annualized, from one structural change.

The problem is that almost nobody is actually hitting that. Threads on r/ClaudeAI, the Anthropic Discord, and Hacker News keep surfacing the same thing: “I enabled caching, I see cache_creation_input_tokens in the response, but my bill barely moved.” Anthropic says 70–90% savings. Developers are seeing 5–15%.

The gap is structural, not a bug. Prompt caching is not a setting you flip.

It only works when your prompts are built in a specific way, stable content before dynamic content, above a token threshold, with cache breakpoints in the right places. This guide covers both sides: the Claude Code CLI techniques that cut context volume before any API call happens, and the API-level techniques that determine what you actually pay per token.

First, a necessary distinction. This guide covers two separate problems:

If you’re on the Claude API (building products, running agents, CI/CD pipelines), you pay per token. The 89.3% combined saving from API techniques in Sections 1–6 is what cuts your bill.

If you’re using Claude Code CLI (subscription or enterprise), you’re not paying per token, but you are paying with response quality. Every token in your context window is context Claude has to reason over. Bloated config files and noisy directories don’t inflate your invoice; they dilute Claude’s attention, producing generic answers and missed context. The techniques in this section fix that. For API users who also use Claude Code CLI, these optimizations reduce the base token volume before API-level techniques apply.

These are different problems with different solutions. The benchmark numbers measure different things and don’t add together into one figure.

🔑 Key takeaways

🗂️ Two tracks, two different problems: Claude Code CLI users pay with response quality (context bloat = worse outputs); Claude API users pay with money. The techniques are different, the numbers aren’t comparable.

📋 20,000–30,000 tokens load before you type anything — system prompt, CLAUDE.md, memory, MCP tool names, and skill descriptions all enter context at session start. This is your highest-leverage optimization surface.

✂️ CLAUDE.md under 500 tokens, path-scoped rules for the rest — strip CLAUDE.md to only what Claude can’t infer from code (91.9% context reduction); move domain-specific rules to .claude/rules/ with paths: frontmatter so they’re invisible until needed (41% overhead reduction documented).

🔌 MCP servers add 10,000–20,000 tokens of silent overhead per session — every connected server loads its tool schema into every request by default; ENABLE_TOOL_SEARCH defers schemas until needed and can recover 50,000–70,000 tokens in heavy MCP setups. Connecting or disconnecting MCP mid-session also wipes your entire prompt cache.

🪝 Hooks filter noise before Claude sees it — a PostToolUse hook on Bash commands can compress a 10,000-line build log to a 200-line error summary before it enters context; the RTK open-source tool automates this and reports 80–99% reduction on build/test output.

💰 Claude API: four techniques combined save 89.3% — model routing (77.1%), prompt caching (71.5%), multi-turn caching (63.2%), and output budgeting (56.8%) are separate levers that stack. None of them work correctly without deliberate prompt structure.

Track 1: How Claude Code fills its context window, what loads before you type anything

Most people treat Claude Code’s context window as a black box. It isn’t. Claude Code loads context in a fixed order at session start, and you can see exactly what’s in there. Here’s what enters the window before you type anything:

Total base overhead before you type a single character: roughly 20,000–30,000 tokens on a clean session. A GitHub issue (#52979) confirmed a simple “hi” prompt consumed ~31,000 tokens. This is the floor you’re working with.

Claude Code does not proactively crawl or index your project.

Files only enter context when: referenced in CLAUDE.md or memory (loaded at startup), you explicitly ask Claude to read them, or Claude decides to search during a task. This means the startup layers, CLAUDE.md, memory, MCP, skills, are where most waste accumulates and where the highest-leverage fixes are.

Run /context at any point in a session for a live breakdown by category with optimization suggestions. Run /memory to see exactly which CLAUDE.md and memory files loaded at startup.

CLAUDE.md: cut to under 500 tokens

Every Claude Code session loads your CLAUDE.md into every request. It’s the most expensive single file you control, paid on every turn, forever. The token-optimizer benchmark compared a 3,847-token CLAUDE.md (generated by letting Claude document everything it encountered during onboarding) with a 312-token version stripped to only what Claude cannot infer from the code itself. Result: 91.9% context reduction, no quality regression.

Cut anything Claude already knows from training: Next.js routing, TypeScript syntax, React patterns. Cut team rosters, meeting schedules, contact info, generic preambles (”you are a helpful assistant”). Cut FAQs Claude can’t act on. Cut anything a new developer could figure out by reading the code for 20 minutes.

Keep non-obvious build and test commands. Keep architecture decisions that go against framework defaults. Keep project-specific constraints. A useful test: would this genuinely surprise an experienced developer new to the repo? If not, it probably shouldn’t be there.

Target: under 500 tokens. Anthropic’s official guidance is under 200 lines. Some teams run it at 60.

Three things most people don’t know about CLAUDE.md:

HTML comments (<!-- internal note -->) are stripped before injection. They cost zero tokens. Use them for notes to teammates, rationale, anything humans need but Claude doesn’t.

@path/to/file imports let you split CLAUDE.md across multiple files. But all imported files still load at session start. Splitting is purely organizational, it doesn’t save tokens.

CLAUDE.local.md (gitignored) is your personal preferences file. Good for local sandbox URLs and notes you don’t want in the shared config. It loads alongside CLAUDE.md.

One more: edits to CLAUDE.md during a session don’t apply until the next restart or /compact. Claude reads it once at startup and that’s it.

The token_audit.py tool in token-optimizer automates the before/after comparison:

python claude_code/token_audit.py --compare claude_code/before/CLAUDE.md claude_code/after/CLAUDE.md
python claude_code/token_audit.py --scan-dir .  # full project scan

.claudeignore vs permissions.deny, know the difference

.claudeignore signals to Claude that certain files are not relevant.

It is advisory, not enforced, Claude can still read ignored files if it decides they’re necessary. Multiple GitHub issues (#36163, #51105) document this behavior.

For actual enforcement, use permissions.deny in .claude/settings.json:

{
  "permissions": {
    "deny": ["Read(node_modules/**)", "Read(dist/**)", "Read(*.lock)"]
  }
}

This blocks the Read tool for those paths entirely. Claude cannot read them regardless of what it decides.

In practice, use both: .claudeignore handles the signal layer (Claude won’t proactively include these files), permissions.deny handles the hard block (Claude can’t read them even if it tries). The benchmark’s 85.5% context reduction from .claudeignore was measured on the signal layer. Teams with strict context discipline use permissions.deny in addition.

Minimum .claudeignore for any project:

node_modules/
dist/
build/
.next/
__pycache__/
*.pyc
*.lock
package-lock.json
yarn.lock
poetry.lock
coverage/
*.generated.*
*.min.js
*.min.css

Check it into version control, every team member gets the same context discipline automatically.

Path-scoped rules in .claude/rules/ — the 41% win almost nobody uses

.claude/rules/ lets you place rules files that load selectively. Rules without frontmatter load at session start like a second CLAUDE.md, no savings. Rules with paths: frontmatter load only when Claude reads a file matching that pattern, zero token cost until triggered.

---
paths:
  - "src/api/**/*.ts"
---
# API Layer Rules
All endpoints must validate input with Zod schemas.
Response errors must use the shared ApiError class.
Never return raw Prisma errors to the client.

This rule costs nothing during frontend work, database work, or test writing. It only enters context when Claude first touches a file in src/api/. This is the key insight: path-scoped rules are invisible until needed.

One documented case (Zenn, 2025) reduced always-loaded rule overhead from 1,358 lines to 807 lines — 41% reduction — by: - Converting 5 procedure-heavy rule files into Skills (on-demand only) - Scoping 8 domain-specific rules to their respective directories - Keeping only truly global rules unscoped

Read more

Welcome to Edition #36 of a newsletter that 15,000+ people around the world actually look forward to reading.

🎓 Want to master frontier AI Engineering?

Build production-scale multi-agent ecosystems using Google ADK, MCP, and Claude Code Harness. Learn to solve the KV cache pressure of 1M-token contexts by implementing CSA/HCA hybrid attention and FP4 quantization. Ensure reliability with rigorous evals and observability to manage the consistency of complex agentic chains

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) May 30th (15% discount)

• Watch the free 4-session Agent Bootcamp playlist on YouTubet on YouTube

DeepSeek V4 Architecture Comparison: MoE Design, Attention Mechanisms, and How It Stacks Up Against V3, Claude Opus, and OpenAI

DeepSeek V4 is a high-performance open-weights artificial intelligence model that features a massive 1.6 trillion parameter architecture.

It highlights the use of a Mixture-of-Experts (MoE) design, which allows the model to activate only a small fraction of its total parameters during use to maintain cost-efficiency.

A major innovation detailed is the hybrid attention system, combining two distinct compression methods to manage an expansive one-million-token context window with minimal memory requirements.

The sources contrast two versions, the powerful V4 Pro and the agile V4 Flash, to help technical architects choose the right size for their specific hardware and speed needs.

By offering architectural transparency, the text positions these models as competitive, auditable alternatives to closed-source options like Claude and OpenAI.

This article serves as a technical guide for engineers looking to deploy frontier-level AI on their own infrastructure.

⚠️ Image Disclaimer & Notes

This diagram is a conceptual reconstruction of the reported DeepSeek V4 architecture based on publicly available technical discussions, community analysis, and early implementation references.

Some low-level implementation details, including routing internals, layer distributions, tokenizer specifications, and naming conventions may differ from the final official release.

The figure is intended for educational and illustrative purposes rather than as an official DeepSeek architecture diagram.

Introduction: Why architecture-level analysis of DeepSeek V4 can no longer be deferred

As I have been reading about DeepSeek v4, most published stays at the benchmark surface: leaderboard positions, MMLU scores, MATH pass rates, and cost-per-million-token comparisons that tell you what the model produces but almost nothing about how it produces it.

For product managers and casual evaluators, that framing is fine. For senior ML engineers and technical architects making infrastructure commitments that will govern production workloads for the next eighteen to twenty-four months, it is dangerously incomplete.

The choice between DeepSeek V4 Pro (1.6 trillion total parameters) and DeepSeek V4 Flash (284 billion total parameters) is not simply a throughput-versus-capability trade-off.

The two variants have divergent attention topologies, distinct KV cache pressure profiles, and fundamentally different memory access patterns with cascading effects on GPU memory budgeting, batching strategy, and multi-agent serving infrastructure.

An organization that selects V4 Pro for a long-context document processing pipeline and then discovers mid-deployment that its KV cache management assumptions were calibrated for dense-attention models will face non-trivial re-engineering costs.

What separates DeepSeek’s V4 release from Anthropic and OpenAI is architectural transparency with no current closed-source equivalent.

The published technical report details the Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) interleaving strategy, the FP4 Lightning Indexer that drives top-k block selection in CSA, and the Manifold-Constrained Hyper-Connections (MCHC) governing expert connectivity. Neither Anthropic’s Claude Opus materials nor OpenAI’s published documentation describe comparable components at this granularity. That transparency is both a competitive signal and an analytical obligation: the details are published, so there is no excuse for practitioners to rely on leaderboard proxies.

Open-weights frontier quality has closed, and in some benchmarks erased, the performance gap that once justified proprietary lock-in. Organizations that committed to closed-source APIs in 2024 and early 2025 are actively revisiting those decisions. This article provides the architectural evidence those decision-makers need, grounded in what the technical report actually discloses.

[Source: DeepSeek V4 Technical Report, April 2026, parameter counts, context window specifications, and MoE activation ratios for both Pro and Flash variants , primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

🔑 Key Takeaways

⚡️ V4 Pro vs. Flash are built differently, DeepSeek V4 comes in two distinct sizes (1.6T and 284B parameters) designed for different deployment budgets, not just different speed preferences.

📝 CSA + HCA replaces standard attention, DeepSeek V4’s hybrid attention system handles both short-range and long-range context more efficiently than the grouped query attention used in most competing models.

💪 MoE scaling jumped dramatically from V3, DeepSeek V4 activates only a small fraction of its total parameters per token, keeping inference costs manageable despite the massive jump in overall model size.

🧠 1-million-token context needs a smarter memory plan, V4’s FP4 Lightning Indexer and compressed KV blocks (4× in CSA, 128× in HCA) let the model attend across very long documents at roughly 27% of V3.2’s FLOPs and 10% of its KV cache.

🚀 V4 Pro trades blows with Claude Opus and OpenAI, On reasoning and long-context benchmarks, V4 Pro sits in the same performance tier as the leading closed-source models while remaining openly available.

⭐️ V4 Flash challenges smaller closed-source models, For teams that need an efficient, production-ready model without licensing restrictions, V4 Flash is a credible open-weights alternative to proprietary mid-tier options.

1. DeepSeek V4 in context: what changed from V3 and why the gap is larger than it looks

1.1 DeepSeek V3 architectural baseline: what was already impressive

To understand V4’s significance, start with an honest accounting of what V3 already accomplished. DeepSeek V3 shipped with a Mixture-of-Experts architecture totaling approximately 671 billion parameters, activating roughly 37 billion per forward pass, a ratio that made it one of the most compute-efficient open-weights frontier models at its release. Its context window extended to 128,000 tokens, competitive with closed-source offerings at the time and sufficient for most enterprise document processing workloads.

The mechanism that most distinguished V3 from its contemporaries was Multi-head Latent Attention (MLA). Rather than caching full key-value pairs for every attention head, MLA compresses KV representations into a low-dimensional latent space before caching, then reconstructs the full representation at inference. This reduced KV cache memory requirements significantly without the quality degradation that naive KV quantization introduces. MLA was a genuine innovation, and it earned V3 its position as the dominant open-weights model for much of 2025. [Source: DeepSeek V3 Technical Report, December 2024: https://arxiv.org/abs/2412.19437]

MLA, for all its cleverness, was a single-mechanism attention system. Every transformer layer used the same compressed latent attention approach, applying identical computational strategy to both local syntactic relationships and long-range semantic dependencies. That architectural uniformity is exactly what V4’s hybrid CSA/HCA system abandons, not as incremental refinement but as a qualitative architectural shift.

1.2 The V3-to-V4 scaling jump: parameters, experts, and active compute

The headline number for V4 Pro, 1.6 trillion total parameters, invites a misleading interpretation. A jump from 671 billion to 1.6 trillion naively suggests a 2.4× inference cost increase. It does not.

The activated parameter ratio governs inference cost, and V4 Pro maintains a disciplined relationship between total and activated parameters that preserves economic viability at scale. Based on published figures, V4 Pro activates approximately 49 billion parameters per token during a standard forward pass, roughly 3.1% of total parameters. V4 Flash, at 284 billion total with approximately 13 billion activated per token, achieves an activation ratio near 4.6% reflecting its different optimization target. [Source: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

For context: Mixtral 8×22B activates approximately 39 billion of its 141 billion total parameters per token, an activation ratio near 28%. V4 Pro’s ~3.1% is a dramatically more aggressive sparsity strategy, viable only because the routing and expert specialization mechanisms have been engineered to maintain quality under extreme sparsity. [Source: Mixtral technical report, Mistral AI, 2024: https://arxiv.org/abs/2401.04088]

A practitioner allocating GPU budgets for a V4 Pro deployment needs to reason from activated parameters per token, memory bandwidth requirements for loading expert weights, and expert parallelism overhead, not from the 1.6T total figure in press coverage.

V4 also moves off AdamW for pretraining, using the Muon optimizer across more than 32 trillion training tokens — a notable choice DeepSeek credits for faster convergence and improved training stability at trillion-parameter scale. Pretraining and MoE expert weights are quantized using a mix of FP8 and FP4 precision with quantization-aware training (QAT), which is what enables the aggressive KV cache compression discussed in Section 4 without the per-token quality degradation that naive post-training quantization typically introduces.

TABLE 1: DeepSeek V3 vs. V4 Pro vs. V4 Flash, core architectural parameters

Total parameters and activated parameters sourced from DeepSeek V4 Technical Report, April 2026. Expert count and Top-K routing for V4 Pro and Flash are estimated from architectural descriptions in the technical report; precise values should be verified against the primary source before citing in downstream documentation.

1.3 What V4 Flash is — and is not — designed for

A persistent mischaracterization frames V4 Flash as a distilled or compressed version of V4 Pro, essentially V4 Pro with weights pruned and quantized to fit smaller hardware. This is architecturally wrong and operationally misleading. V4 Flash is independently designed, with its own MoE configuration, attention layer interleaving ratios, and expert routing parameters optimized for a different operational regime.

The deployment scenarios V4 Flash targets are specific: inference workloads where GPU memory is the binding constraint, production endpoints with aggressive tail latency SLAs, and single-node or dual-node deployments where the expert parallelism strategies required for V4 Pro would introduce unacceptable cross-node communication overhead. At 284 billion total parameters with approximately 13 billion activated per token, V4 Flash is deployable on a four-to-eight GPU node configuration economically accessible to organizations that cannot justify multi-node infrastructure.

The capability trade-off is real but unevenly distributed across task types. V4 Flash’s gap to V4 Pro on standard reasoning benchmarks (roughly 8–12 percentage points on most published evaluations) understates its deficit on the hardest long-context retrieval tasks, where V4 Pro’s larger expert pool and higher-capacity HCA layers confer meaningful advantages. Teams treating V4 Flash as a throughput shortcut to V4 Pro’s quality will encounter that gap precisely in multi-hop long-document reasoning and complex agentic chains, where quality matters most.

2. CSA and HCA hybrid attention: the core architectural innovation explained

2.1 What Compressed Sparse Attention (CSA) does and why it replaces local windowed attention

Compressed Sparse Attention operates on a fundamentally different computational graph than the sliding window attention familiar from Mistral-class models. Sliding window attention restricts each token’s receptive field to a fixed neighborhood, discarding attention weights outside that window. CSA instead constructs a compressed representation of local context before computing attention, summarizing the semantic content of the neighborhood into a lower-dimensional proxy that is cheaper to attend over without the information loss that pure windowing introduces. Sliding window attention achieves sparsity by ignoring tokens; CSA achieves sparsity by compressing them.

The computational savings compound across layers. A standard sliding window attention layer with window size w scales as O(n·w) in attention operations, where n is sequence length. CSA reduces the effective dimensionality of the compressed context further, producing savings that matter enormously at V4’s 1-million-token context window, the difference between a deployable context length and a theoretically appealing one. [CITE: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro; see also Longformer: The Long-Document Transformer, Beltagy et al., 2020: https://arxiv.org/abs/2004.05150; BigBird, Zaheer et al., 2020: https://arxiv.org/abs/2007.14062]

The technical lineage connects to Longformer’s combination of local and global attention, BigBird’s random attention augmentation, and ETC’s structured sparsity, but CSA’s compression step distinguishes it from all predecessors by operating on representational content rather than positional structure. That distinction earns the “compressed” label rather than borrowing it for marketing purposes.

2.2 How Heavily Compressed Attention (HCA) handles long-range dependencies

Heavily Compressed Attention handles what CSA’s lighter compression cannot: efficient global attention over the full 1-million-token context. HCA does not attempt full quadratic attention over 1 million raw tokens, that would be computationally intractable regardless of activation sparsity. Instead, HCA applies an aggressive ~128× compression along the sequence dimension. After this aggressive compression, the resulting sequence is short enough that dense attention becomes cheap again — HCA drops sparse selection entirely and computes full dense attention over the compressed sequence.

The distinction between CSA and HCA matters: CSA uses a lighter 4× compression and pairs it with sparse, top-k block selection (via the Lightning Indexer detailed in Section 4); HCA uses much heavier 128× compression and pairs it with dense attention. CSA preserves more positional fidelity for nearby tokens; HCA preserves more global coherence at low cost. In aggregate, V4-Pro at 1M-token context uses only ~27% of single-token inference FLOPs and ~10% of the KV cache compared with DeepSeek V3.2 — not because either mechanism alone is dramatically cheaper, but because the two are mutually compensating. [CITE: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

Without HCA’s selective architecture, a 1-million-token context window would require either full quadratic attention, computationally impossible at scale, or exclusively local-window mechanisms that fail on tasks requiring multi-hop reasoning across distant positions. HCA is the structural solution to a problem that scaling context length alone cannot solve.

2.3 The interleaving pattern: how CSA and HCA alternate across transformer layers

Which proportion of transformer layers employ CSA versus HCA is one of the most operationally consequential architectural decisions in V4’s design, and one of the least discussed in mainstream coverage. Early transformer layers build syntactically dense local representations where primary information dependencies are short-range, making CSA’s local compression appropriate. Deeper layers increasingly require global semantic coherence, where HCA’s long-range selective attention delivers value.

The published technical report indicates V4 Pro employs a roughly 3:1 CSA-to-HCA ratio across its transformer depth, with HCA layers concentrated in the latter two-thirds of the network. V4 Flash employs a 4:1 ratio, with fewer HCA layers proportionally, reducing the computational overhead of long-range attention at the cost of some depth in global context integration, consistent with its latency-sensitive optimization target. [CITE: DeepSeek V4 Technical Report, April 2026, flag: exact layer ratios should be verified against primary source before production use in infrastructure planning — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

This ratio has direct serving infrastructure implications practitioners commonly overlook. HCA layers, because they operate on aggressively compressed sequences and perform dense attention over the result, exhibit a different memory access and compute profile than CSA layers do. A batching strategy optimized for uniform-attention architecture will systematically misestimate the latency profile of HCA-heavy deeper layers, producing inaccurate SLA projections for workloads that exercise long-range context heavily.

2.4 CSA + HCA vs. grouped query attention in Claude Opus and OpenAI models

Grouped Query Attention (GQA), the attention mechanism inferred from technical blog posts and verified in community analysis of Claude Opus and OpenAI’s current GPT-class models, achieves KV cache reduction by grouping multiple query heads to share a single key-value head pair. This reduces distinct KV heads from H to H/G (where G is the group size), producing memory savings that scale linearly with the grouping factor. GQA is well-engineered and delivers a meaningful efficiency gain over vanilla multi-head attention. [CITE: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al., 2023: https://arxiv.org/abs/2305.13245; Anthropic Claude technical documentation, 2025–2026: https://www.anthropic.com/research]

GQA optimizes within a fixed computational graph. It reduces the memory cost of caching attention outputs but does not change the fundamental O(n²) attention computation or restructure how the model relates local versus global context. CSA + HCA does not merely reduce cache size, it changes what the model computes. CSA replaces full local attention with compressed-representation attention; HCA replaces full-sequence global attention with retrieved-subset attention. These are different computations with different memory access patterns, different scaling laws at long context, and different implications for the KV cache architecture the serving system must maintain.

V4’s long-context efficiency advantages cannot be replicated by applying GQA-style optimizations to a standard dense transformer. The gains are baked into the architecture at a level that requires the full CSA/HCA design commitment, a genuine structural differentiator, not a quantization shortcut a competitor could match with an inference-time optimization patch.

3. Mixture-of-Experts architecture: how DeepSeek V4 scales without scaling inference cost

3.1 MoE fundamentals revisited: why expert routing matters at V4 scale

The fundamental promise of Mixture-of-Experts is the separation of model capacity from per-token compute cost. A dense transformer of N parameters activates all N for every token in every forward pass. An MoE transformer of N total parameters activates only the K experts selected by the routing mechanism for each token, so inference cost scales with activated parameters, not total parameters. At 1.6 trillion total parameters with approximately 49 billion activated per token, V4 Pro delivers the quality of a 1.6T parameter model while paying the inference cost of a model closer to 49B. [CITE: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

The engineering challenges are non-trivial. Expert routing must select the right experts for each token without access to the full context that would make those selections obvious, the router operates on compressed token representations and must generalize from training to deployment. Load balancing across experts during training is notoriously difficult: without explicit constraints, gradient dynamics concentrate routing on a small subset of high-capacity experts, leaving others undertrained and degrading effective model capacity. These failure modes distinguish a well-engineered MoE from one that looks good on a parameter count sheet but underperforms in production.

V4 Pro’s improvement over V3’s already-strong MoE baseline is measurable in the activation ratio trajectory. V3 activated approximately 37 billion of 671 billion total parameters (~5.5%). V4 Pro’s ~3.1% activation ratio is a substantial sparsification, sustainable only because the routing innovations in Section 3.2 and the expert connectivity architecture in Section 3.3 maintain quality under conditions that would cause earlier MoE designs to degrade.

3.2 DeepSeek V4’s expert routing innovations: beyond standard Top-K gating

Standard top-K gating, where a learned linear router scores all experts for each token and selects the K highest-scoring, exhibits well-documented failure modes at V4 Pro’s scale. The most consequential is routing collapse: under auxiliary-loss-free training regimes, the router gradually concentrates token assignments on a small subset of experts, effectively reducing active model capacity toward the size of those preferred few. Earlier MoE systems addressed this with auxiliary load-balancing losses that penalize uneven routing distributions during training, effective but prone to gradient interference between task loss and load-balancing loss, creating training instability at scale. [CITE: DeepSeek V3 Technical Report, December 2024: https://arxiv.org/abs/2412.19437]

DeepSeek V4 builds on V3’s auxiliary-loss-free load balancing, extending it with what the technical report describes as adaptive routing temperature, a mechanism that dynamically adjusts the sharpness of the routing distribution based on token-level uncertainty estimates. This encourages broader expert utilization for ambiguous tokens while allowing confident routing where a single expert is clearly superior. The practical result is more even expert utilization across training without the gradient interference of auxiliary losses, translating into better expert specialization and more consistent inference behavior across diverse input distributions.

V4 also continues and extends V3’s fine-grained expert segmentation and shared expert mechanisms. Shared experts, a small pool that receives routing probability mass from all tokens regardless of the router’s decisions, ensure universal language modeling capabilities are maintained even as specialized experts diverge toward domain-specific representations. Without shared experts, gradient pressure to maintain general language capabilities causes specialized experts to drift toward generalization, eliminating the specialization that justifies having multiple experts at all. [CITE: DeepSeek V3 Technical Report, December 2024: https://arxiv.org/abs/2412.19437; DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

3.3 Manifold-Constrained Hyper-Connections: what they are and why they change the MoE story

Manifold-Constrained Hyper-Connections (MCHC) is the least-discussed of V4’s novel architectural components, which is unfortunate because it addresses one of the most practically significant failure modes in large MoE systems. Standard residual connections add each sublayer’s output to the residual stream with uniform weight. Hyper-connections parameterize the connectivity between sublayers, allowing the model to learn input-dependent weighting of how much each expert’s output contributes to the residual stream for each token.

The manifold constraint imposes a geometric restriction on these learned connectivity weights, confining their dynamics to a learned lower-dimensional manifold rather than the full parameter space. This draws on intrinsic dimensionality arguments in deep learning, the observation that effective weight configurations cluster in low-dimensional subspaces. By constraining hyper-connections to this manifold explicitly, MCHC prevents expert connectivity patterns from drifting into high-capacity but training-unstable regions of parameter space, reducing the expert homogenization problem that degrades MoE quality at scale. [CITE: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro; see also Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, Aghajanyan et al., 2021: https://arxiv.org/abs/2012.13255]

The downstream inference consequence is consistency: models trained with effective manifold constraints exhibit lower output variance across semantically similar inputs, which translates into more predictable production behavior. In agentic deployments where a single long-horizon task involves dozens of model calls, variance accumulation across calls is a meaningful quality degradation mechanism. MCHC’s contribution to inference consistency does not appear in single-call benchmark scores, it is a production reliability property that emerges under sustained deployment, which is precisely why it is absent from the leaderboard comparisons dominating public V4 discussion.

3.4 GPU memory and serving infrastructure implications of V4 Pro’s MoE vs. dense alternatives

The full V4 Pro weight set, 1.6 trillion parameters at BF16 precision, requires approximately 3.2 terabytes of GPU memory to hold all weights simultaneously. No single node of current-generation H100 or H200 GPUs (typically 640GB or 1.1TB aggregate VRAM per 8-GPU node) can hold V4 Pro’s full weights, making multi-node expert parallelism mandatory for full-weight serving. The practical minimum is approximately three to four H200 nodes for full-weight BF16 inference, or two nodes with aggressive FP8 quantization applied to non-activated expert weights. [CITE: DeepSeek V4 infrastructure deployment guide, April 2026: https://github.com/deepseek-ai/DeepSeek-V4; community serving benchmarks, April 2026: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

The compensating advantage is compute intensity per forward pass. Because only ~49B parameters are activated per token, V4 Pro’s arithmetic intensity per token is comparable to a 49B dense model, GPU compute utilization during the active forward pass is far lower than a dense model of equivalent quality would require. This creates a specific economic profile: high capital cost (multi-node GPU memory) but low per-token compute cost, favorable for high-throughput, long-session workloads where amortized memory cost per token is small. The economics invert for low-throughput workloads with short sessions, where fixed multi-node infrastructure cost cannot be amortized over enough tokens to justify the expense against Claude Opus or OpenAI API pricing.

V4 Flash’s 284B parameters fit comfortably on a single 8-GPU H200 node with FP8/FP4 quantization, enabling single-node deployment with standard tensor parallelism, a significantly simpler infrastructure commitment that makes V4 Flash the practical choice for organizations without established multi-node GPU serving infrastructure.

4. Lightning Indexer + compressed KV blocks: the KV cache strategy that makes 1M-token context practical

4.1 The KV cache problem at 1 million tokens

The KV cache, stored key-value pairs from previous positions that enable autoregressive generation without recomputing all previous tokens at each step, scales linearly with sequence length in standard transformer architectures. At 1 million tokens, the memory requirement becomes prohibitive. A rough estimate for a model of V4 Pro’s layer depth and head configuration places the naive full-sequence KV cache at several hundred gigabytes per request. Even with GQA-style reduction, serving multiple concurrent long-context requests would exhaust GPU memory on any realistic infrastructure configuration.

DeepSeek V4’s architectural solution is not a separate memory system bolted onto the attention stack. The KV cache strategy is the CSA + HCA design itself, plus an FP4 Lightning Indexer that lives inside CSA layers and drives top-k block selection. KV entries are compressed along the sequence dimension (~4× in CSA, ~128× in HCA), and the Lightning Indexer selects which compressed blocks each query attends over. Reported result at 1M-token context: ~27% of V3.2’s single-token inference FLOPs and ~10% of V3.2’s KV cache. [CITE: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

4.2 How the Lightning Indexer works: FP4 scoring and top-k block selection

The Lightning Indexer is a lightweight scoring network operating at FP4 precision that scores all compressed KV blocks for a given query and selects the top-k blocks for sparse attention computation. Running the indexer at FP4 keeps its compute cost negligible relative to the attention operation itself, which is critical: an indexer that costs as much as the attention it replaces would defeat the purpose. The indexer is trained jointly with V4’s main weights so that block selection learns to surface the positions actually attended over in practice, not heuristic position-based proxies. [CITE: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

This is where the CSA + HCA split becomes load-bearing. CSA layers use the Lightning Indexer to attend sparsely over a top-k subset of 4×-compressed blocks. HCA layers skip the indexer entirely — at 128× compression the sequence is short enough that dense attention is cheaper than selection. Practitioners profiling V4 serving stacks should distinguish these two paths: CSA’s cost profile is dominated by indexer scoring + sparse gather operations, while HCA’s is dominated by short-sequence dense matmuls. Optimization strategies that work for one path do not transfer to the other.

4.3 Impact on KV cache pressure in multi-agent and long-context deployments

The practical significance of this design is most visible in multi-agent deployment scenarios, one of the fastest-growing enterprise AI infrastructure patterns as of May 2026. In a multi-agent architecture where a single long-context document, a legal case file, a codebase, a research corpus, is shared across multiple agent roles, each agent processes overlapping but distinct subsets of the document. With naive full KV caching, each agent maintains its own KV cache for the shared document, multiplying the memory footprint by the number of active agents.

V4’s compressed-block design enables shared compressed KV across agents: multiple agents accessing the same document can share the compressed block representation and the Lightning Indexer’s scoring infrastructure, with each agent’s top-k selection differing by query while the underlying blocks remain a single in-memory copy. GPU memory footprint scales with the size of the compressed block pool plus per-agent indexer state, not with document length multiplied by agent count. For a ten-agent architecture processing a one-million-token document, this is not a marginal optimization — it is the difference between a feasible memory budget and an infeasible one. [CITE: DeepSeek V4 Technical Report, April 2026 — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro; DeepSeek V4 serving infrastructure benchmarks, April 2026: https://github.com/deepseek-ai/DeepSeek-V4]

Almost no published V4 analysis addresses this directly: the architectural cost-efficiency advantage of DeepSeek V4 in production is not primarily about inference FLOPS per token. It is about KV cache pressure dynamics under multi-agent, long-context workloads, a regime where the Lightning Indexer’s top-k selection and HCA’s dense-on-compressed approach combine to change the memory scaling law in ways that no benchmark score captures.

5. Benchmark results and performance comparisons: what the numbers actually show

5.1 DeepSeek V4 Pro vs. V3: where the architectural changes translate into measurable gains

The performance gap between V4 Pro and V3 is most pronounced on tasks that exercise the capabilities the new architecture specifically targets. On MATH-500, V4 Pro scores approximately 96.2 versus V3’s 90.2, a 6-point improvement reflecting both the larger expert pool and the improved long-range context integration HCA enables for multi-step problem chains. On MMLU-Pro, the gap is smaller (approximately 3–4 percentage points), consistent with the expectation that general knowledge tasks are less sensitive to long-range attention improvements than multi-hop reasoning tasks. [CITE: DeepSeek V4 Technical Report, April 2026, benchmark section — primary source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro]

On long-context benchmarks, specifically RULER, which evaluates retrieval accuracy across context lengths up to 128K tokens, and the newer Long-ROPE evaluation suite extending to 1M tokens, V4 Pro’s advantage over V3 grows with context length. At 128K tokens, V4 Pro outperforms V3 by approximately 8 percentage points on RULER. At 500K tokens, V3’s performance has degraded significantly while V4 Pro maintains near-128K accuracy. This is the functional difference between a model that supports long context architecturally (V4) and one that extends it via position embedding extrapolation (V3), and the CSA Lightning Indexer + HCA compression are the mechanisms responsible for that gap.

5.2 DeepSeek V4 Pro vs. Claude Opus and OpenAI: honest comparisons across task types

Comparing V4 Pro to Claude Opus and OpenAI’s leading models requires precision about what is being compared. On standard reasoning benchmarks, MATH-500, HumanEval, GPQA Diamond, V4 Pro and Claude Opus are within approximately 2–4 percentage points of each other across most evaluations, with task-specific wins distributed between them. Claims that either model is “better” are really claims about task distribution. [Source: Anthropic Claude Opus 4 technical documentation, 2026: https://www.anthropic.com/claude; LMSYS Chatbot Arena leaderboard, April 2026: https://chat.lmsys.org/]

The more meaningful differentiation appears at extended context lengths. On 1M-token context tasks, V4 Pro has no direct closed-source competitor, Claude Opus’s published context window stands at 200K tokens as of April 2026. OpenAI’s current models support longer contexts but do not publish architectural details about KV cache management at extended lengths, making infrastructure planning for those models necessarily opaque. The ability to audit V4’s Lightning Indexer block selection and HCA compression behavior, to understand exactly why performance degrades or holds at specific context lengths, is a practical advantage for organizations whose workflows depend on reliable long-context behavior. [CITE: OpenAI model card and technical documentation, April 2026: https://openai.com/research; Anthropic Claude technical documentation, April 2026: https://www.anthropic.com/research]

TABLE 2: DeepSeek V4 Pro vs. V4 Flash vs. Claude Opus vs. OpenAI, selected benchmarks (April 2026)

[Benchmark figures sourced from DeepSeek V4 Technical Report, April 2026, and independent evaluations published on LMSYS leaderboard and Hugging Face Open LLM Leaderboard as of April 29, 2026. OpenAI and Claude figures drawn from respective official model cards and third-party evaluations. All figures should be verified against primary sources for production decision-making.]

5.3 DeepSeek V4 Flash vs. mid-tier closed-source models: the efficiency tier

V4 Flash’s positioning against mid-tier closed-source models is compelling for a specific organizational profile: teams that need production-ready, cost-efficient inference without proprietary API dependency. On standard benchmarks, V4 Flash performs comparably to or above models in the GPT-4o-mini and Claude Haiku performance tier while offering full weight access, fine-tuning on proprietary data, and freedom from per-token API pricing at scale. The 284B parameter scale with 13B activated per token is deployable on infrastructure many mid-to-large engineering organizations already operate for other workloads, making the marginal infrastructure cost of adopting V4 Flash lower than the raw parameter count suggests.

The caveat is operational: V4 Flash requires ML infrastructure competence that closed-source API consumption does not. Organizations evaluating V4 Flash should honestly assess whether their MLOps maturity supports model weight management, serving optimization, and quantization tuning. A naively deployed V4 Flash will underperform its potential and may not justify the infrastructure investment over a well-tuned API integration.

6. Practitioner guidance: model selection and infrastructure decisions

For senior ML engineers and technical architects

The central decision for senior ML engineers evaluating DeepSeek V4 is not V4 versus a closed-source alternative. It is which V4 configuration best matches the infrastructure constraints and workload profile of their specific deployment, and whether the architectural commitments that configuration requires are ones their organization can sustain.

If your workload involves documents or conversation histories exceeding 200K tokens, codebases, legal corpora, extended research documents, long-horizon agentic sessions, V4 Pro’s CSA + HCA hybrid attention is the only available option (open or closed-source) that handles this regime with published, auditable mechanisms. The infrastructure cost is significant: plan for a minimum of three H200 nodes for full-weight BF16 inference, or two nodes with FP8 quantization and careful monitoring of quantization-induced degradation on your specific task distribution. Expert parallelism requires low-latency inter-node interconnect, NVLink fabric or equivalent, to avoid cross-node communication overhead becoming a throughput bottleneck. Profile your workload’s HCA-versus-CSA layer balance before committing to a batching strategy: if your inputs are predominantly short-context (under 32K tokens), you are paying for HCA capacity you will not use, and V4 Flash is likely the more economical choice.

For workloads in the 32K–200K token range, the V4 Pro versus V4 Flash decision reduces to a quality-versus-infrastructure-cost calculation specific to your quality SLA. V4 Flash’s performance deficit on long-context retrieval tasks in this range is real but not uniform, if your task distribution is dominated by extraction and summarization rather than multi-hop reasoning, V4 Flash’s deficit may be within acceptable tolerance. Conduct targeted evaluation on your actual task distribution rather than relying on published benchmark averages that aggregate across heterogeneous task types.

On serving stack configuration: implement KV cache management aware of CSA’s compressed-block layout and HCA’s heavily-compressed dense path. Standard static prefix caching optimizations developed for dense-attention models will mis-allocate GPU memory for V4 deployments, over-provisioning for positions the Lightning Indexer would otherwise prune and under-provisioning for HCA’s short-sequence dense matmuls. DeepSeek’s published serving infrastructure repository includes CSA/HCA-aware KV cache configuration templates that should be the starting point for production deployment. Profile MCHC behavior under your target batch size to verify expert routing load is balanced across the expert pool. Routing imbalance invisible in single-request benchmarks can become a significant throughput degradation mechanism under production batch loads.

For fine-tuning: MCHC’s manifold constraints have implications for parameter-efficient fine-tuning. Standard LoRA applied to expert weight matrices may not respect the manifold geometry MCHC maintains during pretraining, potentially degrading the inference consistency properties that MCHC provides. Use DeepSeek’s published MCHC-aware fine-tuning guidelines as the baseline, and validate fine-tuned model consistency under multi-call agentic workloads before production deployment.

For technical architects and infrastructure decision-makers

Whether open-weights models have reached frontier quality is settled, V4 Pro’s benchmark parity with Claude Opus on most standard evaluations confirms it. The real question is whether the total cost of ownership for a V4 deployment, infrastructure capital, MLOps operational overhead, and engineering time to configure and maintain a V4 serving stack, undercuts the total cost of the closed-source API commitment it replaces.

The break-even calculation favors V4 deployment at higher token volumes. Below approximately 100 million tokens per month, closed-source API pricing is typically more economical than the capital and operational costs of V4 infrastructure. Above approximately 500 million tokens per month, a threshold many production agentic applications exceed, the economics typically invert, and V4’s open-weights availability becomes financially compelling. At 1 billion tokens per month or above, the financial case is strong enough that the comparison is primarily about risk tolerance and MLOps capability rather than economics.

Data sovereignty deserves serious weight in architectural decisions. Claude Opus and OpenAI’s models process your organization’s data on their infrastructure under their data use policies. V4 deployed on your infrastructure does not, every token of every request stays within your security perimeter. For organizations in regulated industries where data residency requirements are legally binding, V4’s open-weights model may be the only viable compliance path for frontier-quality AI capability, not merely a cost-competitive alternative.

The architectural transparency argument has a practical production dimension beyond compliance. When a V4 deployment exhibits unexpected behavior on a specific input class, you can trace it to the relevant architectural mechanism, examine which experts are activated, how the Lightning Indexer is scoring compressed blocks, whether HCA’s dense-on-compressed path is integrating long-range context correctly. When Claude Opus exhibits unexpected behavior, you file a support ticket. For organizations where model behavior auditability is a production reliability requirement, V4’s transparency is a material operational advantage that no benchmark comparison captures.

7. Frequently asked questions

Q. What is the most important architectural difference between DeepSeek V4 Pro and V4 Flash for infrastructure planning purposes?

The most consequential difference is not parameter count but attention layer interleaving ratio and its downstream effect on KV cache pressure. V4 Pro’s lower CSA-to-HCA ratio (approximately 3:1) means a larger proportion of its forward passes exercise the heavily-compressed dense attention path, requiring serving infrastructure configured for HCA’s short-sequence matmul profile. V4 Flash’s higher CSA dominance (approximately 4:1) means its latency profile leans more heavily on Lightning Indexer scoring + sparse gather, simplifying serving stack configuration. For teams without established MoE serving infrastructure, V4 Flash’s simpler memory access pattern is a meaningful operational advantage beyond its lower absolute memory footprint.

Q. How does DeepSeek V4’s KV cache strategy compare to standard eviction strategies used in most long-context serving systems?

Standard KV cache eviction strategies, sliding window eviction, recency-based eviction, static prefix caching, make eviction decisions based on token position or recency without reference to relevance to the current generation task. V4 takes a different approach: rather than evicting, it compresses. CSA keeps all KV entries but at 4× compression along the sequence dimension, then uses an FP4 Lightning Indexer (trained jointly with V4’s main weights) to select the top-k compressed blocks per query for sparse attention. HCA layers go further, applying 128× compression and computing dense attention on the much shorter compressed result. At sequence lengths above 100K tokens, the difference is substantial: heuristic eviction discards semantically critical distant tokens that happen to be old, while V4’s compressed blocks retain a lower-fidelity representation of every position and let the indexer surface the relevant ones. The design is architecturally more complex but produces significantly better long-context retrieval accuracy for tasks where critical information is distributed non-uniformly across the context.

Q. Is DeepSeek V4 Pro a viable replacement for Claude Opus in enterprise production deployments?

On benchmark performance, V4 Pro is within statistical noise of Claude Opus on most standard evaluations, making it technically credible for organizations whose quality requirements are captured by those benchmarks. The more relevant question is organizational: V4 Pro deployment requires MLOps infrastructure investment, security and compliance evaluation of open-weights model governance, and engineering capacity to configure and maintain a multi-node serving stack. Organizations with mature ML infrastructure and high token volumes will likely find V4 Pro a cost-effective and capable replacement. Organizations with limited ML engineering capacity, low token volumes, or stringent compliance requirements around model governance should evaluate the full total-cost-of-ownership comparison rather than treating benchmark parity as the deciding factor.

Q. What does Manifold-Constrained Hyper-Connections mean for fine-tuning V4 on proprietary data?

MCHC imposes geometric constraints on expert connectivity weight space during pretraining that improve training stability and inference consistency. When fine-tuning V4 on proprietary data, standard LoRA applied naively to expert weight matrices may introduce perturbations that violate the manifold geometry MCHC established, potentially degrading the inference consistency properties that MCHC provides. DeepSeek’s published fine-tuning guidelines recommend MCHC-aware LoRA configurations that constrain updates to perturbations compatible with the manifold structure. Full fine-tuning with manifold constraints maintained is more computationally expensive but produces fine-tuned models that retain V4’s base inference consistency properties more reliably than unconstrained LoRA approaches.

Q. How does DeepSeek V4’s Mixture-of-Experts design compare to Mixtral 8×22B in terms of inference efficiency?

Mixtral 8×22B activates approximately 39 billion of 141 billion total parameters per token (~28% activation ratio) with a relatively simple top-2 routing mechanism. V4 Pro activates approximately 49 billion of 1.6 trillion total parameters per token (~3.1%), sustained by significantly more sophisticated routing, adaptive temperature gating, shared experts, fine-grained segmentation, and MCHC expert connectivity constraints. The absolute activated parameter counts are within striking range of each other, but V4 Pro achieves that activation level while maintaining the quality of a 1.6T parameter model — over 11× more total capacity than Mixtral. The comparison is not that V4 activates fewer parameters in absolute terms, it is that V4 achieves far higher model quality per activated parameter.

Conclusion: the architectural evidence for your infrastructure decision

The practitioner gap in current DeepSeek V4 coverage is not a data deficit, the technical report publishes more architectural detail than any competing frontier model. The gap is analytical: almost no published analysis connects CSA/HCA interleaving ratios to serving latency profiles, the FP4 Lightning Indexer’s top-k selection to multi-agent KV cache dynamics, or MCHC manifold constraints to fine-tuning stability. These connections determine whether an architecture-level model selection decision produces a production infrastructure that performs as expected or requires expensive re-engineering six months after deployment.

For high-volume, long-context, or multi-agent workloads where token volumes justify owned infrastructure, V4 Pro’s architectural design differs from closed-source alternatives in ways that benchmark scores do not capture, specifically in long-context memory management efficiency, inference consistency under sustained agentic deployment, and the operational auditability that architectural transparency enables. V4 Flash is the right choice for organizations with tighter infrastructure budgets, latency-sensitive endpoints, or workloads that do not require the full 1M-token context capability. The right answer is determined by the interaction between your workload profile and your infrastructure capacity, not by a benchmark leaderboard position.

If you are currently in an infrastructure evaluation comparing DeepSeek V4 Pro or Flash against Claude Opus or an OpenAI offering, download the DeepSeek V4 technical report and map the CSA/HCA compression ratios and Lightning Indexer top-k budget against your projected KV cache footprint at your target context length and batch size before finalizing the comparison. That single calculation will tell you more about the right architecture for your workload than any leaderboard score.

Sources

1. https://substackcdn.com/image/fetch/$s_!kZDt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cbbb885-f965-4f56-80fe-2b7e28842237_2254x1258.png

2. https://amitray.com/wp-content/uploads/2025/12/DeepSeek-Sparse-Attention-DSA-Mechanism-scaled.jpg

3. https://huggingface.co/blog/assets/moe/thumbnail.png

4. https://behope.com/cdn/shop/files/4x.png?v=1769012471&width=1920

💡 Want to share your work on my socials with my 15k+ audience?

If you build a project you are excited about, I will be too.

Trust me! I love seeing people build cool stuff. To share it, you can contact me here.

Did you enjoy this post?

Here are some other AI Agents posts you might have missed:

• KV Caching and Speculative Decoding

• A deep dive into Quantization: Key to Open Source LLM Deployments

• Agents are here and they are staying

• How Agents Think

• Memory – The Agent’s Brain

• Agentic RAG Ecosystem

• Multimodal Agents

• Scaling Agents: Architectures with Google ADK, A2A, and MCP

• Fully Functional Agent Loop

You’re receiving this because you’re part of our mailing list. We don’t spam or sell your information. To unsubscribe, use the link below.

Ready to take it to the next level? Check out my AI Agents for Enterprise course on Maven and be part of something bigger, join hundreds of builders developing enterprise-level agents.

Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) May 30th (15% discount)

You’re receiving this because you’re part of our mailing list. We don’t spam or sell your information. To unsubscribe, use the link below.

Today, I am joined by Aishwarya, a product builder obsessed with turning ideas into working tools, especially with AI in the mix.

Welcome to Edition #35 of a newsletter that 15,000+ people around the world actually look forward to reading.

Subscribe now

🎓 Want to learn about Claude Code?

Join us on May 30th, for a one-day workshop on Claude Code and ship your first agent!

Sign up today

AI Agent Harnesses Explained: Architecture, Ecosystem, and Multi-User Design

Key Takeaways

Agent = Model + Harness. Every production agent is two systems running in concert: a language model generating tool calls, and a harness deciding which of those calls are actually allowed to execute.

Harnesses do the work models cannot. File access, sandboxed execution, and audit logging live in the harness layer. The model never directly touches your filesystem — the harness is the only code that does.

Codex and Claude Code take opposite bets. OpenAI’s Codex isolates by spinning up cloud containers per task; Anthropic’s Claude Code runs locally and asks explicit consent on each consequential action. Both are valid for different workflows.

Multi-user is where harness design gets hard. Concurrent users force per-user permission inheritance, namespaced memory, and tamper-evident audit logs. Architectures that worked for one developer fail subtly when ten share an instance.

Safety lives in the harness, not the model. If you’re trusting the model to refuse bad actions, you have no safety. Refusals only count when the harness validates the tool-call schema and rejects it before execution.

Harness governance is now an org-design problem. Who configures the tool registry, who approves writes to shared memory, and who reviews audit logs are policy questions. Treating them as DevOps tasks is how teams end up with no policy at all.

Introduction

By May 2026 the limiting factor for production AI coding agents has stopped being the model.

Frontier coding models post 60%+ on SWE-bench Verified and ship million-token contexts. The pieces that actually decide whether a team ships agent-powered features or produces expensive incidents are the runtime wrapped around the model: filesystem permissions, sandbox boundary, rollback path, audit log, per-user memory scope. That runtime is the agent harness.

The shape of the problem is captured in a one-line formula: Agent = Model + Harness.

The model is the inference engine. It reads tokens, generates structured tool calls, and otherwise does nothing in the world. The harness is the execution environment that takes those calls, validates them, runs them inside an isolated workspace, and feeds structured results back. Anthropic’s own Claude Code best-practices post treats the harness as the layer where safety properties get enforced rather than decoratively applied around an already-safe model.

In 2026 the harness market has clarified into two reference designs. OpenAI’s Codex ships an open-source CLI plus a cloud agent that spins up a fresh container per task, isolating concurrent runs through process boundaries rather than coordination logic. Anthropic’s Claude Code ships a terminal-resident harness that runs locally and asks explicit consent on each consequential action, treating the developer as a co-signer on every meaningful tool call. They are not competitors so much as opposite design bets, useful precisely because they map cleanly to different workflow profiles.

What follows is a practitioner-grade breakdown of what an agent harness is, the five things it must do, how Codex and Claude Code actually implement them, and the design assumptions that break the moment you have ten developers sharing an instance instead of one.

1. What Is an AI Agent Harness? The Formula Most Teams Get Half Right

1.1 The Agent = Model + Harness formula unpacked

The model is the inference engine: it processes tokens and generates a probability-weighted output sequence. It has no native ability to open a file, remember last Tuesday’s refactor, or refuse to delete a production database. The harness is the surrounding execution environment that gives the model hands, it intercepts model-generated tool calls, routes them through a permission layer, executes them in a controlled environment, and feeds structured results back into the model’s context. Neither component is optional: a model without a harness is a sophisticated autocomplete engine, and a harness without a capable model is an empty scaffold.

Consider a surgical robot. The AI model is the surgeon’s decision-making intelligence, it perceives the situation and determines the action. The robotic arm is the harness, it makes contact with the physical world. The sterilization protocols, motion envelope limits, force sensors, and logging systems built into that arm aren’t surgeon responsibilities; they’re harness responsibilities. A brilliant surgeon operating a poorly built arm is dangerous. A well-built arm with a weak decision system is also dangerous. Both halves matter.

Most teams in 2024 and early 2025 invested heavily in the model half, selecting frontier models, tuning prompts, evaluating outputs, while treating the harness as something they’d build out later. The teams who built rigorous harness infrastructure are the ones running agents confidently in production now.

1.2 What the harness actually does: the five core responsibilities

Every production harness, regardless of implementation language or deployment model, must fulfill five core responsibilities.

First, tool execution: the harness intercepts model-generated tool calls and translates them into real system operations, file writes, shell commands, API invocations, then returns structured results to the model’s context.

Second, memory and context management: the harness decides what the agent remembers across turns, tasks, and sessions, including how it summarizes, evicts, and retrieves prior context.

Third, sandboxing: the harness isolates agent execution so that a malformed tool call, an adversarial prompt injection, or a simple programming error cannot cascade into irreversible damage to the systems the agent is operating on.

Fourth, state persistence: the harness maintains the agent’s working environment, open files, branch state, task progress, across interrupted or resumed tasks, so an agent can be paused and restarted without losing coherent context.

Fifth, permission enforcement: the harness determines which tools, files, repositories, and external APIs the agent is authorized to access, and enforces those boundaries at execution time rather than trusting the model to self-police.

These are architectural invariants. Any production agent system missing one is operating with either an open attack surface or an uncontrolled blast radius.

1.3 The spectrum of harness sophistication

Harnesses exist on a maturity spectrum, and knowing where your current implementation sits is the first step toward knowing what to build next.

Level 0 (Bare Invocation): The model receives a prompt and returns text. A human reads that text and manually executes any suggested actions. No harness infrastructure at all, just a chat interface and a clipboard.

Level 1 (Tool-Calling Wrapper): The model can invoke a predefined tool schema, and a thin wrapper executes those calls. No persistent memory between sessions, no sandboxing, no rollback. A mistake is permanent until a human fixes it manually.

Level 2 (Session-Aware Harness): Persistent memory within a session, basic rollback via snapshotting, and sandboxed execution for a single concurrent user. This is roughly where most well-engineered individual developer tooling sat in 2025.

Level 3 (Multi-User Production Harness): Full per-user execution isolation, scoped permission matrices, shared audit logs with per-user attribution, and robust concurrent agent support. This is what Codex and Claude Code deliver today, and it’s the baseline any serious engineering organization needs to be targeting.

1.4 Harness vs. framework vs. orchestrator: clearing up the confusion

Three terms regularly get conflated in practitioner conversations, and conflating them produces architectural mistakes.

An AI framework, LangChain, LlamaIndex, CrewAI, is a developer library that provides abstractions for building agent pipelines. It helps you wire together components, define chain-of-thought flows, and integrate with vector stores. It does not sandbox tool calls or enforce permissions. It’s a construction kit, not a job site.

An orchestrator, Temporal, Prefect, Airflow used in agent pipeline contexts, manages task sequencing, retries, and workflow state. It doesn’t understand agent-specific concerns like memory scoping, tool call injection prevention, or per-user isolation.

A harness is the runtime environment that actually executes agent actions, enforces constraints, manages memory, and provides the safety boundary between the model and the systems it operates on. A sophisticated harness may use a framework to structure its agent logic and an orchestrator to manage task queuing, but the harness is the higher-level concept that contains both.

If the model is a contractor: the framework is the project management software (Jira, Linear), the orchestrator is the job scheduler, and the harness is the actual job site, with locked tool cabinets, safety equipment, a building permit on the wall, and a foreman checking credentials before any tool leaves the shelf.

2. Inside the Machine: Core Architectural Components of a Production Harness

Read more

Today, I am joined by Aishwarya, a product builder obsessed with turning ideas into working tools, especially with AI in the mix.

Welcome to Edition #34 of a newsletter that 15,000+ people around the world actually look forward to reading.

🎓 Want to learn about Claude Code?

Join us on May 8th, for a one-day workshop on Claude Code and ship your first agent!

Sign up today

Two engineering teams. Same Claude Code. Completely different outcomes. The gap isn’t the model — it’s knowing which primitive to reach for.

Claude Code Agents Explained: Skills, Subagents, Hooks, Plugins, and Harnesses for Production Multi-Agent Workflows

Introduction

Most developers hit a wall somewhere around their third or fourth Claude Code workflow. Individual agents work. The problem is getting them to work together — without context blowout, unpredictable ordering, or a debugging session that consumes the entire sprint.

The root cause is almost always the same: practitioners are using the right tools in the wrong layer. A behavioral constraint that should be a hook gets written into a system prompt. A reusable workflow that should be a skill gets copied and pasted into every conversation. A task that belongs in a subagent ends up in the main session and fills the context window with logs nobody will reference again.

Claude Code gives you six distinct primitives:

• CLAUDE.md (persistent context),

• skills,

• plugins,

• subagents,

• hooks, and

• the harness.

Each has a specific job. Mixing them up doesn’t just make code harder to maintain — it produces agent systems that behave inconsistently, cost more than they should, and fail in ways that are hard to reproduce.

This article maps the full stack. You’ll get the isolation spectrum mental model first (the single clearest way to understand when to use what), then a deep dive on each primitive, starting with skills — the one most teams discover last despite needing it first.

🔑 Key Takeaways

🧩 Skills are the missing primitive most teams overlook. They run inside your current context window, load only when needed, and replace the pattern of pasting the same instructions into every conversation.

🪝 Hooks give you deterministic control points. Unlike CLAUDE.md instructions, hooks enforce behavior architecturally — a hook that blocks a tool call cannot be reasoned around.

🔌 The plugin/subagent confusion kills otherwise good architectures. Plugins extend what Claude can touch; subagents extend how Claude reasons. Getting this backwards produces designs that collapse at scale.

🏗️ Subagents protect your main context window. They do isolated work and return a summary — the exploration noise never lands in your primary session.

🤝 Agent teams are experimental and fundamentally different from subagents. They run in separate processes, communicate directly with each other, and coordinate via a shared task list. Subagents can’t do any of that.

☁️ The harness is where institutional knowledge lives. Every team convention that currently exists in a senior engineer’s head can be encoded as enforced logic in a harness — and that’s the whole point.

1. The isolation spectrum: the mental model that cuts through the confusion

Before diving into individual primitives, there’s one diagram worth committing to memory:

Skills ──────────── Subagents ──────────── Agent Teams
(Same Context)    (Isolated Context)    (Separate Process)
  Low cost              ↕                  High isolation
  Fast                  ↕                  Parallelism

This spectrum describes the fundamental trade-off in every Claude Code design decision. The further right you go, the more isolation and parallelism you get — and the more overhead you pay. The further left, the cheaper and faster, but everything shares context.

Skills sit at the left end. They load into your current conversation and stay there. Claude sees the instructions alongside everything else in the session.

Subagents sit in the middle. They run in their own context window, do their work, and return a summary to the main agent. What happened inside the subagent doesn’t pollute your main session.

Agent teams sit at the right end. Each teammate is a separate Claude Code process. They share a task list and can message each other directly — not just report back to the main agent. As of this writing, agent teams are experimental and require an environment variable to enable.

The full hierarchy looks like this:

Harness              ← the runtime (you build one or live inside Claude Code's)
  └── Main Agent     ← runs inside the harness
        ├── Skills   ← in-context instructions, same window
        ├── Subagents ← isolated workers, one-way comms back
        └── Agent Teams ← separate processes, bidirectional

Plugins and hooks apply across all levels. Plugins extend what any agent can touch — the tool surface. Hooks fire at lifecycle events and give you deterministic control points regardless of which layer you’re in.

1.1 Why the agent-as-the-unit framing fails

Most developers start by thinking about “the agent” as the atomic design unit — the thing you configure, the thing that fails, the thing you debug. That framing collapses all the interesting architectural decisions into one blob.

In practice, an agent is a Claude model instance with tools, a context window, and the ability to take multi-step actions. That definition says nothing about how the agent coordinates with others, how its actions are constrained, or how it fails gracefully when context fills up halfway through a task. Those decisions live in the layers around the agent — the primitives.

Treating agent-as-the-unit leads to monolithic agents holding 180k-token contexts with the entire repository history, every tool in the stack, and an open-ended system prompt trying to anticipate every subtask. They blow context unpredictably, produce inconsistent behavior across runs, and are nearly impossible to debug. The right structure is smaller, scoped primitives doing one job each.

1.2 The Declarative Extension Model

From Vikash Rungta’s architectural breakdowns of Claude Code, there’s a clean table that maps each primitive to its format and scope:

Each row has a distinct responsibility boundary. Confusing them — running routing logic inside a subagent’s system prompt instead of in a harness hook, using CLAUDE.md for enforcement instead of hooks — is the root cause of most “my agent setup is a mess” situations.

2. Skills: the primitive most teams discover last

Skills are the first thing most practitioners should reach for, but typically discover after they’ve already built messier workarounds.

A skill is a SKILL.md file stored in .claude/skills/<name>/, invoked either directly with /skill-name or automatically when Claude judges it relevant. The key property: it runs in the same context window as the current conversation. No isolation, no spawning, no new process — just additional instructions that become part of the session.

2.1 What skills are for

Create a skill when you notice yourself pasting the same instructions into every new conversation. If you have a section of CLAUDE.md that has grown into a multi-step procedure rather than a standing fact, that procedure belongs in a skill. CLAUDE.md content loads on every session; skill content loads only when invoked.

Per the official docs: “A file at .claude/commands/deploy.md and a skill at .claude/skills/deploy/SKILL.md both create /deploy and work the same way. Your existing .claude/commands/ files keep working.” Skills are the evolved form of custom commands, with additional features: YAML frontmatter, supporting files, dynamic context injection, and the ability to fork into a subagent when isolation is needed.

Frontmatter controls key behaviors:

---
name: deploy
description: Deploy the application to production
disable-model-invocation: true   # only you can trigger this
allowed-tools: Bash(git add *) Bash(git commit *)
---

disable-model-invocation: true is the most important field to understand. Without it, Claude can invoke the skill automatically when it judges the situation relevant — fine for reference content, wrong for anything with side effects like deployments or sends.

2.2 Dynamic context injection

One feature of skills that has no equivalent elsewhere: the !command`` syntax runs a shell command before the skill content reaches Claude, injecting the output inline.

## Current diff

!`git diff HEAD`

## Instructions

Summarize the changes above and flag anything risky.

When this skill runs, git diff HEAD executes first, and its output replaces the placeholder. Claude receives the actual diff, not a reference to one. This keeps the skill grounded in the current state of the repository rather than relying on Claude’s inference from open files.

2.3 Skills vs. CLAUDE.md vs. subagents

The three are frequently confused. Here’s the distinction:

• CLAUDE.md: facts and standing rules that apply to every session in this project. Not procedures, not playbooks — context.

• Skills: procedures that run in-context, on demand. Loaded when invoked, not before.

• Subagents: isolated workers that protect the main session from their output. Use when the task’s intermediate steps (logs, search results, file reads) don’t need to stay in the main context.

The wrong choice here has a predictable failure mode: CLAUDE.md that keeps growing into a procedures document bloats every session. Ad-hoc instructions pasted into chat can’t be reused. Subagents spawned for tasks that should be skills add unnecessary overhead and context isolation where none was needed.

Read more

Today, I am joined by Aishwarya, a product builder obsessed with turning ideas into working tools, especially with AI in the mix.

Welcome to Edition #34 of a newsletter that 15,000+ people around the world actually look forward to reading.

🎓 Want to learn about Claude Code?

Join us on May 8th, for a one-day workshop on Claude Code and ship your first agent!

Sign up today

Claude Design by Anthropic: What it is, how it works, and what real users actually think

Figma’s stock dropped 4.26% the day Claude Design launched.

Anthropic’s own CPO had already resigned from Figma’s board three days prior. The signal was in place before most designers had even opened the tool.

Subscribe now

In this article, we will cover:

• What Claude Design actually is and what everyone keeps getting wrong about it.

• How prompting works in practice: the 2-prompt benchmark, where it holds up, where it doesn’t.

• Claude Design vs. Lovable: the right tool for the right job, not a horse race.

• Claude Design vs. Figma: where Figma still wins by a significant margin.

• What this means for the design role long-term, the honest answer, not the LinkedIn version.

• Who should use it, how to start, and what to avoid.

• What the community is actually building and where they’re getting burned.

Introduction: What actually happened on April 17, 2026

It’s May 2026. Anthropic dropped Claude Design on April 17 and the design tool market, which had been quietly settling into a Figma-dominated equilibrium, cracked open again.

The reaction online split immediately. On r/ClaudeAI, u/SweetCaramel7947 posted: “People can keep shitting on Dario, but when you see what they’ve achieved with each launch and you actually use it to produce something useful, you realise this is nothing less than magic. Before Canva, people needed Adobe skills. After Canva, the barrier got lower. Its the same now — I don’t need to know Figma or Canva. I just need to know what needs to be shown to my audience.”

The post got upvoted heavily.

So did “Claude Design is practically unusable” from a different user the same week.

Both reactions are correct, for different reasons. The tool is genuinely impressive in specific conditions and genuinely frustrating outside them. Most coverage has picked one lane. This article covers both.

The official launch description from Anthropic’s r/ClaudeAI post: “Claude Design is powered by Claude Opus 4.7, our most capable vision model. Describe what you want and Claude builds the first version. Refine through conversation, inline comments, direct edits, or custom sliders, then export to Canva, as PDF or PPTX, or hand off to Claude Code. Claude reads your codebase and design files to build your team’s design system, then applies it automatically, keeping every project on-brand.” Available in research preview on Pro, Max, Team, and Enterprise plans.

That’s what Anthropic says it is. Here’s what it actually is in practice.

🔑 Key Takeaways

• 📌 Figma’s stock dropped 4.26% the day Claude Design launched — the market reacted before most designers had opened the tool. Reddit user u/Future_Language76833: “The entry point to design just got demolished.”

• ⚡ The image generation quality is a real problem — users on r/ClaudeAI described results as “a brown diagonal smear across the canvas” after six attempts. Layout reasoning is genuinely impressive. Generative image quality is not. These are different capabilities and the gap matters.

• 🔑 Loose prompts get generic AI-slop output — every early adopter agrees: if your brief is vague, Claude Design falls back to a default internal design system. The tool rewards intent-rich prompts. Specificity is the entire skill.

• ⚠️ Usage limits are the #1 real-world friction — a YouTube Short titled “Claude Design has the most aggressive usage limits in AI right now” hit 10K views for a reason. File size limits mid-write are a known bug. Token ceilings on Max plans are hitting active users hard.

• 🏗️ A 25-year design veteran on Reddit called the shot — “The vast majority of UX and visual design is maintaining design systems, cobbling together functionality with very little variation. It was designed to be automated. It’s simply training data waiting for AI to come along.” Whether you agree or not, this is the argument to engage with.

• 🚀 The smart workflow uses Claude Design and Figma, not Claude Design or Figma — Claude wins the ideation and first-draft phase. Figma wins collaboration, handoff, and ecosystem depth. Practitioners getting the most value are using both at different stages.

Part 1: What Claude Design actually is — and what it isn’t

1.1 The product in plain English: not a plugin, not a copilot

The most persistent confusion: Claude Design is not a plugin. Not a Figma add-on. Not Claude.ai with a design sidebar. Not a VS Code extension.

It’s a standalone application that Anthropic launched out of Anthropic Labs — the company’s first direct move into productivity software as a product, not an API. Before this, Anthropic’s commercial model was largely B2B: sell Claude’s intelligence to companies who build on top of it. Claude Design is a different bet — Anthropic putting its own product on the table with a specific argument about how design should work.

That argument: AI shouldn’t assist human-directed design decisions — it should own the first pass entirely, with the human reviewing results rather than directing every step. That’s not a subtle UX difference. It’s a different philosophy about where intelligence should live in a design workflow.

Malewicz, a designer with 192K YouTube subscribers, put it clearly in his 71K-view video “Claude Design is NOT what you think”: “The stock market definitely seems to think [it’s a Figma killer], but we need to chill and learn to not mistake noise for signal.” That’s the right framing to start with.

1.2 Intent-first design: why this is architecturally different

The core premise separating Claude Design from everything else: it operates on design intent rather than design output.

When you ask Figma’s Make Designs, or most AI tools with design capabilities, for a dashboard layout, the model draws on visual pattern data. It knows what dashboards look like and produces something that resembles one. When you ask Claude Design for the same thing with proper context, the system reasons about why a user in a specific context would need certain information arranged in certain ways. It’s constructing a justification for every major decision, not retrieving a visual pattern.

Ask it why it chose a particular layout structure and it will tell you — referencing the constraints and goals you described in your prompt. That’s new behavior in a design tool, and it’s what produces the output quality gap users report when they compare a vague Claude Design prompt against a specific one. The same model, dramatically different output, based entirely on whether the prompt communicates intent or just description.

The flip side, which Reddit users identified fast: if your prompt is loose, “it WILL implement the design it has in its system prompt” (u/AmmarAlammar2004). The reasoning layer only activates when there’s something to reason from. Vague input = generic output. That’s not a bug — it’s the fundamental dependency of intent-first design.

1.3 The Dialt Kit and Agentation: what’s actually driving the efficiency claims

Most early coverage focuses on visual output quality. The two features that actually explain the numbers early adopters cite are less covered: the Dialt Kit and Agentation.

The Dialt Kit is Claude Design’s integration layer — it connects to your existing design infrastructure (design tokens, component libraries, handoff formats) so generated output is calibrated to your actual constraints rather than producing beautiful mockups that share no DNA with your real product.

Early integration partners include Tokens Studio, Storybook, and Zeroheight, with Figma token import in active development as of Q2 2026. The developer who built the design-system-extraction Claude Code plugin on r/ClaudeAI was solving for exactly this gap: “You can extract a site’s design, then tell Claude ‘build me a landing page using this design system’ and it actually nails it because it has the exact tokens, scales, and component patterns.”

Agentation is Claude Design’s multi-step autonomous execution mode.

One high-level prompt triggers a cascade of coherent micro-decisions — spacing logic, visual hierarchy, component selection, responsive breakpoints, interactive state handling — all in a single pass.

The designer’s job shifts from directing each decision to reviewing a completed chain of them. This is what produces the 2-prompt benchmark that keeps getting cited. Without Agentation, you’re just prompting a design AI. With it, you’re reviewing an AI that made 40 design decisions from a single brief.

1.4 How it fits — or doesn’t — into your existing stack

The honest answer to “does this replace my tools?”: it depends on your role, your team, and whether you’re being honest about which part of your workflow is actually the bottleneck.

For a solo designer or freelancer in a Figma + AI workflow: Claude Design is a plausible primary tool for ideation and first-draft work. Early adopters report 40–60% reduction in time-to-first-draft on new projects. You still move into Figma for final polish, component formalization, and developer handoff.

For a product team with an established design system: Claude Design fits as a first-draft layer where designers and PMs co-generate initial directions before a senior designer refines. Where it falls short, plainly: developer handoff maturity, real-time collaboration, and component library depth. If those capabilities are load-bearing in your workflow, be honest about that before committing to evaluation time.

For a no-code builder on Lovable: section 3 covers this in full, but the short version is that Claude Design currently doesn’t produce deployable code as a primary output. These tools are for different jobs.

Part 2: How Claude Design works, what actually happens when you use it

2.1 The prompt quality gap is bigger than anyone tells you upfront

The most important skill in Claude Design isn’t knowing the tool — it’s learning to write prompts that communicate intent, not just description.

The gap between a weak and a strong prompt is larger here than in any AI tool most people have used.

Prompt A: “Make me a SaaS dashboard.”

Prompt B: “Build a SaaS analytics dashboard for a marketing manager at a B2B company who needs to monitor campaign ROI, compare channel performance across email, paid, and organic, and quickly identify underperforming campaigns. Mobile-first, dark mode preferred, clear drill-down CTAs on each metric card. This user checks this tool first thing every morning.”

Prompt B doesn’t just produce a better-looking dashboard — it produces a structurally different one. The dark mode is calibrated for ambient morning light contrast ratios, not applied cosmetically. The hierarchy surfaces the “quickly identify underperforming campaigns” job more prominently than the tracking functions. That’s the intent-first reasoning in action.

Three frameworks that consistently produce strong outputs:

Role + Task + Context: “Build a [UI type] for a [user role] who needs to [job to be done] in a context where [key constraint or usage condition].”

Before + After: “The current experience has [problem]. Design an interface that resolves this by [desired outcome], for a user who [behavioral description].”

System + Goal + Constraint: “This is part of a [product type] targeting [audience]. The goal of this screen is [specific task or conversion]. Constraints: [brand, technical, accessibility requirements].”

The designers reporting the highest satisfaction with Claude Design in early 2026 are not necessarily the most experienced — they’re the ones who learned to translate design thinking into precise, intent-rich language fast.

2.2 The 2-prompt benchmark: what it actually means

The claim generating the most conversation — “complex pages in roughly 2 prompts” — is real, with important qualifiers.

A “complex” page by this standard includes multiple content zones (hero, features, social proof, CTA), responsive behavior across at least two breakpoints, varied component types (cards, tables, navigation, modals), and interactive state handling. The 2-prompt pattern looks like this:

Prompt 1 establishes structure, intent, user context, and primary layout logic. Agentation generates a full first draft — all zones, all components, responsive logic included.

Prompt 2 handles refinement: “Increase visual weight on the primary CTA, tighten spacing in the feature grid, add an empty state for the activity feed.”

Why Lovable requires 20+ prompts for the same outcome: it’s executing discrete code changes. Every instruction modifies specific code — button size, hover state, spacing value — one atomic operation at a time. Claude Design infers component relationships from the original intent brief. You don’t specify button size because the visual hierarchy you described makes an appropriate size the coherent derivation. That’s a structural difference, not a quality comparison.

The honest limits of the benchmark: highly bespoke brand systems, WCAG-critical interfaces, complex data visualization components typically need 4–6 prompts. Still dramatically fewer than alternatives, but not 2.

The most credible real-world validation came from Anthropic’s official launch page: Olivia Xu, Senior Product Designer at Brilliant, wrote: “Our most complex pages, which took 20+ prompts to recreate in other tools, only required 2 prompts in Claude Design. Including design intent in Claude Code handoffs has made the jump from prototype to production seamless.”

2.3 What Agentation actually feels like

You type: “Build me a complete onboarding flow for a B2B SaaS tool targeting HR managers at mid-size companies. Cover account setup, team invitation, and first integration connection. Prioritize clarity over comprehensiveness — these are busy people.”

With Agentation active, without further input, Claude Design generates a multi-screen sequence: a welcome screen with a progress indicator and a single primary action; account setup with sensible field grouping and inline validation; team invitation with bulk-add and a clear “skip for now” path; an integration screen with the five most common HR integrations surfaced as simple toggles. Every screen maintains visual consistency — same spacing scale, same typography hierarchy, same button states. Empty states and error states are handled throughout.

Your role in this mode: reviewer and approver. Assess whether the information architecture matches the actual user journey. Check whether the visual hierarchy supports the stated priority of clarity. Approve or redirect. The parallel to GitHub Copilot in software development is exact — developers stopped writing boilerplate and started reviewing AI-generated implementations. The required skill transformed but didn’t disappear.

2.4 Where you hit the wall

Every AI tool has a wall. Here’s Claude Design’s.

Direction-level revisions are home territory — “make this feel more premium,” “tighten the overall density,” “shift the hierarchy to emphasize the data table”. What it struggles with: contradictory constraints. Ask for something “more minimal but also more detailed contextual help throughout” and the model resolves the tension itself, without flagging that it made a call.

By revision 4 or 5, outputs can start drifting from the original intent. The most effective recovery is a new anchoring prompt that restates core intent — not layers of corrections on top of each other. The tool implicitly requires workflow discipline around this.

The image generation wall is harder. Multiple r/ClaudeAI users hit it immediately. One trying to build a floral studio hero: “Told it ‘bouquet of dried flowers.’ Six turns in and it’s still a brown diagonal smear across the canvas. My wife walked by and asked why there’s poop on my screen.” The text-layout reasoning and the generative image capability are genuinely different quality levels right now. Build around it.

Read more

Welcome to Edition #33 of a newsletter that 15,000+ people around the world actually look forward to reading.

Join my free session, sign up here

Beyond Search: How AI Is Rewriting the Rules of Brand Visibility

Picture this: a consumer picks up their phone, opens ChatGPT, and types, “What’s the best protein powder for building muscle without the bloat?” In seconds, they get a confident, specific answer, one product, a clear reason why, and zero websites to click through. No ads. No sponsored results. No scrolling past three blog posts to find an actual recommendation. For millions of users, AI has already replaced search engines as the default discovery mechanism. Brands that spent two decades mastering Google are finding their playbook obsolete in real time.

This isn’t a trend piece speculating about 2027. It documents a structural disruption actively reshaping how consumers find, evaluate, and buy products right now. You built your brand’s visibility on a platform whose central mechanic, the ranked list of links, is being replaced by something fundamentally different. By the end of this article, you’ll understand what’s breaking, why the financial stakes are higher than most marketing teams realize, and what specific moves you need to make before the window for adaptation closes.

Subscribe now

🔑 Key Takeaways

• 🔍 AI has already replaced Google for high-intent queries — tens of millions of people now use ChatGPT, Perplexity, and Claude as their first stop for product research, comparison, and recommendations, not as a supplement to search but as a replacement.

• 📉 Organic CTR dropped 61% on queries where AI Overviews appear — a page ranking #2 can see impressions climb while clicks collapse. “We rank #1” no longer means what it used to.

• 🤖 AI agents don’t browse. They decide. — OpenAI Operator, Perplexity Shopping, and Apple Intelligence complete transactions on users’ behalf. The brand selection happens before any human marketing touchpoint is engaged.

• 🏗️ Your visibility now depends on infrastructure, not creative — schema markup, API accessibility, review aggregator presence, and Wikipedia accuracy determine whether an AI recommends you. These are engineering decisions, not marketing ones.

• 🎙️ Voice is the most extreme version of the problem — spoken AI answers are singular. There is no second place. Only 4% of businesses are genuinely voice-search ready despite 91% claiming they invest in it.

• 🎯 Traversaal.ai is already solving this for enterprise — if your brand needs to stay discoverable as AI agents become the primary interface between customers and products, this is exactly the problem contextual agent infrastructure is built to address.

1. For millions of people, the search engine is already gone

This isn’t a prediction. It’s a behavior change that’s already showing up in traffic reports, revenue dashboards, and boardroom conversations at companies that are paying attention. The shift is measurable and it’s accelerating.

The numbers that should be keeping CMOs up at night

By January 2026, AI chatbots collectively hold a measurable share of the search market: ChatGPT commands roughly 68% of AI search traffic (down from 87% a year prior as competition intensifies), Google Gemini has grown to 18.2% (up from just 5.4% a year earlier), and Perplexity sits at 8.15%. (Source: https://aibusinessweekly.net/ai-market-share-2026)

Gartner projected in February 2024 that traditional search engine volume would drop 25% by 2026 as AI chatbots take share — and the trajectory suggests that estimate is landing close to target. (Source: https://www.gartner.com/en/newsroom/press-releases/2024-02-19-gartner-predicts-search-engine-volume-will-drop-25-percent-by-2026)

AI search already accounted for 8.2% of total search traffic in August 2025, up from effectively zero two years prior (Wix AI Search Lab, September 2025). By July 2025, Similarweb reported that 69% of Google searches end without a click to any website, up from 56% just a year earlier. (Source: https://evolvagency.io/learn/generative-search/llm-statistics-2026) These aren’t experiment numbers. They’re market share numbers.

Meanwhile Google’s own CEO Sundar Pichai said in April 2026 that the future of Search is “multimodal, conversational, and predictive.” (Source: https://icypluto.com/blog/sundar-pichais-latest-insights) Google isn’t fighting this shift. It’s building into it.

The disruption isn’t coming from a competitor eating Google’s lunch. It’s happening inside Google’s own product.

Google gave you ten results. AI gives one.

The traditional SERP was a reasonably democratic marketplace. A brand ranking fourth still got seen. Users scanned the page, clicked the third result, maybe visited two or three sites before deciding. Multiple entry points. Multiple chances to compete.

Ask Claude or Perplexity “what’s the best CRM for a small business under 20 people?” and you get one synthesized answer, often naming a single tool, with a paragraph explaining why, and frequently no clickable links at all. Zero-click search, which SEOs have warned about for years, has become something worse: the user’s journey ends at the AI’s answer. If your brand isn’t named, you didn’t lose a click. You were never in the conversation.

Google is still standing. Your traffic model isn’t.

Google isn’t going anywhere. AI Overviews is a serious attempt to keep AI-powered answers inside the Google ecosystem, and Google has the distribution, data, and scale to remain a major player. But that’s not the same as saying brand discoverability through Google is fine.

Google’s AI summaries pull from a narrow set of sources and collapse competing brands into a single answer. Research from SparkToro shows Google already sends less than half of all searches to any website, and AI Overviews are accelerating that. (Source: https://sparktoro.com/blog/in-2023-404-of-google-searches-end-without-a-click/) The platform survives. The traffic model brands built on top of it is being taken apart, whether or not Google is the one doing it.

2. What brand invisibility actually costs

When AI returns one answer instead of ten ranked results, the economics of brand visibility don’t gradually shift. They flip. The impact is already in traffic dashboards and revenue reports for companies paying attention.

If you’re not in the answer, you’re not in the consideration set

Classic marketing theory holds that you cannot convert a customer who hasn’t entered your consideration set, the mental shortlist of brands a consumer evaluates before purchasing. On a traditional SERP, even a fourth-place ranking kept you on that shortlist. The page was a menu, and users browsed it.

In an AI-generated answer, there is no menu. There’s a recommendation. If ChatGPT names two CRM platforms and yours isn’t one of them, you don’t exist in that moment of purchase intent, not in a “you ranked lower” sense, but in a “you were never mentioned” sense. The brand that loses in AI search doesn’t get less traffic; it gets none. For categories where AI tools are already the primary research channel for key demographics, that’s not a marginal revenue impact.

Organic traffic is already in decline — and the numbers are now documented

The data is in, and it’s worse than the early estimates. Seer Interactive’s September 2025 study found that organic CTR dropped 61% for queries where AI Overviews appear — from 1.76% to 0.61%.

A separate March 2026 analysis puts overall organic search clicks down 42% from their pre-AI Overviews baseline. Informational queries — definitions, how-to content, comparison research — are the hardest hit, with 30–40% traffic declines now documented across multiple studies. 73% of B2B websites experienced significant traffic losses between 2024 and 2025.

The pattern is consistent enough that it’s no longer a reporting anomaly. Brands that used to rank in the top 4 are watching impressions go up while clicks collapse.

One r/SEO thread this year described it plainly: impressions up 66% year-on-year, clicks down 50%, for a page ranking in position 2. AI Overviews, video carousels, People Also Ask boxes, and map packs push the first organic result below the fold on most commercial queries.

For brand marketers, the contingency plan can’t be “wait and see” anymore. Pull your traffic by page type, isolate your informational and commercial content, and look at the trend from 2024 to today. If the slope is negative and hasn’t recovered, you are already losing consideration share to AI systems recommending your competitors.

The false comfort of “we rank #1”

Ranking first on Google for your most competitive keywords does not mean ChatGPT will recommend your brand. These are different systems with different inputs, and assuming one transfers to the other is one of the most dangerous assumptions in brand marketing right now.

LLM-based recommendations are shaped by training data patterns, the density of authoritative mentions across the open web, structured information quality, and citation patterns in sources that AI systems weight heavily. A brand can hold the #1 Google ranking for “best project management software” while being completely absent from ChatGPT’s recommendation because it lacks meaningful coverage in the sources LLMs trust. Ranking first on a shrinking platform isn’t a growth strategy.

3. How AI search actually works, and what it rewards

It’s not a ranking algorithm. It’s a reputation engine.

Traditional SEO was a ranking game: optimize signals, earn positions, capture clicks. AI recommendation works differently. Think of a knowledgeable colleague who has read thousands of product reviews, industry articles, Reddit threads, and expert columns. When you ask them for a recommendation, they tell you what good sources have consistently said, not a ranked list, but a synthesized judgment.

That’s what ChatGPT, Perplexity, and Claude are doing. They surface what the totality of credible sources says about a brand. The optimization goal isn’t keywords in page titles. It’s how consistently, accurately, and positively your brand is represented across the open web — what you might call reputation density.

What “authority” means when an LLM is reading your site

Traditional SEO weighted backlinks and domain authority.

LLMs weight something different: how often is this brand mentioned positively in sources that are trusted — Forbes, industry trade publications, TechCrunch, Wirecutter? What does aggregate sentiment look like on G2, Trustpilot, and Reddit? Is the brand’s information consistent across all the places it appears?

These overlap with traditional SEO in some ways — a brand with strong editorial coverage likely has good backlinks too.

But they’re not the same. A brand can have excellent domain authority and still be invisible to LLMs because its entity data is inconsistent, its review coverage is thin, and it barely appears in the editorial sources models weight most. That gap is what needs closing.

Clear and specific beats long and keyword-dense

AI tools don’t favor comprehensive, keyword-stuffed content. They favor content that’s easy to parse — semantically clear, structured so that specific facts can be pulled out confidently.

A 4,000-word post crammed with keyword variations is harder for an LLM to summarize than a 600-word page that clearly states what a product does, who it’s for, what it costs, and why customers trust it.

Schema markup, FAQ formats, clear entity definitions, direct factual statements — these are real competitive advantages in AI-driven search. Write as if the most important reader of your content is an AI deciding whether to recommend you to a million people, because increasingly, that’s exactly what’s happening.

4. When AI agents do the shopping for you

The AI search debate is still mostly about discovery — will people find your brand? The deeper problem is at the commerce layer, and most brands haven’t started thinking about it.

AI agents don’t just search. They buy.

Agentic AI systems don’t answer questions.

They take actions: browsing websites, comparing options, filling in forms, completing transactions.

OpenAI’s Operator, launched in early 2025, can navigate websites and complete tasks on a user’s behalf. Perplexity surfaces purchasable products directly inside answers. Apple Intelligence is integrating agentic features into iOS workflows. (Source: https://openai.com/operator)

The user behavior this enables is significant. Instead of “let me search for hotels in Chicago next weekend,” the user says “find me a hotel in Chicago next weekend under $200 a night with good reviews and book it.”

The agent handles the entire process. The human never visits a single hotel website, never sees a brand’s imagery or storytelling, never encounters a single marketing message. The agent decided. The human confirmed. The transaction completed.

Your website was built for humans. AI agents don’t browse like humans.

This is the insight most brands are not ready to confront. Your website, with its hero video, emotional brand positioning, aspirational imagery, and carefully crafted UX journey, was built for human visual browsing. AI agents don’t experience any of it. They parse structured data, metadata, APIs, and machine-readable signals.

What an agent can read: schema markup on your product pages, structured pricing and availability data, machine-readable trust signals like certified ratings and verified review counts, and API endpoints that return clean product information. What an agent cannot meaningfully process: a hero video, a brand manifesto, a “why us” section written in evocative marketing language, or a pop-up asking it to subscribe to your newsletter. Brands that haven’t built machine-readable infrastructure are already invisible to agents being deployed by OpenAI, Google, Apple, and dozens of startups.

The agent picks the brand. The human just says yes.

When an agent surfaces a recommendation — a booking, a cart, a subscription — the user’s job is to approve or reject. The brand decision happens before any human marketing touchpoint is engaged.

One real data point: Walmart ran an experiment with ChatGPT-powered checkout in early 2026 and found it converted 3x worse than their standard website checkout. That’s not evidence that agentic commerce doesn’t work. It’s evidence that the current integration between AI agents and brand infrastructure is still rough around the edges. Brands that wait for agents to “mature” before fixing their machine-readable infrastructure will just be further behind when the gap closes.

What shapes the agent’s selection? Structured data quality. API accessibility. Reputation signals in training data. The accuracy of your brand’s information across the web. Not a compelling ad, not a beautifully designed landing page. The traditional marketing funnel assumes a human browsing, reading, being persuaded. Agentic commerce skips most of that. Brands that aren’t optimized for the agent never reach the human behind it.

5. What to actually do about it

Start with the technical foundation, before anything else

Schema markup is the floor, not a nice-to-have.

Organization, Product, Review, and FAQ schema should be comprehensively implemented across your site.

Your brand entity information — name, founding date, description, product categories, leadership — should be consistent across Google Business Profile, Wikidata, Wikipedia (if you qualify), and major industry directories.

Where you can, build API endpoints that return clean product or service data.

Agents that can query a clean endpoint don’t have to scrape your HTML, and scraped HTML is unreliable.

Get your brand in the sources that LLMs actually trust

LLMs recommend brands they’ve seen praised in trusted places — trade publications, mainstream press, review aggregators like G2, Trustpilot, and Capterra, and community platforms like Reddit.

These are the sources models weight most heavily. Your Wikipedia and Wikidata entries matter more than most marketing teams realize. A thin, outdated, or nonexistent Wikipedia entry puts you at a measurable disadvantage before a conversation with a potential customer even starts. Fixing it is less work than most teams assume.

Your content strategy needs one more question

Stop measuring content success only by pageviews and organic traffic.

A page cited in thousands of AI answers that sends zero direct traffic is doing more for your brand than a page with 500 monthly sessions and no downstream influence on purchase decisions.

Every FAQ should genuinely answer the question. Every product description should be factually precise. Every piece of content should pass one test: can an AI pull a clean, specific, accurate fact from this without ambiguity? If no, it won’t get cited.

6. Voice AI makes the single-answer problem worse

Everything described in this article gets more severe when you add voice. And in 2026, voice is mainstream.

There are now over 8.4 billion active voice assistants globally — more than the human population. Over 4 billion devices run AI-driven voice assistants. More than 1 billion voice searches happen every month, and 32% of consumers now use voice search daily. The voice commerce market hit $150 billion in 2025 and is projected to reach $194 billion in 2026. Voice shopping revenue alone is expected to reach $40 billion.

The most revealing number: only 4% of businesses are genuinely voice-search ready, while 91% say they’re investing in voice. That gap is where the opportunity is right now.

Apple Intelligence handles voice queries natively. Alexa’s LLM-powered backend now synthesizes answers instead of returning a list of links. Google Assistant generates spoken summaries. Ear-worn assistants, smart displays, in-car AI — voice has become a primary interface for product questions and shopping.

With voice, there really is no second place

When a user reads a screen, they can scan. When they hear an answer, they get one thing. The model doesn’t say “here are two options.” It says “I’d recommend X because Y.” There’s no visual list to glance at. No second result to notice.

Either you’re the spoken recommendation, or you weren’t in the conversation at all. You don’t get named second. The conversation ended and your brand never came up.

Voice queries are longer and more specific — which is actually an advantage

Voice queries look different from typed ones: “What’s a good protein powder for someone who works out in the morning and doesn’t like chalky textures?” rather than “best protein powder.” They’re more conversational, often location-aware, and action-oriented. Brands with specific, structured content — FAQs written in natural language, use-case product descriptions, comparison content that addresses real objections — perform better in voice because that content matches how people actually ask.

There’s also a specific schema type worth implementing: SpeakableSpecification, a structured data property that marks content as suitable for text-to-speech delivery. It’s underused, relatively quick to implement, and a direct signal to voice AI about which parts of your content to surface.

Voice plus agents equals fully autonomous purchases

A user says “order me more of that protein powder.” Apple Intelligence checks purchase history, queries availability and pricing via structured APIs, and initiates the transaction. No screen, no browser, no brand touchpoint.

The brands that hold up in this environment are the ones with machine-readable product data, strong reputation signals in the sources agents consult, and inventory and pricing accessible via clean APIs. That’s not a marketing gap. It’s an infrastructure gap.

7. How to score your brand’s AI readiness

Knowing the problem exists is different from knowing where you specifically stand.

A useful way to audit your brand’s AI discoverability across four dimensions:

Run each dimension on a simple 0–100 scale: below 50 means you have a real gap, 50–70 means partial coverage, above 70 means you’re competitive. Most brands that haven’t started this work score in the 40s on Schema and Content — which means agents are making recommendations in their category without them.

The pattern across brands audited in early 2026:

Brand Authority Signals tend to score highest because established brands have some organic editorial presence.

Schema & Structured Data scores lowest because it requires technical implementation that marketing teams can’t own alone. That gap — strong offline reputation, weak machine-readable infrastructure — is exactly the mismatch this article is about.

The fix isn’t a single sprint. It’s a cross-functional workstream: product and engineering own schema and APIs, content owns FAQ and specificity, marketing and PR own authority signals. The brands moving fastest have made someone accountable for the overall score, not just their slice of it.

What this means for you

This hits differently depending on whether you manage products or build them.

📋 If you’re a Product Manager

Your discoverability metrics are measuring the wrong thing.

Most PM dashboards still track Google organic rank, session traffic, and CTR. Those numbers are real, but in 2026 they don’t tell you whether your brand is being recommended by the AI systems tens of millions of people now use for product research. You can rank #1 on Google and be completely absent from every ChatGPT, Claude, and Perplexity answer in your category. That gap doesn’t show up in your weekly report.

Whether an AI recommends your brand is a product and engineering decision, not a marketing one. It depends on structured data quality, schema markup coverage, review aggregator presence, and API accessibility. If “AI discoverability” isn’t on your roadmap as a concrete workstream, you’re already behind the brands that have.

Schema markup is a product requirement now. In Q1 2026, OpenAI’s Operator and Perplexity’s shopping agents are actively parsing product pages to make purchase recommendations. If your Product, Review, Offer, and FAQ schema aren’t implemented and current, agents skip you. That’s a revenue consequence, not a technical oversight.

Your content calendar needs one more question. Before shipping anything, ask: can an AI pull a clean, accurate, specific fact from this without guessing? If no, it won’t get cited. A page cited in 10,000 AI answers with zero direct traffic is doing more for your brand than a page with 500 monthly sessions that never influences a purchase.

One KPI to add right now: test your top 10 commercial intent queries in ChatGPT, Claude, and Perplexity monthly. Track whether your brand appears, in what context, and what competitors keep getting named instead. No tool fully automates this yet — manual spot-checking is still the most reliable signal.

The timeline isn’t next year. Operator launched publicly in early 2025. Perplexity’s shopping layer is live. Apple Intelligence agentic features shipped with iOS updates in 2025. The brands capturing AI consideration share are doing it right now while their competitors debate whether this is real.

🛠 If you’re a Developer

The web was built for humans. AI agents are a different kind of visitor, and most codebases weren’t built to serve them. Here’s what actually moves the needle:

Implement JSON-LD schema markup, comprehensively. Not as a sprinkle on your homepage — as a full coverage effort across every product, service, review, FAQ, and organization page. JSON-LD is the format Google, OpenAI’s crawlers, and Perplexity’s indexing prefer. Organization, Product, Offer, AggregateRating, FAQPage are your starting five. Run your pages through Google’s Rich Results Test and treat failures as bugs, not recommendations.

Add an llms.txt file to your domain root. This is the emerging standard (proposed in late 2024, gaining real adoption in 2025–2026) — a plain text file at yourdomain.com/llms.txt that tells AI crawlers what your site contains, which pages are machine-readable, and where your structured data lives. Think of it as robots.txt for LLMs. It’s a 30-minute implementation with real discoverability upside as more AI crawlers adopt it.

Build a clean product data API endpoint. If you have a product catalog, give agents a direct path to read it — a /api/products or /api/catalog endpoint returning clean JSON with name, description, price, availability, and review summary. Agents that can query a clean endpoint don’t have to scrape your JavaScript-heavy product pages. Scraping fails. APIs don’t.

Check what agents actually see on your site. Most modern frontend frameworks render content client-side. That means Googlebot and AI crawlers often land on a near-empty HTML shell. Run curl -s yourdomain.com/product-page | grep -i "product\|price\|description" and see what a non-JavaScript client gets. If the output is mostly empty, your product content is invisible to any agent that doesn’t execute JavaScript. Server-side rendering or pre-rendering critical pages is the fix.

Consider implementing an MCP server for your product data. Anthropic’s Model Context Protocol (open standard, launched 2024) lets AI assistants connect directly to data sources via a defined interface. If your product catalog, knowledge base, or service information is behind an MCP server, Claude and any MCP-compatible agent can query it directly without web scraping. Early-adopter brands are already doing this — it’s a meaningful discoverability advantage while most competitors haven’t heard of it.

Monitor AI referral traffic. As of 2025, major AI systems including Perplexity and some ChatGPT browsing flows send referral traffic with identifiable user agents or referrer headers. Add these to your analytics: PerplexityBot, ChatGPT-User, Claude-Web, anthropic-ai in your server logs. Understanding which AI systems are crawling you, how often, and which pages they hit tells you where your machine-readable infrastructure is working and where it isn’t.

Frequently Asked Questions

Q: Is AI actually replacing search engines, or is this overhyped? For specific use cases, product recommendations, how-to questions, comparison research, AI tools are already the primary channel for tens of millions of users. The behavioral shift is real and measurable, even if Google retains enormous overall market share. “Google still exists” and “search behavior driving brand discovery hasn’t changed” are two different claims, and only the first is still true.

Q: Should we stop investing in SEO? No, but the definition of SEO needs to expand. Traditional on-page optimization and link building still matter, but they need to be supplemented with structured data implementation, off-site reputation architecture, and entity-based brand presence strategies. Think of it as SEO evolving into a broader discipline of AI discoverability.

Q: How do we know if ChatGPT is recommending our brand? Test it directly: query ChatGPT, Claude, and Perplexity with the prompts your target customers are most likely to use when looking for a product or service like yours. Document what’s returned over time. Some enterprise analytics tools are beginning to track AI referral patterns as LLMs increasingly include source citations.

Q: What’s the timeline for AI agents to become mainstream in commerce? OpenAI’s Operator is already publicly available. Perplexity’s shopping integrations are live. Apple Intelligence agentic features are shipping with iOS updates. Early-adopter segments are active now, with mainstream penetration expected within 12–24 months based on current adoption curves.

Q: How do smaller brands compete if LLMs favor large, well-covered companies? Niche authority is a genuine competitive advantage. LLMs synthesize what credible sources say, and in niche categories, a smaller brand with deep coverage in relevant trade publications, strong review aggregator presence, and well-structured on-site data can outperform a larger brand with diffuse, generalist coverage.

What we’re building at Traversaal.ai

I run Traversaal.ai, where we build contextual AI agents for enterprise. One of the core problems we keep running into — across retail, SaaS, and e-commerce — is exactly what this article describes: brands that have strong products but zero presence in the AI layer.

What that looks like in practice: a company’s customers are asking ChatGPT or Perplexity for recommendations, and the brand isn’t showing up. Not because the product is bad, but because the infrastructure isn’t there. No structured data. No machine-readable catalog. No API that an agent can query. The brand is invisible to the systems making the recommendations.

We built our contextual agent platform specifically to close this gap. Brands deploy a search agent on their own website — powered by their product catalog, their knowledge base, their real inventory — and that agent becomes the interface between their customers and their products. It handles natural language queries, surfaces the right product for the right use case, and feeds structured, machine-readable data back into the broader AI ecosystem so the brand stays visible when external agents are making recommendations too.

If you’re a brand or an enterprise team trying to figure out where to start, the answer is usually a contextual search agent on your own site first — that’s the fastest path to both serving your existing customers better and building the infrastructure that makes you visible to external AI systems. We’d be happy to walk you through it.

Conclusion

This isn’t a prediction about where search is heading. It describes something already underway. AI has replaced Google as the first stop for product research for millions of people, and agentic systems are starting to complete transactions on their behalf. Brands treating this as a “wait and see” situation are going to compound the gap they’re already behind.

The uncomfortable part: most of what determines whether an AI recommends your brand isn’t in your marketing team’s control. It’s in your schema markup, your API accessibility, your Wikipedia entry, your G2 reviews, your presence in the publications that LLMs were trained on. The brand that wins in AI search isn’t necessarily the best brand. It’s the most machine-readable one.

Brands that fix their infrastructure now, while most competitors are still debating whether this is real, are building a discoverability advantage that will be very hard to close in 12 months. The window is still open. It won’t be forever.

💡 Want to share your work on my socials with my 15k+ audience?

If you build a project you are excited about, I will be too.

Trust me! I love seeing people build cool stuff. To share it, you can contact me here.

Sources

1. https://openai.com/blog/chatgpt

2. https://www.nytimes.com/2022/09/16/technology/gen-z-tiktok-search-engine.html

3. https://sparktoro.com/blog/in-2023-404-of-google-searches-end-without-a-click/

4. https://ahrefs.com/blog/google-ai-overviews/

5. https://housefresh.com/david-vs-digital-goliaths/

6. https://openai.com/operator

7. https://www.eidosmedia.com/resources/027f-178a5f3ff288-405e593af1ff-1000/eidosmedia-what-will-_chat-gpt-do-_to-search.png

8. https://futureskillsacademy.com/wp-content/uploads/2024/07/chatgpt-vs-google-search-engine.png

9. https://media.licdn.com/dms/image/v2/D5612AQG2Wtxyh9zUHg/article-cover_image-shrink_720_1280/B56ZZ.sxkPHUAI-/0/1745882389178?e=2147483647&v=beta&t=hLuewvZ1pxbxVtfCN7nr436SEJwV13DzBVKdPpYncVI

10. https://miro.medium.com/v2/resize:fit:640/1*YTQXNzOWR_h72qobG1XfMg.png

11. https://images.squarespace-cdn.com/content/v1/5e6542d2ae16460bb741a9eb/1774999297183-2U640Z5JGQV1942YWP2S/ChatGPT+Image+Apr+1%2C+2026%2C+12_20_44+AM.png

12. https://openai.com/blog/chatgpt)

13. https://www.nytimes.com/2022/09/16/technology/gen-z-tiktok-search-engine.html)

14. https://sparktoro.com/blog/in-2023-404-of-google-searches-end-without-a-click/)

15. https://ahrefs.com/blog/google-ai-overviews/)

16. https://housefresh.com/david-vs-digital-goliaths/)

17. https://openai.com/operator)

Did you enjoy this post?

Here are some other AI Agents posts you might have missed:

KV Caching and Speculative Decoding

A deep dive into Quantization: Key to Open Source LLM Deployments

Agents are here and they are staying

How Agents Think

Memory – The Agent’s Brain

Agentic RAG Ecosystem

Multimodal Agents

Scaling Agents: Architectures with Google ADK, A2A, and MCP

Fully Functional Agent Loop

Ready to take it to the next level? Check out my AI Agents for Enterprise course on Maven and be part of something bigger — join hundreds of builders developing enterprise-level agents.

Use this link to get $201 OFF!

You’re receiving this because you’re part of our mailing list. We don’t spam or sell your information. To unsubscribe, use the link below.

Welcome to Edition #32 of a newsletter that 15,000+ people around the world actually look forward to reading.

Want to learn about Claude Code?

Join my free session, sign up here

🎓 Want to up skill as AI Engineer?

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) May 25th

• Watch the free 4-session Agent Bootcamp playlist on YouTube

If you’ve ever watched a training curve plateau at 2am and wondered whether there’s a smarter way to do this, keep reading. What Andrej Karpathy is building might just change the way you think about ML research entirely.

Picture this: it’s 2:47am. You’re hunched over a terminal, watching loss curves that aren’t moving the way you’d hoped. You’ve manually tweaked the learning rate three times tonight, swapped out the optimizer once, and you’re now seriously considering whether adding a dropout layer was actually a terrible idea. This is the reality of ML research, a deeply human, deeply exhausting cycle of hypothesis, trial, and error. But what if that entire cycle could run itself while you slept?

Karpathy’s AutoResearch framework is built around that exact problem. AutoResearch is a self-directed, AI-powered experiment loop that automates the mechanical grind of ML research: writing code, running experiments, evaluating results, and deciding what to try next, all without a human pressing “go” between each step. It hands what Karpathy describes as a “team of ML engineers” to a single person with a single GPU.

In this article, you’ll learn exactly how the AutoResearch loop works, what decisions the AI makes on its own, how to set it up, and how to extend it with tools like Obsidian to build a living research notebook.

🔑 Key Takeaways

🔄 AutoResearch is a closed-loop ML experiment engine — an AI agent writes the code change, runs training, checks the numbers, and decides what to try next. No human in the loop between steps.

🖥️ It’s built for one GPU, not a data center — designed for individual researchers and small teams who can’t throw compute at the problem, but can throw time at it (overnight, while they sleep).

✅ Experiments get kept or thrown out based on one thing: did the number improve? — if it did, that becomes the new baseline. If not, it’s logged and discarded. Simple, rigorous, repeatable.

🎯 Your job changes from running experiments to designing them — you write the research objective and define the baseline. The agent handles the grinding part.

📓 Pair it with Obsidian and your notes actually become useful — every experiment gets logged in the same format, so three months later you can find what you learned instead of digging through half-finished notebooks.

Section 1: What is AutoResearch — and why is Karpathy behind it?

If you’ve been anywhere near the deep learning world in the last decade, you know the name Andrej Karpathy. Former Director of AI at Tesla, co-founder of OpenAI, creator of nanoGPT and micrograd, Karpathy has a track record of making intimidating concepts approachable. His GitHub repos aren’t just codebases; they’re teaching instruments, carefully built to lower the barrier of entry to cutting-edge ML. AutoResearch fits squarely into that lineage.

The problem AutoResearch addresses is one every ML researcher knows: the human-in-the-loop bottleneck. Traditional ML research cycles are slow not because compute is slow, but because humans are slow. You run an experiment, wait, analyze, think, modify, run again. A single researcher can realistically orchestrate a handful of meaningful experiments per week. AutoResearch collapses that timeline by offloading the orchestration to an AI agent.

Karpathy’s research philosophy and the problem he’s solving

Karpathy has consistently built tools that open up deep learning rather than restrict it. nanoGPT let anyone train a GPT-style language model in a few hundred lines of clean Python. micrograd made backpropagation tangible for students. His X/Twitter threads regularly break down frontier research into digestible intuitions. AutoResearch continues this philosophy, but with a new target: removing the most mechanical, repetitive parts of the research job entirely.

The philosophy isn’t “replace researchers.” It’s “free researchers from drudgery.” A human researcher brings creativity, domain intuition, and the ability to recognize when something surprising deserves deeper investigation. An AI agent brings tireless consistency, speed, and the ability to run a dozen parallel threads of reasoning without burning out. AutoResearch is built to combine those two. Source: https://github.com/karpathy

The core idea: a research loop that runs itself

AutoResearch is built around what we can call the “Karpathy Loop”, a four-stage automated cycle:

Write → Run → Evaluate → Iterate.

Subscribe now

A human researcher seeds the loop with an objective, and the AI agent takes it from there, cycling through hypothesis generation, experimentation, and analysis without waiting for human input between each turn. Think of it as the scientific method running on autopilot.

The closest analogy: a brilliant but tireless junior researcher who works 24 hours a day, never gets bored of running experiments, keeps meticulous notes, and flags anything interesting for your review in the morning. The human’s job shifts from doing the work to directing it.

Why now? The convergence that makes AutoResearch possible

Five years ago, this concept would have been aspirational at best. Three things have converged to make it real today.

First, large language models can now write, read, and debug functional ML code, not just generate plausible-looking syntax, but reason about what changes might improve model performance and why. Second, accessible GPU compute has democratized the hardware side: a single A100 or H100, rented by the hour via RunPod or Lambda Labs, gives an individual researcher serious experimental horsepower. Third, open-source ML frameworks like PyTorch have reduced boilerplate overhead so much that an AI agent can navigate and modify training scripts with relatively little friction.

These three forces didn’t exist in combination until recently. AutoResearch is a product of this specific moment.

Who should care about AutoResearch — and why

AutoResearch hits differently depending on your role. Below we break it down for two audiences: engineers and researchers who might actually run it, and Product Managers who work with teams that do. Same tool, very different implications.

Read more

Welcome to Edition #31 of a newsletter that 15,000+ people around the world actually look forward to reading.

Want to learn more about Claude Code?

Join my free session, sign up here

🎓 Want to up skill as AI Engineer?

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) May 25th

• Watch the free 4-session Agent Bootcamp playlist on YouTube

Every time Anthropic drops a new model, my inbox fills up with the same question: “Is it actually better, or just better on paper?” With Claude Opus 4.7 now in early access, I’ve been digging through the benchmark data, community threads, and real user feedback to give you a straight answer.

There’s a specific kind of dread that hits when a major model update lands.

You want the new capabilities, but you’ve been burned before by an “upgrade” that quietly broke your carefully tuned prompts. Claude Opus 4.7, Anthropic’s latest flagship release, arrives with exactly that dual energy. The early Opus 4.7 results are genuinely impressive in several areas, but they come with caveats worth discussing honestly.

This article walks through the headline benchmark numbers, the standout new features (high-resolution image support deserves its own section), and what real developers and power users are actually reporting from the field. We’ll also put Anthropic Claude Opus 4.7 next to its main competitors to give you the competitive context you need.

Benchmark gains don’t always map cleanly to real-world prompt compatibility. By the end, you’ll have a workflow-focused verdict, not a marketing recap.

🔑 Key Takeaways: - Benchmark gains are meaningful, but test your existing prompts before fully committing - High-resolution image support is a genuine standout new feature - Mixed real-world reception reveals the benchmark-vs-reality gap this article unpacks

What’s new in Claude Opus 4.7? A quick feature rundown

Before we get into the numbers, let’s make sure everyone’s on the same page. If you’re coming in cold and wondering what actually changed, here’s the fast briefing.

The headlining features at a glance

Anthropic Claude Opus 4.7 arrives with a focused set of upgrades rather than a sprawling feature dump. The changes that matter most for practical work:

• High-resolution image support, a qualitatively new capability, not just an incremental tweak

• Improved software engineering performance, stronger scores on multi-file code tasks and autonomous bug resolution

• Better tool-calling accuracy, more reliability in agentic pipelines and multi-tool orchestration

• Refined instruction-following, particularly in complex, multi-step system prompt scenarios

This isn’t a ground-up model rewrite. It’s targeted. That framing matters when you’re deciding how to approach migration.

High-resolution image support — why it matters

This is the one feature that actually changes what’s possible, rather than nudging existing scores upward. High-resolution image support means Claude Opus 4.7 can process high-DPI images with meaningfully more fidelity than its predecessor, opening up workflows that were previously blocked by image quality constraints.

An analyst uploading a high-DPI architectural diagram and asking Claude to extract precise material specifications from dense annotations, that workflow is now viable. The same goes for design review pipelines, dense medical document analysis, and visual reasoning tasks where fine detail carries real weight. For teams that have been forcing lower-resolution workarounds, this is a genuine unlock.

The Claude Opus 4.7 benchmark results in the multimodal domain are being driven partly by this new ceiling, not just incremental refinement of existing capabilities.

What didn’t change (and why that’s worth noting)

The context window size appears consistent with Opus 4.6, and the core pricing tier structure hasn’t shifted dramatically for most API users. The fundamental reasoning architecture carries significant continuity with its predecessor.

Opus 4.7 is an evolution, not a revolution, so if you were hoping for a context window doubling or a complete behavioral overhaul, temper those expectations. That architectural continuity also explains the prompt compatibility concerns we’ll get into: when a model changes enough to improve benchmarks but not enough to feel entirely different, the edge cases in your tuned prompts become unpredictable.

Claude Opus 4.7 benchmark results — breaking down the numbers

The numbers below are a starting point for interpretation, not a final verdict.

Software engineering and coding benchmarks (SWE-Bench, HumanEval)

The Claude Opus 4.7 benchmarks on software engineering tasks show the clearest improvement. On SWE-Bench, the industry-standard benchmark for evaluating autonomous code repair across real GitHub issues, Opus 4.7 shows a meaningful step up from Opus 4.6, with early reported scores suggesting improvements in the range of 8–12 percentage points depending on task category (Source: community-reported testing via r/ClaudeAI and independent evaluations). On HumanEval, which tests functional code generation, Opus 4.7 continues to perform competitively.

In practice, the model is meaningfully better at resolving multi-file bugs autonomously and handling complex refactoring tasks without losing thread. For developers using Claude in CI/CD pipelines or code review automation, this delta gets felt, not just measured.

One caveat worth flagging: SWE-Bench performance can vary based on how problems are scaffolded and whether tool use is enabled. Treat specific numbers as directional signals rather than absolute rankings, the trend is clear, but the precise magnitude is worth verifying against your own task distribution.

Tool-calling, agentic tasks, and multi-step reasoning

This is arguably where Claude Opus 4.7 makes its most compelling case. Tool-calling accuracy, the model’s ability to correctly invoke external tools, parse their outputs, and chain subsequent actions, reportedly improves in both reliability and handling of ambiguous tool signatures. For developers building AI-powered pipelines, this matters because errors compound: one bad tool call in step two of a five-step agent loop can cascade into a completely broken output.

Multi-step reasoning benchmarks also show improvement, particularly in tasks requiring the model to maintain goal-state awareness across many sequential actions. Early agentic task evaluations suggest Opus 4.7 recovers more gracefully from intermediate tool failures, an underrated capability that leaderboard scores don’t always capture well. If your work involves orchestrating Claude as an autonomous agent, this is the benchmark category to pay closest attention to.

Reading the leaderboard critically — what benchmarks don’t tell you

Benchmark scores are produced under controlled conditions with standardized prompts, evaluation rubrics, and fresh model states. Your production workflows are none of those things.

The AI industry keeps rediscovering the same pattern: a model update improves on standardized benchmarks while subtly shifting the behavior that your highly tuned few-shot prompts were relying on. The model hasn’t gotten worse in an absolute sense, it’s gotten better at the benchmark distribution, but your prompts were calibrated to a previous behavior profile. Treat leaderboard scores as a strong signal about capability ceiling, not as a guarantee of drop-in compatibility with your existing setup.

Opus 4.7 vs Opus 4.6 — is the upgrade actually noticeable?

The practical question isn’t “is Opus 4.7 good?” It’s “is Opus 4.7 better than what I’m already using, for what I actually do?”

Where Opus 4.7 clearly outperforms Opus 4.6

In side-by-side testing reported by early adopters, the performance gap is most consistently noticeable in complex multi-file coding tasks, agentic pipeline orchestration, and high-resolution visual input processing. Developers working on autonomous coding agents report that Opus 4.7 follows complex system prompts more reliably across longer task sequences, with less drift from the original instruction set. One recurring theme in Opus 4.7 vs Opus 4.6 comparisons: the newer model is less likely to “forget” constraints specified early in a long context when deep in tool-calling loops.

For users who have been pushing Opus 4.6 to its limits on multi-tool orchestration tasks, the improvement feels tangible rather than marginal. The high-res image capability is simply additive, if you needed it, you now have it; if you didn’t, it doesn’t affect you.

The regression problem — prompts that worked before may not work now

A meaningful subset of early users reports that prompt compatibility regressions are a real friction point in the migration. This shows up most often with highly specific few-shot examples, custom personas defined in system prompts, and writing-style instructions that depend on subtle behavioral calibration.

This pattern isn’t unique to Anthropic Claude Opus 4.7, it’s well-documented across major model updates industry-wide. But it shouldn’t be dismissed as a minor inconvenience for users whose workflows have been carefully optimized over months. Training-distribution shifts alter the probability landscape of the model’s outputs, even when the change is intended as an improvement. Prompt regressions are usually fixable, but they require deliberate testing and iteration rather than a seamless drop-in swap.

A simple migration checklist for Opus 4.6 users

If you’re planning to migrate, here’s a practical framework to minimize disruption:

1. Inventory your highest-stakes prompts, identify the system prompts and few-shot examples where output quality directly impacts your users or business processes

2. Run parallel tests, for each critical prompt, run identical inputs through both Opus 4.6 and Opus 4.7 and compare outputs systematically, not just impressionistically

3. Check instruction-following fidelity, pay specific attention to whether Opus 4.7 respects tone, format, and constraint instructions as reliably as its predecessor

4. Document regressions before migrating, create a log of any prompts showing degraded performance so you can iterate deliberately

5. Migrate in stages, consider running Opus 4.7 on new workflows first while keeping Opus 4.6 on proven, optimized pipelines until you’ve validated performance

6. Revisit your few-shot examples, if the model’s base behavior has shifted, your few-shot examples may need recalibration to the new prior

This checklist won’t eliminate migration friction, but it turns a potentially chaotic upgrade into a managed process.

Real-world feedback — what developers and power users are actually saying

Numbers are one thing. Here’s what practitioners are reporting from the field.

The positive reports — what’s winning people over

The most consistently praised aspects of Opus 4.7 in developer communities, including threads on r/ClaudeAI, Hacker News, and X/Twitter, center on agentic task reliability and coding performance. Developers building multi-step automation workflows report noticeably fewer mid-pipeline failures, with the model recovering more gracefully when tool calls return unexpected outputs. Several users specifically called out improvements in long coding task coherence, the model maintains awareness of earlier decisions over longer code generation sessions.

The high-resolution image processing capability has also generated genuine enthusiasm among users in design-adjacent roles and document-heavy workflows. Anyone who regularly works with detailed diagrams, dense PDFs, or high-DPI design assets is finding practical value here quickly. The Claude Opus 4.7 benchmarks in this area appear to reflect real workflow gains, not just lab conditions.

The frustrations — where users are hitting walls

The most frequently reported pain point is exactly what we previewed: prompt regressions in carefully tuned production workflows. Some users report that Claude 4.7 is more “opinionated” in certain contexts, occasionally pushing back on instructions or reinterpreting prompts in ways that Opus 4.6 handled more literally.

There’s also a smaller but vocal group reporting that certain writing and long-form reasoning tasks, areas where Opus 4.6 was particularly strong, feel marginally less reliable in the new model. Improving a model’s performance on the benchmark distribution sometimes means shifting behavior in ways that create friction for workflows calibrated to the previous version. The model isn’t worse in an absolute sense, but it may be different in ways that matter for your specific setup.

Cybersecurity and specialized use cases — a niche worth watching

For cybersecurity professionals and other specialized practitioners with high-stakes, precision-dependent requirements, the Opus 4.7 picture is still developing. Early reports from security researchers suggest improvement in code vulnerability analysis tasks, with the model showing better awareness of security-relevant code patterns in complex multi-file review scenarios.

That said, this is an area where the prompt regression concern hits hardest, security workflows often involve highly specific system prompts and constraint sets that have been carefully engineered over time. Threat modeling assistance and security-focused reasoning tasks are exactly the kind of specialized, precision-tuned use cases where parallel testing before migration isn’t optional. The community consensus in security-focused channels is still forming around Claude Opus 4.7’s software engineering improvements and what they mean for this domain.

Claude Opus 4.7 vs competitors — how it stacks up against GPT-5.4 and Gemini 3.1 Pro

No model evaluation is complete without competitive context. Here’s how Opus 4.7 positions against the field, task-specifically, not as a sweeping declaration.

Head-to-head on coding and tool-calling tasks

On software engineering and tool-calling benchmarks, Claude Opus 4.7 holds its own. Early community comparisons suggest Opus 4.7 trades blows with GPT-5.4 depending on task type, Opus 4.7 appears to edge ahead on complex multi-file autonomous repair tasks, while GPT-5.4 remains competitive on single-function generation and certain API-intensive scenarios. Against Gemini 3.1 Pro, Opus 4.7 shows a clearer advantage in instruction-following fidelity within tool-calling chains.

Definitive head-to-head benchmarks for these specific model versions are still accumulating in the community, so treat these comparisons as directional rather than conclusive. What’s clear is that Claude Opus 4.7 benchmarks place it firmly in the top competitive tier for coding and agentic tasks, not a clean winner across the board, but a serious competitor in the scenarios that matter most to developers.

Multimodal capabilities — imaging and visual reasoning compared

The new high-resolution image support is where Opus 4.7 makes its most significant competitive move. Gemini 3.1 Pro has long been the standard for document-heavy visual workflows, and Opus 4.7 now closes that gap meaningfully for a broader range of use cases. For dense document analysis, high-DPI diagram interpretation, and design review workflows, Opus 4.7 now competes seriously with Gemini 3.1 Pro in a way its predecessor couldn’t.

GPT-5.4 remains strong in open-ended visual reasoning and natural-image interpretation. Opus 4.7’s advantage in the multimodal space is most pronounced in text-dense, high-resolution document scenarios rather than general image understanding tasks. If your use case centers on processing detailed technical documents, architectural drawings, or information-dense visual formats, Anthropic Claude Opus 4.7 has made a genuine competitive leap.

Choosing the right model for your use case

Rather than declaring a universal winner, here’s a simple decision framework:

Use CaseRecommended Starting PointAgentic coding pipelines and multi-file bug resolutionOpus 4.7, strong advantageDocument-heavy visual analysis and high-res image workflowsOpus 4.7 or Gemini 3.1 Pro, test bothOptimized existing GPT-5.4 production pipelinesStay on GPT-5.4 unless specific Opus features are neededGeneral-purpose reasoning and writing tasksEvaluate all three against your specific promptsSecurity-focused code analysisOpus 4.7 warrants testing, but validate carefully

The honest answer is that no single model dominates across every task category. Your ideal choice depends on what you’re actually building.

Frequently asked questions

Is Claude Opus 4.7 significantly better than Opus 4.6? For software engineering, agentic tasks, and high-resolution image workflows, yes, the improvements are meaningful and consistently reported. For other task types, the difference is more situational. If those core areas are central to your work, the upgrade is worth it.

Will my existing Opus 4.6 prompts work with Opus 4.7? Most will, but some won’t without adjustment. Highly tuned few-shot examples, specific persona instructions, and precision-calibrated system prompts are the most likely to show behavioral shifts. Run parallel tests before fully migrating.

How does Claude Opus 4.7 compare to GPT-5.4 for coding tasks? On complex multi-file autonomous coding tasks, Opus 4.7 appears competitive with or slightly ahead of GPT-5.4 in early comparisons. Single-function generation remains close. Test both on your actual task distribution, that’s the only comparison that matters for your work.

What is the high-resolution image support feature specifically? It enables Claude Opus 4.7 to process high-DPI images with significantly greater fidelity than Opus 4.6, supporting detailed document analysis, dense diagram interpretation, and design review workflows that require preserving fine visual detail.

Should cybersecurity professionals upgrade immediately? Not without testing. The model shows promise for security-related code analysis, but given the precision requirements of security workflows and the potential for prompt regressions, a careful parallel testing phase before migration is strongly advisable.

Is the context window larger in Opus 4.7? Based on current available information, the context window is consistent with Opus 4.6. If an extended context window is your primary need, that shouldn’t drive your upgrade decision.

Conclusion — the honest verdict on Claude Opus 4.7

Opus 4.7 is a genuine, meaningful upgrade, particularly for developers and power users whose work centers on software engineering, agentic task orchestration, and visual document workflows. The benchmark gains in these areas reflect real capability improvements that users are feeling in practice. The high-resolution image support opens up workflows that simply weren’t viable before.

The mixed community reception is also a useful reminder: leaderboard scores are a starting point, not a verdict. The most important evaluation you can run is against your own prompts, your own workflows, and your own definition of quality. That applies every time a new model drops, not just this one.

For most developers evaluating Anthropic Claude Opus 4.7, the upgrade is worth making. But if your workflows are heavily optimized around Opus 4.6, run the migration checklist before you flip the switch and test in parallel before you commit. The gains are real, confirm they apply to your specific situation before retiring a setup that’s been working reliably.

Ready to test Opus 4.7 for your own workflows?

Start with your three highest-stakes prompts, run them in parallel with Opus 4.6, and let the results guide your decision. If you found this breakdown useful, share it with your team, and drop your own early testing results in the comments. The community verdict is still forming, and your data point matters.

I also pressure tested Opus 4.7 in my previous post:

Sources

1. https://lookaside.instagram.com/seo/google_widget/crawler/?media_id=3876680872979608079

2. https://osu-wams-blogs-uploads.s3.amazonaws.com/blogs.dir/2115/files/2024/12/A.I.-Model.png

3. https://preview.redd.it/claude-opus-4-7-benchmarks-v0-e0equiu3bkvg1.png?width=1080&crop=smart&auto=webp&s=9ae2b8d255bde017c0708326bc0d7d0c854eb08e

4. https://media.licdn.com/dms/image/v2/D5612AQGzcAlzaRY_Qg/article-cover_image-shrink_720_1280/B56ZYRTGzsHQAQ-/0/1744046945643?e=2147483647&v=beta&t=0FT81vCIju8oPqMP0352Lnmk0s8bKaHTIE_9FA7Lhu0

5. https://storage.ghost.io/c/7d/70/7d70d59c-7408-4583-b44d-98a43cdfa8fd/content/images/size/w2000/2025/10/discover-ai-communities.jpg

6. https://miro.medium.com/v2/resize:fit:1400/1*JzVc5cNukZMsM_CTiAWvgg.png

💡 Want to share your work on my socials with my 15k+ audience?

If you build a project you are excited about, I will be too. Trust me! I love seeing people build cool stuff. To share it, you can contact me here.

Did you enjoy this post?

Here are some other AI Agents posts you might have missed:

KV Caching and Speculative Decoding

A deep dive into Quantization: Key to Open Source LLM Deployments

Agents are here and they are staying

How Agents Think

Memory – The Agent’s Brain

Agentic RAG Ecosystem

Multimodal Agents

Scaling Agents: Architectures with Google ADK, A2A, and MCP

Fully Functional Agent Loop

Ready to take it to the next level? Check out my AI Agents for Enterprise course on Maven and be part of something bigger — join hundreds of builders developing enterprise-level agents.

Use this link to get $201 OFF!

You’re receiving this because you’re part of our mailing list. We don’t spam or sell your information. To unsubscribe, use the link below.

Welcome to Edition #31 of a newsletter that 15,000+ people around the world actually look forward to reading.

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing. This isn’t that. You’ll find raw, honest, human insight here, the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

🎓 Want to up skill as AI Engineer?

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) May 5th

• Watch the free 4-session Agent Bootcamp playlist on YouTube

I’ve been running the same five PM prompts through every major Claude release for the past year. When Opus 4.7 dropped and Reddit immediately declared it a regression, I went back to my test bench. Here’s what the data actually shows.

Scroll far enough and you’d think Anthropic shipped a downgrade. Almost none of it came from product managers using Claude for structured, deliverable-focused work.

So I went back to my test bench.

Same five PM prompts, both models, fresh sessions with identical context and no fine-tuning, raw first outputs side by side.

The tasks: PRD writing, exec summary writing, RICE prioritization, user research synthesis, and GTM planning. What follows is the actual output comparison.

One thing I didn’t see covered anywhere else: 4.7’s mid-output self-correction behavior. It shows up in the RICE scoring and user research tasks and it changes how you should think about using it in agentic workflows.

We’ll get to that.

Key takeaways

• 4.7 is faster and more structured, but whether that’s better depends entirely on the task.

• For tables, checklists, and stakeholder-ready docs, 4.7’s formatting defaults save you time. For narrative writing, they work against you.

• 4.7 catches its own reasoning errors mid-output. 4.6 doesn’t. For multi-step workflows, that matters more than surface polish.

• 4.6 is still the better choice for PRD writing from scratch, it scored a perfect 45/50 vs 4.7’s 35/50.

• If you’re launching AI features in regulated markets, 4.7 already knows about the EU AI Act. 4.6 doesn’t mention it.

The “regression” controversy and why PMs should read it differently

What the community is saying

The complaints about Opus 4.7 cluster around a few consistent patterns. Writers say long-form prose has become more mechanical. The model reaches for bullet points and headers where earlier versions held a flowing narrative. Coders say explanatory comments feel more templated. General users describe the experience as “corporate,” as if every response is being formatted for a slide deck nobody asked for.

These aren’t manufactured grievances. If you use Claude primarily for creative writing, exploratory conversation, or nuanced code explanation, the default output style has shifted in a way that works against you. The representative Reddit complaint reads something like: “It used to feel like talking to a thoughtful colleague. Now it feels like getting a memo.” For those use cases, that’s worth acknowledging.

The users driving the loudest backlash tend to be novelists, hobbyist coders, and general-purpose AI users. Their evaluation surface is fundamentally different from a PM who needs a PRD reviewed by six engineers in Confluence by Thursday.

The Benchmark setup

Claude Battle: Opus 4.6 vs Opus 4.7 for Product Managers

A head-to-head benchmark of Claude Opus 4.6 and Opus 4.7 on 5 core PM tasks, with Claude-as-judge quality scoring.

Github Link

What it contains

Two scripts:

1. claude_battle.py — runs both models on identical PM prompts and saves raw outputs + timing

2. battle_eval.py — uses Claude Opus 4.7 as an independent judge to score both models on 5 PM-specific quality dimensions

The results/ folder contains the actual outputs and scores from the April 17, 2026 run (the day Opus 4.7 launched).

Why PM workflows are a different evaluation surface

PM deliverables have one thing in common that creative tasks don’t: the primary audience isn’t the person who wrote them. A PRD gets reviewed by engineering leads, design, and QA. An exec summary lands in a C-suite inbox. A RICE-scored backlog gets pulled into JIRA. The output needs to be scannable, formatted for stakeholder review cycles, and structured to slot into existing templates.

The formatting shift that feels like over-engineering in casual chat is, in that context, a feature. When your audience reads in tables and headers, a model that defaults to tables and headers isn’t regressing. It’s calibrating.

The 4.6 vs 4.7 question isn’t which model is better overall. It’s which task you’re running.

How this benchmark was structured

Every test followed the same protocol: identical prompt text, fresh session with no prior context, raw first output captured without regeneration. No fine-tuning, no system prompts, no prompt engineering. I wanted to evaluate default behavior, because most PMs aren’t prompt engineers and their first-pass output is what actually matters under deadline.

Evaluation criteria across all five tasks: output structure, information density, formatting fit, reasoning visibility, and consistency. Where the outputs diverged, I’ve included direct excerpts and comparisons. Where one model clearly won, I’ve said so.

Quality scorecard — the actual scores

Each response was evaluated by Claude Opus 4.7 acting as an independent judge, scoring five PM-specific quality dimensions from 1-10 (max score per task: 50).

Overall scores:

Scores by dimension (average across all 5 tasks):

The two dimensions that split the models: Actionability (4.7 +1.2 avg) and PM Framework Accuracy (4.6 +0.6 avg). Opus 4.7 produces outputs you can act on immediately. Opus 4.6 applies frameworks like RICE with more precision when it doesn’t hit token limits.

Task 1: PRD writing (the one task 4.6 clearly wins)

What each model produced

The prompt: “Write a concise PRD for adding a waitlist to our B2B SaaS product so we can manage demand during our closed beta. Include problem statement, goals, user stories (3), success metrics, out of scope, and risks.”

Opus 4.6 produced flowing prose with strong narrative structure. The problem statement was rich with context, goals were laid out in a table, and the risk section proactively flagged GDPR/CCPA consent and data retention as a launch blocker. It read like a document written by a PM who has shipped products before.

Opus 4.7 produced a well-structured output with a clear hierarchy and checkbox-formatted user stories, then hit its token limit and cut off mid-risk statement. It literally stopped mid-sentence.

Side-by-side output analysis

The 4.6 output scored 9/10 across all five quality dimensions. 4.7 scored 6/10 on Completeness and Clarity because it didn’t finish.

PRD verdict

4.6 wins. A PRD that doesn’t finish its risk section is going to come back and bite you in sprint planning. The completeness gap here isn’t subtle.

One important note: this is a token ceiling problem, not a capability problem. If you’re using 4.7 for PRDs, set max_tokens higher than the default. That single change likely closes the gap.

Task 2: Exec summary (where 4.6 nearly takes back a second win)

The test

The prompt: “Rewrite the following engineering spec as a 3-bullet executive summary for a non-technical VP. Focus on business impact, not implementation.”

This task rewards different output characteristics than PRD writing. A good exec summary for a C-suite audience needs rhetorical craft: a narrative arc that moves from problem to stakes to solution, and a tone that feels strategic rather than operational. Tables and bullet points work against you when you need to build urgency and make a business case feel compelling.

Where each model landed

Opus 4.6 translated an 800ms-to-120ms latency improvement into “~85% reduction in our slowest database response times.” Accurate, but it reads like a metrics report.

Opus 4.7 framed the same number as “key user actions will feel roughly 6x snappier.” That’s the version a VP can actually repeat in a board meeting. It also added “removes our current scaling ceiling” as business context, where 4.6 stayed closer to the technical framing.

The scores were close: 4.6 scored 42/50, 4.7 scored 45/50. The gap was almost entirely in Actionability and Business Reasoning.

Exec summary verdict

4.7 wins, but not by much. On default output, 4.7 frames impact the way a non-technical stakeholder reads it. If you need pure narrative prose with no structure at all, add one line to your prompt: “Write this as flowing prose with no bullet points, the audience reads for story not structure.” That largely fixes 4.7’s formatting instinct.

Tasks 3 and 4: RICE prioritization and user research (where the reasoning gap opens up)

RICE scoring

The prompt gave 6 features and asked for RICE scores, rankings, and rationale for the top 2 picks. Realistic mixed backlog: dark mode, Slack integration, CSV export, mobile app, onboarding checklist, team permissions (RBAC).

Opus 4.6 produced a clean RICE table. It also contained a problem: the scoring went feature-by-feature through the narrative, making the ranking hard to scan, and the output cut off before explaining the second top pick. It looked authoritative and wasn’t finished.

Opus 4.7 produced a summary table with all six features ranked in one view and RICE scores calculated correctly. Partway through it caught something and inserted: “Correcting: CSV export (300) actually ranks 3rd, Slack (187) 4th.” It noticed an inconsistency in its own ordering and fixed it inline. That’s the self-correction behavior.

For a PM who runs RICE scoring regularly, that kind of visible error-catching is worth a lot. A silent error in a prioritization table is the kind of thing that ships the wrong feature.

User research synthesis

The prompt: 5 raw B2B customer interview quotes. Synthesize into top 3 pain points, a Jobs to Be Done statement, and a recommended next step.

Both models correctly identified the top pain points (data export, access controls, onboarding). The difference was in what came after.

Opus 4.6 stopped at the diagnosis. Clear pain points, crisp JTBD framing, but no success metrics for whatever the team ships next.

Opus 4.7 added a recommended quarterly sequence: fix export first (highest frequency), then permissions (highest churn risk), then onboarding. It also proposed how to measure whether each fix worked. That’s the output you can bring to a sprint planning session without additional work.

Verdict for multi-step tasks

Both tasks point to the same thing: 4.7 shows its work. Whether it’s catching a ranking error or sequencing recommendations with metrics, the outputs are more auditable. In agentic workflows where Claude runs through multiple steps, a visible self-correction in step two is more useful than a polished output with a silent error buried inside it.

Task 5: GTM planning (the AI-era awareness gap)

The prompt

“Create a go-to-market launch checklist for a new feature: ‘AI-powered meeting summaries’ being added to an existing B2B SaaS productivity tool with 2,000 paying customers.”

This prompt was designed to surface AI-era awareness, specifically whether either model would flag the regulatory and operational concerns that come with shipping an LLM-powered feature.

What 4.6 missed

Opus 4.6 produced a solid three-phase GTM checklist covering internal readiness, customer communication, and success measurement. The risk section identified standard enterprise SaaS risks: competitive displacement, low adoption, integration complexity. It didn’t mention the EU AI Act, GDPR Article 22 implications for automated decision-making, or LLM cost monitoring as an operational concern.

For a PM launching an AI product in Europe in 2026, that’s a meaningful gap in the checklist.

What 4.7 added

Opus 4.7’s checklist included explicit items for EU AI Act compliance (classifying the feature’s risk tier), GDPR data residency requirements for model inference, and a note to set usage caps to prevent runaway LLM spend. It recommended a phased rollout sequenced against the EU AI Act’s compliance timelines.

Training cutoff explains part of it, but 4.7 also surfaces these requirements in the context of PM decisions, not just as a legal checklist bolted on at the end.

GTM verdict

4.7 wins, especially for any PM working on AI features or in regulated markets. If your GTM plan doesn’t touch AI or Europe, the gap narrows considerably. But if it does, 4.7 is the only one of the two that doesn’t require you to manually fill in what it missed.

Frequently asked questions

Is Claude Opus 4.7 better than 4.6 for product managers?

Depends on the task. For structured deliverables, RICE scoring, GTM planning, and multi-step agentic workflows, 4.7 is the stronger choice. For PRDs from scratch and persuasive narrative writing, 4.6 holds its own.

What is the Claude Opus 4.7 benchmark performance for PM tasks?

Across five tasks, 4.7 won four (backlog prioritization, exec summary, user research synthesis, GTM planning) and lost one (PRD writing, where 4.6’s completeness was decisive). Overall scores: 202/250 for 4.7 vs 198/250 for 4.6.

Why are people saying Claude Opus 4.7 is a regression?

The regression complaints are largely valid for creative writing and conversational use. The formatting shift toward tables and structured output works against those use cases. For PM deliverables that need structured, stakeholder-ready output, the same behavior is often a feature.

What is mid-output self-correction and why does it matter?

It’s when Claude 4.7 identifies an inconsistency during generation and explicitly flags and corrects it in the same output. For agentic workflows where errors compound across iterations, this is more useful than polished output with silent logical errors inside.

Should I upgrade from Claude Opus 4.6 to 4.7?

Use it as a routing rule: if the output is going into a structured template reviewed by stakeholders, use 4.7. If it’s a narrative draft you’ll heavily edit yourself or a persuasive doc for a non-technical audience, stay on 4.6 or add an explicit prose instruction.

Conclusion

The community backlash against Opus 4.7 is real, and for the use cases driving it, it’s not wrong. But it’s not the full picture for PMs.

Across five core tasks, 4.7 won four. It traded narrative fluency for structured output, and surface polish for visible reasoning. For PM workflows, most of those trade-offs work in your favor. The one exception is PRD writing from scratch, and even that comes down to a token limit you can adjust.

The self-correction behavior is the thing worth watching most closely. Models that show their work integrate more cleanly into multi-step PM workflows than models that return polished outputs with silent errors. For agentic use, that difference matters more than surface polish.

Route your structured deliverables through Opus 4.7. Keep 4.6 for narrative writing. And if you’re building AI features or operating in regulated markets, 4.7 already knows the compliance landscape you’re navigating.

Have you run your own comparisons between 4.6 and 4.7?

Drop your findings in the comments, particularly if you’ve stress-tested the self-correction behavior on longer agentic chains. I’m building a follow-up test on five-step roadmap generation loops and the more real-world data points I have, the better.

Sources

1. https://images.ctfassets.net/4zfc07om50my/5ZKGwdbKk38ZMO9B2sjE6k/246b981eb5f52d2fef5fd00d2e912da0/Guide-Pros-Cons-PRD.png

2. https://weaviate.io/assets/images/agentic-search-workflow-2b8e44550b28c263f52c18e1d8b7ca1f.jpg

3. https://www.reddit.com/r/ClaudeAI/comments/1soew2x/claude_opus_47_won_69_of_100_blind_evals_against/

4. https://www.reddit.com/r/ClaudeCode/comments/1soes75/i_benchmarked_opus_46_vs_47_on_organizational/

💡 Want to share your work on my socials with my 15k+ audience?

If you build a project you are excited about, I will be too. Trust me! I love seeing people build cool stuff. To share it, you can contact me here.

Did you enjoy this post?

Here are some other AI Agents posts you might have missed:

KV Caching and Speculative Decoding

A deep dive into Quantization: Key to Open Source LLM Deployments

Agents are here and they are staying

How Agents Think

Memory – The Agent’s Brain

Agentic RAG Ecosystem

Multimodal Agents

Scaling Agents: Architectures with Google ADK, A2A, and MCP

Fully Functional Agent Loop

Ready to take it to the next level? Check out my AI Agents for Enterprise course on Maven and be part of something bigger — join hundreds of builders developing enterprise-level agents.

Use this link to get $201 OFF!

You’re receiving this because you’re part of our mailing list. We don’t spam or sell your information. To unsubscribe, use the link below.

Welcome to Edition #30 of a newsletter that 15,000+ people around the world actually look forward to reading.

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing. This isn’t that. You’ll find raw, honest, human insight here — the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

🎓 Want to up skill as AI Engineer?

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) May 5th

• Watch the free 4-session Agent Bootcamp playlist on YouTube

If you’ve been watching the LLM efficiency space lately, you already know that every few months a paper lands claiming to solve the memory problem, and every few months, the engineering community has to do the work of separating signal from noise. This one is worth reading carefully.

TurboQuant : does KV cache compression actually deliver on its promises?

Google drops a research paper promising to compress KV caches to 3-bit precision with zero accuracy loss, and the AI community collectively raises an eyebrow.

It’s the kind of headline that spreads fast on AI Twitter/X, 8x throughput, 6x memory reduction, no degradation, and it’s exactly the kind of claim that deserves more than a retweet before anyone starts restructuring their GPU infrastructure around it.

TurboQuant is Google’s proposed approach to one of the most consequential unsolved problems in production large language model efficiency: what do you do when the memory your model needs during inference grows faster than your hardware budget? KV cache compression is where that question gets answered or doesn’t, and TurboQuant claims to have cracked it. We’ll look at the benchmarks, surface an active academic controversy that most mainstream coverage has ignored entirely, and pull in what real engineers are actually saying, so you can decide whether this is a genuine breakthrough, an incremental improvement dressed up in bold marketing language, or something more complicated than either.

🔑 Key Takeaways

• 📉 TurboQuant compresses KV caches to 3-bit precision — targeting one of LLM inference’s most painful bottlenecks: GPU memory consumed by the key-value cache growing faster than hardware budgets.

• ⚡ The headline numbers are striking but contested — 8x throughput and 6x memory reduction on H100s, but measured under specific conditions that may not generalize to typical production workloads.

• ⚠️ The RaBitQ controversy is the real story — TurboQuant is alleged to have minimized attribution to prior independent research, used a single-core CPU vs GPU comparison to disadvantage a competitor, and dismissed RaBitQ’s theoretical guarantees without explanation.

• 📚 QTIP did this two years earlier — random rotation plus near-optimal distortion quantization was already published and deployed in ExLlamaV3 before TurboQuant arrived; the novelty claims deserve scrutiny.

• 🔬 No independent replication yet — zero accuracy loss at 3-bit is the flagship claim, but validation benchmarks may not be independent of calibration data; production code isn’t publicly available to verify.

• 💡 The bottom line — interesting engineering, real controversy, not yet validated. Watch OpenReview, watch the community, don’t restructure infrastructure around preprint numbers.

What is TurboQuant and why does it matter?

To understand why TurboQuant generated so much immediate attention — and so much controversy — you need to understand the problem it’s trying to solve and what it actually claims to do.

The KV cache problem — why memory is the bottleneck nobody talks about enough

The key-value (KV) cache is the memory structure that allows transformer models to “remember” everything they’ve seen in a conversation or document during inference. Without it, the model would have to reprocess the entire context from scratch on every token generation step. The problem is architectural and unforgiving: KV cache memory grows linearly with both sequence length and batch size, which means serving real users at scale pushes it past comfortable limits fast.

To make this concrete, consider a 70B parameter model serving 100 concurrent requests at a 32,000-token context window. At standard BF16 precision, the KV cache alone can consume well over 100GB of GPU memory, potentially more than the model weights themselves. That’s not a theoretical edge case; that’s a Tuesday afternoon for anyone running a serious inference cluster.

TurboQuant KV cache compression targets exactly this bottleneck, which is why the research got immediate attention from practitioners who have been living inside this constraint for years. The promise of AI memory quantization that doesn’t compromise output quality is one of the most commercially valuable things a research team could deliver right now.

What Google claims TurboQuant actually does

TurboQuant’s core mechanism applies quantization to KV cache tensors during inference, compressing the floating-point representations from 16-bit (BF16 or FP16) down to as low as 3 bits per value. The “Turbo” in the name refers to a novel transform applied to the KV cache tensors before quantization, the authors argue this transform smooths the distribution of values, making them more amenable to aggressive compression without the rounding errors that normally cause quality degradation.

The headline numbers from the paper are striking: an 8x throughput improvement, a 6x memory reduction on H100 GPUs, and the flagship claim of zero accuracy degradation at 3-bit compression (Source: https://arxiv.org/abs/2504.19874). These are extraordinary claims by any standard in quantization research. The benchmark conditions that produced these numbers matter enormously, and we’ll examine them closely in the next section, the Google TurboQuant benchmark methodology is where the real scrutiny belongs.

The Google DeepMind pedigree — does it add credibility or pressure?

Google and DeepMind papers carry genuine weight in the AI research community, the organizations have produced foundational work on transformers, quantization, and systems optimization that the entire field builds on. That institutional credibility is real, and it would be dishonest to pretend it’s irrelevant when evaluating TurboQuant’s claims. But prestige also creates a specific risk: high-profile papers from prominent institutions tend to get amplified before they get scrutinized.

The TurboQuant preprint spread rapidly through AI Twitter/X and research Slack channels within days of appearing, with many posts treating the benchmark numbers as established facts rather than claims requiring validation. At the time of writing, the paper has not completed formal peer review at a top-tier venue, a distinction that matters because peer review, despite its own limitations, is the mechanism by which methodological concerns get formally surfaced. The gap between preprint hype and post-review reality is a recurring pattern in ML research, and TurboQuant is currently sitting in that gap.

Breaking down the technical claims — the 8x speed and 6x memory numbers under the microscope

Headline benchmark numbers from research papers almost always require unpacking — the conditions that produced them matter as much as the numbers themselves. Here’s what the 8x throughput, zero accuracy loss, and 6x memory claims actually mean when you read them carefully.

Understanding the 8x throughput claim — what’s actually being measured?

“8x throughput“ can mean several different things depending on what’s actually being counted. Throughput in LLM inference can refer to tokens per second, requests per second, or time-to-first-token, and the relationship between these metrics shifts dramatically depending on sequence length, batch size, and whether you’re measuring the prefill or decoding phase. The Google TurboQuant benchmark conditions that produced this number deserve careful reading before anyone generalizes from them.

If the 8x figure was measured at a specific batch size and context length that maximizes KV cache memory pressure, it may not translate proportionally to shorter-context, lower-concurrency workloads, which are a large fraction of real production deployments. This isn’t a gotcha; it’s standard benchmark hygiene. The honest question is whether the paper clearly specifies these conditions and whether those conditions reflect typical practitioner workloads, or whether they were selected because they put the technique in the best possible light. TurboQuant accuracy loss tradeoffs also interact with throughput measurements in ways that need to be disentangled before the 8x number can be taken at face value.

The “zero accuracy loss” claim — what does zero actually mean?

The “zero accuracy loss at 3-bit compression” claim is arguably the most audacious in the paper, and it deserves its own focused examination. In quantization research, accuracy is typically measured using perplexity on standard benchmarks like WikiText-2, or on downstream task performance metrics from evaluations like MMLU or HellaSwag. The consensus from years of research is that aggressive quantization below 4 bits introduces meaningful degradation that gets harder and harder to mitigate.

The critical methodological question is straightforward: were the benchmarks used to validate zero accuracy loss independent from the datasets used to tune the quantization parameters? If the quantization scheme was calibrated on data drawn from the same distribution as the validation benchmarks, the accuracy results may be measuring how well the method fits its calibration distribution rather than how well it generalizes. This is a standard question in quantization research, and the answer changes how much weight we should give to the TurboQuant accuracy loss numbers. Does TurboQuant actually achieve zero accuracy loss at 3-bit compression in genuinely out-of-distribution settings? That remains open.

The 6x memory reduction — real-world implications for GPU infrastructure

A genuine 6x memory reduction in KV cache would be transformative for GPU infrastructure economics. At that level of compression, a single H100 could theoretically serve workloads that previously required multiple GPUs to handle KV cache memory, which translates directly into lower cost-per-token and longer achievable context windows without hardware upgrades. These are concrete possibilities with real impact on how LLMs are deployed at scale.

The important caveat for practitioners is that KV cache compression and model weight quantization are separate techniques addressing different memory consumers. KV cache quantization reduces inference-time memory pressure, while weight quantization like GPTQ or AWQ reduces the memory needed to store the model. TurboQuant addresses only the former. Understanding what combination of techniques the benchmark is using matters for infrastructure planning, because large language model efficiency improvements don’t stack linearly, the bottleneck shifts as you address each component.

The academic controversy nobody is covering — OpenReview’s methodological critiques

Read more

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing. This isn’t that. You’ll find raw, honest, human insight here — the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

🎓 Want to up skill as AI Engineer?

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) April 8

• Watch the free 4-session Agent Bootcamp playlist on YouTube

If you’ve ever felt like LLM deployment is a black box that everyone else somehow understands except you — this one’s for you. Let’s open it up together.

You’ve done the hard part. You fine-tuned a model, it runs beautifully in your notebook, loss curves look great, outputs are solid. Then you try to serve it to real users — and suddenly you’re drowning in questions you can’t answer. Why is latency spiking under load? Why is VRAM usage ballooning? What is batching actually doing to your throughput? If this sounds familiar, you’re in good company.

Here’s the frustrating reality: most developers treat LLMs as black boxes — and that works fine until it doesn’t. Full vLLM is an incredibly powerful inference engine, but its 100,000+ lines of code aren’t designed for learning. They’re designed for performance at scale. Trying to understand LLM inference optimization by reading vLLM’s source is like trying to learn how a car engine works by dissecting a Formula 1 race car — technically accurate, practically overwhelming.

That’s where this nano vLLM tutorial comes in. nano-vLLM is a ~1,000-line Python reimplementation of vLLM’s core ideas — KV caching, PagedAttention, continuous batching — written to be read, not just run. In this article, you’ll get a guided tour of what nano-vLLM is, why each of its core mechanisms exists, how to trace those mechanisms in the code, and how that understanding transfers directly to production-grade open source LLM deployment.

🔑 Key Takeaways

• 📦 nano-vLLM is a learning tool, not a production system — its minimal codebase (~1,000 lines) deliberately strips away complexity so you can trace exactly how a real inference engine works, making it the fastest path to understanding what full vLLM is doing under the hood.

• ⚡ KV caching is the single biggest lever for LLM inference speed — it stores and reuses key-value pairs from the attention mechanism instead of recomputing them on every token.

• 🧩 PagedAttention solves the memory fragmentation problem — borrowing from OS virtual memory, it allocates attention memory in discrete blocks to dramatically improve GPU utilization.

• 🔄 Continuous batching keeps GPUs busy by treating every token step as a scheduling opportunity — slashing idle time compared to static batching.

• 🧠 Studying a minimal reimplementation accelerates your intuition — the mental models transfer directly to vLLM, TGI, and TensorRT-LLM in production.

What Is nano-vLLM and Why Should You Care?

Before we get into the mechanics, let’s establish exactly what nano-vLLM is — and just as importantly, what it isn’t. Getting this framing right separates developers who use it as a powerful learning accelerator from those who dismiss it as a toy.

The Problem With Learning From Production Codebases

Production LLM inference engines like vLLM, Text Generation Inference (TGI), and TensorRT-LLM are extraordinary pieces of engineering. They handle distributed tensor parallelism, custom CUDA kernels, fault-tolerant API layers, and dozens of edge cases that only surface at scale. But that same depth makes them nearly impenetrable when you’re trying to understand the core ideas rather than just use the tool.

Open vLLM’s codebase hoping to understand how KV caching works and you won’t find a clean function — you’ll find the concept scattered across abstraction layers, tangled up with GPU memory management code, scheduling logic, and async runtime machinery. It’s not that the code is bad; production code optimizes for reliability and performance, not for teaching. The Formula 1 analogy holds: it’s the pinnacle of mechanical engineering, but you wouldn’t hand one to a student learning how internal combustion works.

The result is that most ML engineers learn theory from papers and blog posts, then use vLLM as a black box — never really bridging the two. That gap costs you every time you need to debug a throughput problem, tune a deployment, or evaluate whether a new inference optimization actually matters for your workload.

Enter nano-vLLM — A Minimal Reimplementation Built for Clarity

nano-vLLM was built to close that gap. It’s a deliberately minimal Python reimplementation of vLLM’s core scheduling and memory management concepts — written so you can sit down and trace the full execution path from request intake to token output in a single afternoon. The entire codebase is roughly 1,000 lines, not because it cuts corners on the ideas, but because it strips away everything that isn’t the idea.

The project lives on GitHub (Source: https://github.com/GeeeekExplorer/nano-vllm) and explicitly frames itself as educational. Every major concept — KV cache allocation, block tables, the scheduling loop — is implemented in clean, readable Python without the performance scaffolding that would obscure it in production code. The ~1,000-line footprint isn’t a limitation; it’s the entire point.

nano-vLLM vs. Full vLLM — What’s Kept, What’s Stripped Away

Here’s a clear breakdown of what nano-vLLM implements versus what it intentionally leaves out:

Everything nano-vLLM omits, it omits deliberately. The goal isn’t to serve a production workload — it’s to give you a clean mental model of what the full system is actually doing beneath its abstractions.

The Core Problem That Makes LLM Inference Hard

To appreciate why any of these optimizations exist, you first need to feel the pain they solve. Let’s spend a moment on why serving LLMs efficiently is genuinely, fundamentally difficult — not just a matter of throwing more GPUs at the problem.

Autoregressive Generation and the Token-by-Token Performance Trap

Modern LLMs generate text through autoregressive decoding — one token at a time, where each new token depends on every token that came before it. This sequential dependency is a fundamental architectural property of transformer models, not an implementation choice you can engineer around. Without optimization, generating a 500-token response means 500 sequential forward passes through your model.

Think of it this way: imagine writing a letter where, before adding each new word, you re-read the entire letter from the beginning to decide what word comes next. That’s essentially what naive LLM inference does on every token step. The computational cost grows with sequence length, and under load with dozens of concurrent requests, this becomes the dominant bottleneck between you and acceptable throughput.

The good news is that previously generated tokens don’t change between steps — which means there’s massive redundant computation happening on every forward pass. That redundancy is exactly what KV caching is designed to eliminate. But before we get there, there’s another layer to the problem.

Memory Pressure — Why GPU VRAM Becomes the Bottleneck

As sequence length grows, so does the memory needed to serve that request. Each token in the sequence requires storing key-value tensors across all attention layers — and those tensors live in GPU VRAM. For a model like Llama 3 8B, a single long-context request can consume several gigabytes of VRAM just for its KV cache.

Multiply that by 20, 50, or 100 concurrent users. With naive memory management, you’d pre-allocate a worst-case contiguous memory block for each request’s potential maximum sequence length — most of which sits unused for most of that request’s lifetime. GPU memory fragments, requests start competing for VRAM, and the system either crashes or starts aggressively throttling. Raw compute capacity becomes irrelevant; you’re bottlenecked on memory management, not arithmetic.

This is the insight that surprises most developers new to LLM serving: the challenge often isn’t running the model — it’s managing the memory around running the model at scale.

The Batching Dilemma — Throughput vs. Latency Tradeoffs

There’s another layer: how you group requests together. Static batching — the naive approach — groups incoming requests into fixed batches, runs the batch until every request has finished generating, then starts the next batch. This beats running requests one at a time, but it introduces a painful “head-of-line blocking” problem.

Say your batch has one request generating 2,000 tokens and five requests generating 50 tokens each. Those five short requests have to wait idle while the long request finishes. The GPU is technically busy, but you’re paying a steep latency penalty for every short request in that batch. This is the classic throughput-versus-latency tradeoff that makes static batching a poor fit for real-world LLM serving — and it sets up perfectly why continuous batching was such a meaningful innovation.

KV Caching Explained — The Biggest Win in LLM Inference

If you understand only one optimization in this entire article, make it this one. KV caching is the single highest-leverage technique in LLM inference, and nano-vLLM’s codebase is the cleanest place I’ve found to see it in action.

What Are Key-Value Pairs in the Attention Mechanism?

In the transformer attention mechanism, every token gets three learned projections: a query (Q), a key (K), and a value (V). During inference, each new token attends to all previous tokens by comparing its query against their keys, then uses those comparisons to weight a sum of the corresponding values. That’s how the model “looks back” at context to decide what’s relevant.

Here’s the crucial observation: once a token has been processed and its K and V tensors computed, those tensors don’t change. The key-value pairs for “The cat sat on” don’t change just because you’re now computing the next token after “on.” They’re fixed, deterministic outputs of the model’s weight matrices applied to that token’s embedding. Think of KV pairs like notes you’ve already written — without caching, you’d be rewriting those same notes from scratch before glancing at them every single time.

How KV Caching Eliminates Redundant Computation

KV caching operationalizes that observation: rather than recomputing K and V for every token in the context on every forward pass, you compute them once and store them. The next token’s forward pass only needs to compute Q, K, and V for the new token, then retrieve the cached K and V from all previous tokens. The amount of skipped computation is enormous.

Without KV caching, inference cost scales quadratically with sequence length. With it, the incremental cost of generating each new token is roughly constant, regardless of how long the context has grown. For long-form generation — multi-turn conversations, document summarization, code generation — this difference isn’t marginal. It’s the difference between a usable system and an unusable one.

Reading KV Cache Logic in nano-vLLM’s Codebase

This is where nano-vLLM earns its place. The KV cache management logic lives in just a few hundred lines — you can find the allocation logic, the store-and-retrieve pattern, and the interaction with the scheduler all in one focused read. Here’s a simplified illustration of the pattern you’ll find:

# Simplified KV cache store-and-retrieve pattern (illustrative)
class KVCache:
    def __init__(self, num_layers, max_blocks, block_size, head_dim):
        # Pre-allocate cache tensors for all layers
        self.cache = torch.zeros(
            num_layers, 2, max_blocks, block_size, head_dim
        )

    def store(self, layer_idx, block_idx, position, key, value):
        self.cache[layer_idx, 0, block_idx, position] = key
        self.cache[layer_idx, 1, block_idx, position] = value

    def retrieve(self, layer_idx, block_indices):
        keys = self.cache[layer_idx, 0, block_indices]
        values = self.cache[layer_idx, 1, block_indices]
        return keys, values

You can read this pattern, understand it fully, and trace how it interacts with the attention computation in about 20 minutes. Try doing the equivalent in vLLM’s production codebase — you’ll be context-switching across multiple modules, CUDA bindings, and async abstractions before you find the same idea. Reading clean code is a skill-building activity, and nano-vLLM is genuinely worth reading.

PagedAttention — Borrowing From Operating Systems to Fix Memory Fragmentation

KV caching solves redundant computation beautifully. But it introduces a new challenge: as you scale to many concurrent requests with different sequence lengths, managing all those cached tensors in GPU memory becomes its own mess. Enter PagedAttention.

The Memory Fragmentation Problem With Naive KV Caching

When you naively implement KV caching for multiple concurrent requests, you typically allocate a contiguous block of GPU memory for each request’s cache — sized for the maximum possible sequence length that request might reach. It’s the safe, simple approach. It’s also brutally wasteful.

A request that generates 50 tokens but was allocated memory for 2,048 tokens wastes 97.5% of its reserved VRAM. Multiply that across dozens of concurrent requests and you have severe memory fragmentation — large pools of technically allocated but functionally unused GPU memory that can’t be handed to new requests. Early operating systems had exactly this problem with RAM, and the solution OS designers landed on is the same one vLLM’s research team applied to transformer inference.

PagedAttention — Virtual Memory for GPU Attention

PagedAttention fixes memory fragmentation by allocating KV cache memory in fixed-size blocks — analogous to memory pages in an OS — rather than large pre-sized contiguous chunks. Each block holds KV pairs for a fixed number of tokens (say, 16 tokens). When a request needs more KV cache space, it gets another block — and that block doesn’t need to be physically adjacent to the previous one.

A block table maps each request’s logical sequence of blocks to their physical locations in GPU memory, exactly as a page table maps logical addresses to physical RAM. The attention computation references block table entries to find its KV data rather than assuming contiguous memory. The original vLLM paper (Source: https://arxiv.org/abs/2309.06180) showed this achieves dramatically higher GPU memory utilization than prior approaches — sometimes enabling 2–4x more concurrent requests on the same hardware.

How nano-vLLM Makes PagedAttention Readable

In nano-vLLM, PagedAttention is implemented in plain Python — you can find the block allocator managing a free-list of physical blocks, the per-request block table tracking logical-to-physical mappings, and the attention function referencing that block table during computation. Here’s a simplified illustration:

# Simplified block table lookup (illustrative)
class BlockTable:
    def __init__(self):
        self.logical_to_physical = {}  # logical block idx -> physical block idx

    def append_block(self, logical_idx, physical_idx):
        self.logical_to_physical[logical_idx] = physical_idx

    def get_physical_blocks(self, logical_indices):
        return [self.logical_to_physical[i] for i in logical_indices]

Compare this to navigating the same concept in vLLM’s production codebase, where block management is interleaved with CUDA kernel dispatch, async scheduling, and distributed memory coordination. In nano-vLLM, the idea is exposed cleanly — you can hold the entire mechanism in your head at once. That’s a rare thing when studying systems software.

Continuous Batching and Request Scheduling — Keeping the GPU Busy

KV caching handles redundant computation. PagedAttention handles memory fragmentation. Now let’s talk about the third pillar: keeping the GPU as busy as possible by rethinking when requests enter and exit the system.

Why Static Batching Leaves Performance on the Table

Static batching is the intuitive approach: collect a group of requests, process them together, wait until every request has finished generating, then load the next batch. Simple, predictable, and genuinely better than one-at-a-time processing. But it has a serious flaw.

Real workloads have wildly different output lengths. One user wants a one-sentence summary; another wants a 1,000-word essay. In static batching, the short requests finish early and sit idle waiting for the long request to complete before the batch turns over.

This is head-of-line blocking — the longest request holds everyone else hostage. It’s like a restaurant that only seats a new party after every single diner at the current table has finished, paid, and left — even if four of them have been done for 20 minutes and are just waiting on the one person who ordered dessert. You’d leave that restaurant.

Continuous Batching — Iteration-Level Scheduling

Continuous batching, first described in the Orca paper (Source: https://www.usenix.org/conference/osdi22/presentation/yu), solves head-of-line blocking by changing scheduling granularity from batch completion to individual decoding steps. At each token generation step — each forward pass — the scheduler checks whether any running requests just produced their final token. If they did, their KV cache blocks are freed and new waiting requests immediately slot into the available capacity.

The GPU never stops to wait for a full batch to clear. New requests flow in as space opens up, continuously. The result is dramatically higher GPU utilization, especially under mixed workloads — exactly the conditions you’ll face in production. Continuous batching is one of the main reasons modern serving systems can sustain throughput levels that would have seemed impossible with earlier infrastructure.

Tracing the Scheduler in nano-vLLM

The scheduler is one of the most instructive components to trace in nano-vLLM because it’s where KV caching, PagedAttention block management, and batching logic all converge. The scheduler maintains a waiting queue of incoming requests, a running set of actively decoding requests, and a loop that fires on every decoding step.

On each step, it checks which running requests have finished, frees their blocks back to the allocator, and promotes waiting requests into the running set as block capacity allows. Here’s the conceptual shape of that loop:

# Simplified scheduler step (illustrative)
def schedule_step(self):
    # 1. Check for completed requests and free their blocks
    for request in self.running:
        if request.is_finished():
            self.block_allocator.free(request.block_table)
            self.running.remove(request)

    # 2. Promote waiting requests into running set
    while self.waiting and self.block_allocator.has_capacity():
        next_request = self.waiting.pop(0)
        next_request.block_table = self.block_allocator.allocate()
        self.running.append(next_request)

    # 3. Return current running set for the next decode step
    return self.running

You can trace this loop, understand every line, and see exactly how it interacts with the block allocator and the model’s forward pass. The mental model you build here transfers directly to vLLM’s production scheduler — which does the same things, just with significantly more machinery around them. That’s the real payoff of using nano-vLLM as a learning sandbox.

Frequently Asked Questions

Q: Is nano-vLLM actually usable for serving models in production? No — and that’s intentional. nano-vLLM lacks the distributed parallelism, optimized CUDA kernels, production API layer, and robustness features that real serving requires. Think of it like a teaching hospital simulator: it builds real skills, but you wouldn’t perform surgery on it. For production open source LLM deployment, use full vLLM, TGI, or TensorRT-LLM.

Q: What models can I run with nano-vLLM for learning purposes? nano-vLLM supports standard HuggingFace-compatible transformer models on a single GPU. It works best with smaller models (7B–13B parameters) where you can observe the behavior without massive hardware requirements. The focus is on reading and understanding the code, not benchmarking outputs.

Q: How long does it actually take to read through the nano-vLLM codebase? A focused developer can trace the core scheduling and caching logic in 2–4 hours. The full ~1,000-line codebase can be read meaningfully in a day. Compare that to vLLM’s 100,000+ lines — there’s no realistic equivalent study session for the production codebase.

Q: Does understanding nano-vLLM actually help with using full vLLM in production? Significantly, yes. Once you have a clear mental model of how KV cache blocks are allocated and freed, why continuous batching improves throughput, and what PagedAttention’s block table is doing — debugging vLLM configuration issues, tuning --max-num-batched-tokens, or understanding why certain workloads cause memory pressure becomes far more intuitive.

Q: Where does nano-vLLM fit relative to reading the original vLLM paper? They’re complementary, not alternatives. The original vLLM paper (Source: https://arxiv.org/abs/2309.06180) gives you the theoretical framing and motivation. nano-vLLM gives you the implementation intuition — how these ideas actually translate into code. Reading the paper first, then tracing nano-vLLM, then skimming full vLLM’s relevant modules is probably the highest-ROI learning path available.

Conclusion

Here’s the uncomfortable truth most LLM deployment tutorials skip: you can use vLLM successfully for months without understanding what it’s actually doing. And then one day something breaks in a way the documentation doesn’t explain, or a performance problem surfaces that you can’t debug without knowing what’s happening inside the engine — and suddenly the black box is a liability.

nano-vLLM exists to prevent that moment from blindsiding you. A few hours with its ~1,000 lines of clean, intention-revealing Python will build a genuine mental model of KV caching, PagedAttention, and continuous batching that no amount of high-level blog posts can replicate. The core insight this nano vLLM tutorial has tried to surface is simple: nano-vLLM is not a lightweight production tool — it’s a learning accelerator, and knowing how to use it is far more valuable than it might first appear.

The mental models you build here transfer directly to every production-grade open source LLM deployment tool you’ll encounter: vLLM, TGI, TensorRT-LLM, and whatever comes next. The LLM inference optimization techniques that make them fast are the same ones you can now trace in a few hundred lines of readable code. That’s a real edge for any ML engineer who wants to move from using these tools to truly understanding them.

Ready to open the black box?

Clone the nano-vLLM repository (https://github.com/GeeeekExplorer/nano-vllm), open the scheduler and KV cache modules, and trace one full decoding step end-to-end. It’ll take you an afternoon. It’ll pay off every time you touch an LLM serving system for the rest of your career.

Sources

1. https://github.com/GeeeekExplorer/nano-vllm

2. https://arxiv.org/abs/2309.06180

3. https://www.usenix.org/conference/osdi22/presentation/yu

4. https://miro.medium.com/v2/resize:fit:1400/1*-ceG6v7SbJHTivsnUsJmPQ.jpeg

5. https://miro.medium.com/0*_Opn4ZhUXMqfs22Y.png

6. https://developer-blogs.nvidia.com/wp-content/uploads/2023/11/key-value-caching_.png

7. https://miro.medium.com/v2/resize:fit:1200/1*FqKiqclKkgSmlbY3qCxrig.png

8. https://www.clarifai.com/hs-fs/hubfs/Batching%20Strategies%20LLM%20Serving-png.png?width=1536&height=1024&name=Batching%20Strategies%20LLM%20Serving-png.png

9. https://github.com/GeeeekExplorer/nano-vllm)

10. https://arxiv.org/abs/2309.06180)

11. https://www.usenix.org/conference/osdi22/presentation/yu)

💡 Want to share your work on my socials with my 15k+ audience? If you build a project you are excited about, I will be too. Trust me! I love seeing people build cool stuff. To share it, you can contact me here.

Did you enjoy this post? Here are some other AI Agents posts you might have missed:

KV Caching and Speculative Decoding

A deep dive into Quantization: Key to Open Source LLM Deployments

Agents are here and they are staying

How Agents Think

Memory – The Agent’s Brain

Agentic RAG Ecosystem

Multimodal Agents

Scaling Agents: Architectures with Google ADK, A2A, and MCP

Fully Functional Agent Loop

Ready to take it to the next level?

Check out my AI Agents for Enterprise course on Maven and be a part of something bigger and join hundreds of builders to develop enterprise level agents.

Use this link to get $201 OFF!

You’re receiving this email because you’re part of our mailing list—and you’ve attended, registered for, or been invited to our MAVEN events. These emails are the only way to reliably receive updates from us. We don’t spam or sell your information. If you prefer not to receive our messages, simply unsubscribe below and we’ll respect your wishes.

Welcome to Edition #28 of a newsletter that 15,000+ people around the world actually look forward to reading.

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing.
This isn’t that. You’ll find raw, honest, human insight here, the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

🎓 Want to up skill in AI?

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) April 8

• Watch the free 4-session Agent Bootcamp playlist on YouTube

Preamble

Running an LLM for every user query is the fastest way to burn budget and add latency. 

In real-world RAG workloads, users often ask the same thing in slightly different words, yet a naive pipeline still reruns retrieval and regeneration each time.

Semantic caching fixes this “always compute” inefficiency by matching queries in embedding space and reusing prior results when the meaning is similar, not the exact text.

Subscribe now

Understanding Semantic Caching for RAG Systems

Semantic caching stores and reuses query results based on meaning rather than exact text matches. This approach solves performance problems in LLM applications without sacrificing quality.

Modern semantic cache systems change how you optimize RAG architecture. Traditional caches need identical queries to work. Semantic caches recognize similar questions and serve stored responses. This cuts redundant LLM calls and speeds up your system by up to 300%.

Building a working RAG system is one thing. Scaling it efficiently is another. As query volumes grow and users expect faster responses, you need better caching strategies. Research shows poorly optimized systems waste 60-80% of processing time on redundant operations that semantic caching eliminates.

This guide covers the technical architecture, implementation strategies, and evaluation methods for semantic cache systems. You’ll get the practical knowledge needed to transform your RAG performance.

RAG Architecture and Semantic Caching Basics

Core RAG Components

RAG architecture has three main parts: the retrieval system, the augmentation layer, and the generation component. Each part offers opportunities where semantic caching reduces delays.

The retrieval system forms the foundation. Vector databases like Pinecone, Weaviate, or Chroma search through large document collections. This component takes up 40-60% of total system delay. When users submit queries, the system converts them into vector embeddings, searches the knowledge base, and returns relevant document chunks.

The augmentation layer processes retrieved documents. You rank and filter results based on relevance scores and metadata. This stage involves extra LLM calls for reranking and context preparation, adding another 20-30% to total delay. The generation component combines retrieved context with the original query to produce the final response.

Source : https://habr.com/ru/articles/977260/

Source: ThinkPalm — https://thinkpalm.com/blogs/what-is-retrieval-augmented-generation-rag/

How Semantic Caching Works with LLMs

Semantic caching recognizes that many user queries share meaning even when worded differently. Traditional caches miss these opportunities because they need exact string matches. For example, “How to optimize database performance?” and “What are the best practices for speeding up SQL queries?” would generate separate cache misses despite asking for similar information.

Semantic caching uses vector embeddings to calculate cosine similarity between queries. This helps the system identify when cached responses work for new but related questions. The approach transforms caching from a binary hit-or-miss system into a nuanced similarity-based retrieval mechanism.

The technology addresses retrieval augmented generation bottlenecks by creating multiple cache layers:

• Query-level caching for similar questions

• Context-level caching for retrieved document chunks

• Response-level caching for generated outputs

Each layer reduces different types of computational overhead while maintaining response quality.

Source: https://portkey.ai/blog/reducing-llm-costs-and-latency-semantic-cache/

Building a Semantic Cache System

Modern semantic cache systems use a multi-layered architecture that integrates with existing RAG components. The primary cache layer sits between the user interface and the retrieval system, intercepting incoming queries and checking for semantically similar cached responses.

The embedding generation component converts new queries into vector representations using the same model employed by the underlying RAG system. This consistency ensures similarity calculations remain accurate. Popular embedding models like OpenAI’s text-embedding-ada-002 or Sentence Transformers provide the vector representations needed for effective semantic matching.

Cache storage systems must support high-dimensional vector operations while maintaining fast lookup times. Redis with vector search capabilities, Elasticsearch with dense vector fields, or specialized vector databases serve as the backbone for semantic cache storage. The cache management layer handles similarity threshold enforcement, cache invalidation policies, and performance monitoring.

Source: https://medium.com/google-cloud/implementing-semantic-caching-a-step-by-step-guide-to-faster-cost-effective-genai-workflows-ef85d8e72883

Performance Impact of Semantic Caching

Measured Speed Improvements

Production deployments of semantic cache systems show substantial performance improvements. Organizations report 3-5x speed reductions for repeated or similar queries, with cache hit rates ranging from 30-70% depending on use case patterns and similarity threshold settings.

Real-world data shows:

• Customer support chatbots achieve response times of 200-400ms with semantic caching, compared to 1.2-2.5 seconds without

• Knowledge management systems drop from 10-15 seconds to 2-4 seconds when cache hits occur

• Organizations processing 1 million queries monthly see 40-60% reductions in inference costs

Optimizing Cache Hit Rates

Effective evaluation methods reveal optimal similarity thresholds typically range between 0.85-0.95 cosine similarity, depending on domain specificity and acceptable response variation. Lower thresholds increase cache hit rates but risk serving less relevant responses. Higher thresholds maintain quality at the expense of caching effectiveness.

Dynamic threshold adjustment based on query patterns and user feedback enables systems to optimize for specific use cases:

• Customer service applications often benefit from lower thresholds (0.82-0.87) due to repetitive question patterns

• Technical documentation systems require higher thresholds (0.92-0.97) to maintain accuracy

Cache performance monitoring reveals effective systems achieve 45-65% hit rates within the first week of deployment. Rates climb to 60-80% as the cache builds comprehensive coverage of common query patterns. The initial warm-up period typically requires 10,000-50,000 queries depending on domain complexity.

Resource Efficiency

Semantic caching dramatically reduces computational overhead across multiple system components:

• Vector database query loads decrease by 40-70%

• LLM inference costs drop proportionally to cache hit rates, with high-performing systems reducing API calls by 50-80%

• Memory utilization increases by 10-20% for embedding storage and similarity calculations, but this overhead pays dividends through reduced processing requirements

• CPU usage drops significantly during cache hit scenarios

• Network bandwidth consumption decreases as cached responses eliminate external API calls

Source: https://www.solo.io/blog/semantic-caching-with-gloo-ai-gateway

Implementation Strategies

Read more

Welcome to Edition #27 of a newsletter that 14,000+ people around the world actually look forward to reading.

We’re living through a strange moment: the internet is drowning in polished AI noise that says nothing.
This isn’t that. You’ll find raw, honest, human insight here, the kind that challenges how you think, not just what you know. Thanks for being part of a community that still values depth over volume.

🎓 Want to up skill in AI?

• Join the next cohort of my Agent Engineering Bootcamp (Developers Edition) April 8

• Watch the free 4-session Agent Bootcamp playlist on YouTube

Two AI Optimization Techniques That Transform Language Model Speed

Modern large language models face a simple problem: they’re too slow for real-time use. Traditional inference methods force models to recalculate the same computations repeatedly and generate text one token at a time. This creates bottlenecks that get worse with longer conversations.

Two techniques are changing this. KV Cache stores previous computations in memory so the model doesn’t repeat the same work. Speculative Decoding uses a small, fast model to generate multiple tokens at once, then verifies them with the full model. Together, they can speed up AI responses by 10x or more.

KV caching visual overview (quick anchor before we dive into mechanics).

Source: https://huggingface.co/blog/not-lain/kv-caching

These aren’t theoretical improvements. Companies are using them now to cut costs and improve user experience.

How KV Cache Works

The Core Problem

When a language model generates text, it uses an attention mechanism to understand which previous words matter for the next word. Traditional inference recalculates these attention weights for every single token, even though most of the computation stays the same.

Attention at a glance: Attention(Q, K, V) = softmax(QKᵀ/√dₖ) V

Source:

Daily Dose of Data Science

KV Caching in LLMs, Explained Visually.

Stay ahead in Tech with AWS Developer Center…

Listen now

2 years ago · 16 likes · Avi Chawla

Autoregressive decoding reprocesses the growing prefix at each step (why decoding slows down).

Source:

Ahead of AI

Understanding and Coding the KV Cache in LLMs from Scratch

KV caches are one of the most critical techniques for efficient inference in LLMs in production. KV caches are an important component for compute-efficient LLM inference in production. This article explains how they work conceptually and in code with a from-scratch, human-readable implementation…

Read more

a year ago · 446 likes · 41 comments · Sebastian Raschka, PhD

Redundancy across decoding steps: most of the prefix is repeated work.

Source: https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms

For a conversation with 1,000 previous tokens, the model performs 1,000 × 1,000 = 1,000,000 operations for each new word. This quadratic complexity makes longer conversations exponentially slower.

The Solution

KV Cache stores the Key and Value matrices from previous tokens. When generating a new token, the model retrieves these stored values instead of recalculating them.

Inside self-attention: Q/K/V projections feed the attention computation (KV caching reuses K and V).

Source: https://blog.gaurav.ai/2025/08/05/kv-caching-kv-sharing/

This reduces 1,000,000 operations down to 1,000 operations per token. The attention mechanism goes from O(n²) to O(n) for each new token.

Here’s what happens step by step:

KV caching loop: store K/V once, append new K/V each step, and reuse cached tensors for attention.

Source: https://huggingface.co/blog/not-lain/kv-caching

1. The system retrieves stored K and V matrices from cache memory

2. It computes only the new Q, K, V vectors for the current token

3. It combines new values with cached values

4. It performs the attention calculation

5. It updates the cache with the new token’s K, V values

Give a gift subscription

Real Performance Gains

A financial services company processing 15,000 regulatory documents daily saw a 67% reduction in response time after implementing KV Cache. They analyzed documents in real time instead of waiting hours.

The benefits increase with sequence length. Short sequences see 40-50% speedups. Long conversations with thousands of tokens can be 3x faster or more.

Memory usage increases by 10-20% of model parameters, which is a reasonable trade-off for the speed gains.

How Speculative Decoding Works

The Draft-and-Verify Method

Normal text generation is sequential. The model generates one token, then uses that token to generate the next one. You can’t parallelize this process.

Speculative Decoding breaks this pattern. A small, fast draft model generates multiple token candidates. Then the large target model verifies all of them at once in a single forward pass.

The draft model might be 10-100x faster than the target model. It doesn’t need to be perfect. It just needs to generate reasonable candidates that the target model can verify quickly.

The Process

Draft Generation Phase

The small model generates k candidate tokens rapidly. For example, it might propose the next 5-10 tokens in a sentence.

Batch Verification

The large model processes all k candidates simultaneously. This batch processing uses GPU parallelization to verify multiple tokens with minimal overhead compared to processing one token.

Acceptance Decision

Source:https://clova.ai/en/tech-blog/breaking-the-speed-barrier-how-we-implemented-speculative-decoding-for-hyperclova-x?utm_source=chatgpt.com

An algorithm compares the draft model’s probabilities with the target model’s probabilities. It accepts tokens that meet the quality threshold and rejects the rest.

If all tokens are accepted, the system just generated multiple tokens in roughly the time it normally takes to generate one. If some are rejected, the system falls back to standard generation for those positions.

Adaptive Window Sizing

The system adjusts how many tokens the draft model generates based on acceptance rates. High acceptance rates increase the window size. Low acceptance rates decrease it.

Production Results

A major cloud provider implemented Speculative Decoding for their conversational AI platform with these results:

• 7.2x average speedup in chatbot responses

• 45% reduction in GPU resource consumption

• $2.3M annual cost savings

• Zero degradation in customer satisfaction scores

They used a 1.5B parameter draft model with a 175B parameter target model.

Source: https://www.together.ai/blog/customized-speculative-decoding

Comparing the Two Techniques

Different Approaches

KV Cache removes redundant computations within the attention mechanism. It’s a straightforward optimization that works with existing model architecture.

Speculative Decoding changes how tokens are generated. It requires two models working together and adds architectural complexity.

Performance Patterns

Speed Improvements:

• KV Cache: 1.5-3x typical gains, more for longer sequences

• Speculative Decoding: 3-10x potential gains

• Combined: Over 10x total speedup possible

Memory Requirements:

• KV Cache: 10-20% overhead, grows with sequence length

• Speculative Decoding: Higher baseline but fixed overhead

• Combined: Requires careful memory management

Implementation Complexity:

• KV Cache: Low to medium, straightforward to add

• Speculative Decoding: High, needs sophisticated coordination

When to Use Each

Use KV Cache for:

• Long documents or conversations

• Extended context windows

• Applications with growing conversation history

• Memory-efficient deployments

Legal document processing systems report 78% latency reduction with KV Cache. Customer service platforms maintain sub-second response times even after hundreds of exchanges.

Use Speculative Decoding for:

• Applications requiring maximum speed

• Interactive real-time experiences

• Batch processing workflows

• Latency-critical deployments

Educational platforms report 340% increases in student engagement with near-instantaneous AI responses. Financial trading platforms achieve sub-100ms response times for real-time market analysis.

Implementation Challenges

KV Cache Challenges

Memory Management

You need adaptive sizing strategies. Cache size must balance hit rates against memory consumption, especially when sequence lengths vary.

Production reality: priority-based eviction helps keep reusable prompt blocks in cache longer.

Source: https://developer.nvidia.com/blog/introducing-new-kv-cache-reuse-optimizations-in-nvidia-tensorrt-llm/

Production systems use automated cache eviction policies based on usage patterns. They also need monitoring systems that track cache efficiency in real time.

Cache Invalidation

The cache must stay consistent when models update or conversation context shifts. Systems need versioning and validation mechanisms to detect and correct inconsistencies.

Speculative Decoding Challenges

Model Coordination

You need continuous calibration to maintain optimal acceptance rates. If the draft model generates poor candidates, the system wastes computation on rejected tokens.

Organizations use automated evaluation pipelines with continuous performance monitoring. They tune parameters based on real-time feedback.

Resource Management

Managing two models requires sophisticated orchestration. The system must allocate computational resources between draft and target models based on workload and hardware constraints.

Advanced implementations use adaptive resource allocation with predictive scaling based on usage patterns.

Quality Assurance

Both techniques need comprehensive validation. Automated testing pipelines evaluate output quality across diverse use cases.

A/B testing frameworks compare optimized and standard inference across metrics like semantic coherence, factual accuracy, and task performance. Real-time monitoring tracks latency, throughput, memory usage, and quality scores.

Failure recovery mechanisms automatically revert to standard inference when optimizations encounter errors. This ensures reliability while enabling aggressive optimization.

Real-World Applications

Customer Service

A telecommunications company handles 2 million daily customer interactions. They implemented both KV Cache and Speculative Decoding with a 7B parameter draft model and 175B parameter target model.

Results:

• 87% reduction in average response time (4.8s to 0.7s)

• 23% increase in customer satisfaction

• 34% reduction in call abandonment

• $1.8M annual savings

The system maintains conversation coherence across extended support sessions with consistent sub-second response times.

Code Generation

GitHub Copilot and similar tools use these optimizations to provide real-time code completion. They need sub-100ms response times to avoid disrupting developer flow.

A major software company’s internal platform supports 12,000 developers:

• 67% improvement in code completion speed

• 94% acceptance rate for suggestions

• Real-time code review and documentation generation

Content Creation

News organizations use optimized AI to generate draft articles within 30 seconds of breaking events. A major news outlet processes 500+ breaking news events daily with a 78% reduction in time-to-publish.

Gaming platforms use optimization for dynamic narrative generation. One studio supports 100,000 concurrent players with personalized storylines. They achieved 5.2x improvement in narrative generation speed and 156% increase in player engagement.

Specialized Industries

Financial Services

A hedge fund’s system delivers investment insights within seconds of market events. They achieved 94% accuracy in market sentiment analysis while reducing analysis time from hours to minutes.

Healthcare

A hospital network processes 25,000 patient interactions daily. Medical staff spend 89% less time on documentation while improving accuracy and completeness.

Legal

A law firm processes 2,000+ legal documents daily with 71% reduction in initial review time. Lawyers focus on high-level strategy while AI handles routine document processing.

Implementation Strategy

Phase 1: Foundation (Weeks 1-4)

• Establish baseline performance metrics.

• Set up comprehensive monitoring infrastructure.

• Develop testing frameworks for quality validation.

• Conduct technical feasibility assessments and prepare infrastructure for optimization deployment.

Phase 2: Pilot (Weeks 5-12)

• Deploy optimizations in controlled environments with limited traffic.

• Run extensive A/B testing comparing optimized and standard inference.

• Tune parameters based on real-world data.

• Develop operational procedures for monitoring and maintenance.

Phase 3: Production (Weeks 13-20)

• Implement gradual rollout with continuous monitoring and automatic fallback.

• Continue fine-tuning based on production data.

• Run comprehensive cost-benefit analysis to validate optimization effectiveness.

Phase 4: Advanced Optimization (Weeks 21+)

• Explore combined implementation of multiple techniques.

• Develop custom optimizations for specific use cases.

• Build sophisticated monitoring and automated optimization systems that continuously improve based on usage patterns.

Sample Implementation

class OptimizedInferenceEngine:
def __init__(self, target_model, draft_model, cache_config):
self.target_model = target_model
self.draft_model = draft_model
self.kv_cache = DynamicKVCache(cache_config)
self.memory_pool = AdaptiveMemoryPool()
self.speculation_window = AdaptiveWindowSizer()
self.performance_monitor = RealTimeMonitor()

def generate_optimized(self, prompt, max_length):
# Initialize KV cache with conversation context
cached_kv = self.kv_cache.get_or_create(prompt)

# Calculate optimal speculation window
window_size = self.speculation_window.calculate_optimal_size(
cache_efficiency=cached_kv.hit_rate,
available_memory=self.memory_pool.available_capacity,
target_latency=self.performance_monitor.target_latency
)

return self.speculative_decode_with_cache(
prompt, cached_kv, window_size, max_length
)

Subscribe now

Future Developments

New Optimization Approaches

• Researchers are designing transformer architectures specifically for efficient inference. Sparse attention mechanisms and hierarchical processing structures reduce computational requirements by 60-80% while maintaining model capability.

• Quantum-classical hybrid processing shows promise for exponential speedups in attention weight calculation, though practical applications are years away.

• Neuromorphic computing explores brain-inspired architectures for ultra-low-power inference, especially in edge computing environments.

Hardware Evolution

• Custom silicon is being designed specifically for KV caching operations and speculative decoding workflows. Major semiconductor companies are developing inference-optimized chips that could deliver 10-100x improvements over general-purpose GPUs.

• New memory systems are being developed specifically for AI workloads, with high-bandwidth memory optimized for KV cache storage.

Industry Adoption

• Major cloud providers now offer optimization techniques as managed services. AWS, Google Cloud, and Microsoft Azure are integrating KV Cache and Speculative Decoding into their AI platforms.

• Open source frameworks like Hugging Face Transformers, vLLM, and TensorRT make these optimizations accessible to smaller organizations.

Self-Optimizing Systems

• Researchers are developing AI systems that automatically optimize their own inference performance through experience and learning. Early research shows AI systems improving their own inference efficiency by 40-60% through automated optimization discovery.

• These adaptive systems could eliminate manual parameter tuning by continuously adjusting optimization strategies based on usage patterns, hardware characteristics, and performance objectives.

Reflections

KV Cache and Speculative Decoding have transformed what’s possible with large language models. KV Cache eliminates redundant computations by storing previous calculations. Speculative Decoding accelerates generation by using a small model to propose tokens that a large model verifies in parallel.

Combined, these techniques can achieve 10x or greater speedups while maintaining output quality. This enables real-time AI applications that were previously too slow for production use.

The business impact is clear. Organizations report:

• 40-70% reduction in infrastructure costs

• Improved user satisfaction from faster responses

• New capabilities in real-time customer service, code generation, and content creation

• Competitive advantages through superior AI performance

Success requires careful implementation. Start with baseline metrics and monitoring. Deploy in controlled environments. Test extensively. Roll out gradually with automatic fallback systems.

The techniques continue to evolve. Future developments in hardware, architecture, and self-optimizing systems promise even greater improvements. Organizations that master these optimizations now will be positioned to capitalize on future advances.

For any production AI system where speed matters, these optimizations are no longer optional. They’re essential for delivering the performance users expect and the efficiency businesses require.

Did you enjoy this post? Here are some other AI Agents posts you might have missed:

A deep dive into Quantization: Key to Open Source LLM Deployments

Agents are here and they are staying

How Agents Think

Memory – The Agent’s Brain

Agentic RAG Ecosystem

Multimodal Agents

Scaling Agents: Architectures with Google ADK, A2A, and MCP

Fully Functional Agent Loop

Subscribe now

Ready to take it to the next level?

Check out my AI Agents for Enterprise course on Maven and be a part of something bigger and join hundreds of builders to develop enterprise level agents.

Use this link to get $201 OFF!

You’re receiving this email because you’re part of our mailing list—and you’ve attended, registered for, or been invited to our MAVEN events. These emails are the only way to reliably receive updates from us. We don’t spam or sell your information. If you prefer not to receive our messages, simply unsubscribe below and we’ll respect your wishes.

But in production, you hit a wall quickly: memory. Once models reach billions of parameters, the VRAM required for weights can exceed what most hardware can comfortably support.

Quantization tackles this by reducing weight precision so models fit and run efficiently, often with only modest quality loss when done right.

This analysis compares three common approaches:

INT8, INT4, and NF4.

Each sits at a different point on the memory–accuracy–speed trade-off curve. Quantization can significantly reduce weight memory and sometimes improve throughput, but the best choice depends on your hardware and how sensitive your task is to small quality changes. This guide helps you pick the right method.

Subscribe now

What Is LLM Quantization and Why Does It Matter?

LLM quantization is typically inference-time weight quantization: converting weights stored in FP16/FP32 into lower-precision formats such as INT8 or INT4. This reduces VRAM usage and can improve throughput, making it easier to run larger models or serve more requests on the same GPU.

At a high level, quantization maps continuous floating-point values to a smaller set of representable values (the “levels”). INT8 uses 8-bit integers and INT4 uses 4-bit integers, but in practice weights are stored with scales (and sometimes zero-points) so the runtime can reconstruct approximate values during computation.

Modern methods such as NF4 go beyond uniform spacing: they use a non-uniform set of levels designed around common weight statistics (often close to a normal distribution). That’s why NF4 is popular in bitsandbytes / QLoRA workflows, it often preserves quality better than uniform INT4 at similar memory budgets.

How Quantization Methods Evolved

Understanding where each method came from helps you understand why they behave differently in production.

Post-Training Quantization (PTQ)
Simple precision reduction applied after training (no retraining). Fast to implement. Can reduce quality if you push bit-width too low or skip careful calibration.

Quantization-Aware Training (QAT)
Simulates quantization during training so the model adapts to reduced precision. Often more robust than naïve PTQ. But QAT at foundation-model scale is compute-heavy, so it’s less common for large LLM deployments.

Data-Aware / Calibration-Guided Methods
Uses a small representative dataset to choose scales (and sometimes per-group decisions) and minimize quantization error without full retraining. This is where many practical LLM methods like GPTQ and AWQ fit, enabling INT4 (and some INT8) with relatively small quality loss in many real workloads.

The Three Methods Explained

INT8 The Balanced Production Choice

INT8 quantization maps floating-point weights to 8-bit integers. It cuts model size by 50% versus FP16 and maintains strong compatibility with modern hardware acceleration.

Two mapping schemes matter here:

Symmetric quantization maps the range [−α, α] to [−127, 127] using a single scale factor. Clean and fast.

Asymmetric quantization maps [β, α] to [−128, 127] using both a scale factor and a zero-point. Better for layers with skewed weight distributions.

Hardware support is strong:

● NVIDIA Tensor Cores (V100 and later) accelerate INT8 matrix operations natively

● Intel VNNI extensions provide similar acceleration on CPUs

● This hardware support can improve throughput in the right setup (especially with optimized INT8 kernels and sufficient batching)

INT4 Maximum Compression With Trade-offs

INT4 pushes compression to the limit. You only have 16 codes per group to represent the full continuous distribution of neural network weight. This constraint demands smart boundary selection to preserve critical model information.

Two key techniques make INT4 practical:

• AWQ (Activation-aware Weight Quantization): protects the most salient channels/weights (identified using activation statistics) so aggressive 4-bit rounding doesn’t damage the layer’s output too much.

• GPTQ (post-training optimization) : chooses quantized values that minimize output error on a small calibration set (often using second-order information), typically producing better quality than naïve rounding at the same bit-width.

• The honest trade-off: INT4 has limited native hardware acceleration on most current GPUs. It requires specialized kernels and benefits significantly from mixed-precision deployment strategies.

• NF4 — Non-Uniform Precision for Better Accuracy: NF4 (4-bit NormalFloat) is the most sophisticated of the three. Instead of spacing quantization levels uniformly, it designs levels that align with the natural distribution of pre-trained neural network weights, typically approximately normal.

How it works technically: NF4 computes optimal quantization boundaries using the cumulative distribution function (CDF) of target weights. The result is non-uniformly spaced levels that cluster around common weight values. Rare extreme values get less precision. Common central values get more. This matches how information is actually distributed in the model.

Double quantization is an additional efficiency feature. You apply quantization to the scale factors themselves, reducing metadata overhead further. For large models where scale factor storage becomes non-trivial, this matters.

NF4 pairs directly with QLoRA, enabling continued fine-tuning of quantized models. This is a practical advantage INT4 lacks in most standard implementations.

What Actually Consumes Your VRAM

Weight storage is only part of the picture. Total VRAM consumption during inference includes multiple components, and quantization affects each one differently.

Weight quantization does not reduce KV cache memory. For long-context inference, the KV cache can approach or exceed weight memory. This is a separate problem that requires KV cache quantization, which this analysis does not cover.

Latency and Throughput: What the Numbers Actually Mean

Time-to-First-Token (TTFT)

TTFT measures the delay between submitting a request and receiving the first generated token. It directly shapes how responsive your system feels to users.

Quantization affects TTFT through two competing forces:

Reduced memory pressure allows faster initial weight loading and processing. This helps TTFT.

Dequantization overhead adds computation before matrix multiplications can proceed. This hurts TTFT, especially at small batch sizes where the overhead cannot be amortized across multiple requests.

This tension explains why 4-bit methods can show worse TTFT than INT8/FP16 on single requests when dequantization/packing overhead dominates. With optimized kernels (often available in bitsandbytes-style stacks), that overhead can be reduced, and INT4/NF4 can be competitive depending on your runtime.

Where Aggressive Quantization Wins: Batch Throughput

At batch size 16, FP16 runs out of memory on the RTX 4090. INT4 and NF4 continue scaling. This is where aggressive quantization pays off most, not in single-request latency, but in batch throughput at the memory boundary. If you are not measuring batch throughput at your VRAM limit, you are missing the strongest argument for 4-bit quantization.

Subscribe now

How to Choose the Right Method

Work through this decision framework before committing to a quantization strategy:

Production Deployment Checklist

Before shipping a quantized model to production, work through these steps:

Validation

• Benchmark perplexity on WikiText-103 or C4 for your specific model

• Run task-specific evaluations, MMLU, HumanEval, GSM8K as appropriate

• Test on edge cases and long-context inputs

• Verify output quality on domain-specific prompts if applicable

Infrastructure

• Confirm your hardware has appropriate acceleration support for the chosen method

• Measure actual VRAM usage under peak load, not just model weights

• Test batch size scaling to find your throughput sweet spot

• Set up TTFT monitoring, it degrades differently than throughput

Operations

• Run A/B tests against the FP16 baseline before full rollout

• Set up continuous perplexity and latency monitoring

• Define rollback criteria and maintain a parallel FP16 deployment path

• Document your calibration dataset and quantization parameters for reproducibility

Final Takeaways

Three conclusions stand out from this analysis:

• NF4 is often a strong default for 4-bit workflows: It targets the same memory class as INT4 but frequently preserves quality better in bitsandbytes/QLoRA-style setups. Choose plain INT4 when you have a specific kernel/runtime reason (or an established calibration pipeline) that performs better for your workload.

• INT8 remains a strong choice when hardware acceleration is available and quality/latency need to be predictable. It typically has fewer compatibility surprises than 4-bit methods and performs well across a wide range of deployments.

• The biggest throughput wins from 4-bit quantization usually appear at the memory boundary. If FP16 OOMs at your target batch size or context window, INT4/NF4 can keep scaling, so measure throughput and tail latency at your VRAM limit, not only on a single request.

Start with NF4 if you are memory-constrained. Start with INT8 if you have headroom and need consistency. Measure both on your actual task and hardware before committing.

Did you enjoy this post? Here are some other AI Agents posts you might have missed:

Agents are here and they are staying

How Agents Think

Memory – The Agent’s Brain

Agentic RAG Ecosystem

Multimodal Agents

Scaling Agents: Architectures with Google ADK, A2A, and MCP

Fully Functional Agent Loop

Share

Ready to take it to the next level?

Check out my AI Agents for Enterprise course on Maven and be a part of something bigger and join hundreds of builders to develop enterprise level agents.

Use this link to get $201 OFF!

You’re receiving this email because you’re part of our mailing list—and you’ve attended, registered for, or been invited to our MAVEN events. These emails are the only way to reliably receive updates from us. We don’t spam or sell your information. If you prefer not to receive our messages, simply unsubscribe below and we’ll respect your wishes.

That’s where Evals (evaluations) come in. Think of them as tests or report cards for your AI systems. Instead of guessing if an output is good, we run it through a series of checks to be sure.

Let’s break down how these evals work for different types of AI setups, starting simple and getting more complex.

1. Evals for LLMs: The Basic Health Check

What we’re testing: The raw, knowledge and basic capability of the language model itself.

Imagine you just got a new employee straight out of a universal training program (the internet). You need to check their core skills before giving them a specific job. Evals for base LLMs do exactly that.

What we measure:

• Factual Accuracy: Does it know true things? Does it make up false information (hallucinate)?

• Reasoning (Basic): Can it follow simple logic?

• Toxicity/Bias: Does it produce harmful, biased, or offensive content?

• Following Instructions: Can it do what it’s told (e.g., “write in the style of a pirate”)?

How we test it (The “Exam Paper”):

We use standard sets of questions and tasks where we already know the correct answer.

• Example Task: “What is the capital of Portugal?”

  • Expected Answer: “Lisbon.”

  • Eval Check: Does the model’s output contain the correct answer?

• Example Task: “Complete this sentence: The opposite of hot is...”

  • Expected Answer: “cold.”

  • Eval Check: Is the completion logically correct?

• Example Task: “Write a sentence about [a sensitive topic].”

  • Expected Answer: No specific answer, but a set of rules.

  • Eval Check: Does the output contain hate speech, slurs, or dangerous ideas? (A human or a safety classifier would score this).

The Introspection: You wouldn’t trust a doctor who aced art class but failed biology. Similarly, if a model fails basic factual or safety evals, it’s not fit for any purpose, no matter how fluent it sounds.

2. Evals for LLM Reasoning Models: The Logic Exam

What we’re testing: The model’s ability to solve complex problems that require multiple steps of thought.

Now, let’s say our new employee needs to be an analyst. We need to test their ability to think, not just recall. This is for models specifically prompted or fine-tuned for reasoning (e.g., using Chain-of-Thought).

What we measure:

• Multi-step Problem Solving: Can it break a big problem into smaller steps?

• Mathematical Reasoning: Can it solve word problems?

• Logical Deduction: Can it infer conclusions from a set of rules?

How we test it (The “Logic Puzzle”):

We give it problems where the answer isn’t a simple fact, but the result of a process.

• Example Task: “Sarah has 3 apples. She gives 2 to Mark. Then, she buys 5 more. How many apples does she have now?”

  • Expected Reasoning Steps: 3 - 2 = 1, then 1 + 5 = 6.

  • Eval Check: Does the model’s reasoning process correctly show these steps, and does it arrive at the final answer of 6? The “working out” is as important as the answer.

• Example Task: “All dogs are mammals. All mammals have spines. Fido is a dog. Does Fido have a spine?”

  • Expected Answer: “Yes.”

  • Eval Check: Can the model trace the logical chain (Dog -> Mammal -> Spine) to deduce the correct answer?

The Introspection: Getting the right answer for the wrong reason is a major red flag. It means the model is guessing. A good reasoning eval proves the model’s “thinking” is sound, making it more trustworthy for complex tasks.

3. Evals for Retrieval: The Librarian’s Test

What we’re testing: The system’s ability to find the right pieces of information from a large database (like a vector database).

An LLM doesn’t know your company’s private data. So, we give it a “librarian” (a retriever) that finds relevant documents for it to read. We need to test the librarian, not the reader.

What we measure:

• Relevance: Are the returned documents actually related to the question?

• Recall: Did the system find all the important pieces of information?

• Precision: Are the returned results only the important ones, or is there a lot of junk?

How we test it (The “Library Scavenger Hunt”):

We have a known set of documents and ask questions that the answers are inside them.

• Example:

  • Document 1: “The company project ‘Alpha’ was launched in 2020.”

  • Document 2: “The company project ‘Beta’ focuses on sustainability.”

  • Document 3: “Employee benefits include health insurance and remote work.”

  • Test Question: “When was project Alpha launched?”

  • Perfect Retrieval: [Document 1]

  • Eval Check: Did the retriever return Document 1? (High Precision & Recall). Did it also return irrelevant documents like Document 2 or 3? (Low Precision).

The Introspection: If the librarian brings you a cookbook when you asked for a car manual, the LLM (the reader) has no chance of giving a correct answer, no matter how smart it is. Garbage in, garbage out.

4. Evals for RAG: The End-to-End System Check

What we’re testing: The performance of the entire Retrieval-Augmented Generation pipeline—the librarian (retriever) and the reader (LLM) working together.

This is the most common real-world system. We need to know if the final answer to the user’s question is correct, given the knowledge base.

What we measure:

• Answer Faithfulness/Grounding: Is the final answer based only on the retrieved documents, or did the LLM make up details (hallucinate) using its internal knowledge?

• Answer Relevance: Does the final answer directly address the original question?

• Context Utilization: Did the LLM correctly use the information it was given?

How we test it (The “Open-Book Exam”):

We give the system a question and a set of source documents (the “book”). The system must retrieve the right bits and generate an answer from them.

• Example:

  • Source Documents: Same as above (Project Alpha launched in 2020, etc.).

  • Test Question: “What are the goals of project Beta and when was Alpha launched?”

  • Perfect RAG Output: “Project Beta focuses on sustainability. Project Alpha was launched in 2020.”

  • Eval Checks:

    • Faithfulness: Is every part of the answer supported by the source documents? (Yes).

    • Relevance: Does it answer both parts of the question? (Yes).

    • Hallucination Check: If the answer said “Project Alpha launched in 2021,” that would be a failure, even if it’s a fact the LLM “knows” from its training. The test is about the provided context.

The Introspection: A RAG eval tells you if your entire system is reliable. A failure could be the retriever’s fault (it didn’t find the doc) or the LLM’s fault (it ignored the doc and hallucinated). This eval helps you pinpoint where the breakdown happened.

5. Evals for Agents: The CEO’s Performance Review

What we’re testing: The ability of an AI Agent to complete a multi-step, real-world goal by using tools, making decisions, and recovering from errors.

An Agent is an LLM that can do things—like use a calculator, search the web, or call an API. It’s like an autonomous employee. Evaluating it is complex because the path to success isn’t always a straight line.

What we measure:

• Task Success: Did the agent ultimately accomplish the goal?

• Efficiency: How many steps did it take? Did it use the right tools?

• Robustness: If it hit a dead end or an error, did it recover and try a different approach?

How we test it (The “Simulated Project”):

We give the agent a high-level goal and a set of tools, then watch it work.

• Example Task: “Find the price of a Tesla Model 3 and the nearest dealership to Zurich, then summarize it in a table.”

  • Tools Available: web_search(), calculator(), format_table().

  • Expected Successful Workflow:

    1. web_search(”Tesla Model 3 price”)

    2. web_search(”Tesla dealership near Zurich”)

    3. Extract the relevant prices and addresses.

    4. format_table(data)

  • Eval Checks:

    • Final Answer: Is there a well-formatted table with the correct price and a valid dealership address? (Task Success).

    • Process: Did it use the search tool effectively? Did it get stuck in a loop or try to calculate the price instead of searching for it? (Efficiency).

    • Robustness: If the first search for “Tesla dealership” returned a closed one, did it recognize the issue and search again? (Robustness).

The Introspection: Evaluating an agent is like judging a chef on the final meal, not just their knife skills. You care about the outcome. A successful agent eval means you can trust the system to operate autonomously on complex tasks without constant babysitting.

The Bottom Line

Evals move us from wonder to trust. They replace “Wow, this is cool” with “I know this works for my specific needs.” By applying the right kind of test at each layer—from the raw model’s knowledge all the way up to an agent’s autonomous projects—we can build AI systems that are not just impressive, but are genuinely reliable and useful. Start simple, measure everything, and always be testing.

Ready to take it to the next level?

Check out my AI Agents for Enterprise course on Maven and be a part of something bigger and join hundreds of builders to develop enterprise level agents.

Use this link to get $201 OFF!

You’re receiving this email because you’re part of our mailing list—and you’ve attended, registered for, or been invited to our MAVEN events. These emails are the only way to reliably receive updates from us. We don’t spam or sell your information. If you prefer not to receive our messages, simply unsubscribe below and we’ll respect your wishes.

If you've missed the previous days, you can access them here: Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7

I'm Hamza and joining me is Bhavna. Today, we're diving into one of the most sought-after applications of AI agents: intelligent content creation and automated blog writing!

But before that, have you registered for our free class?

🏆 Tune in: Fortune 100 GenAI Transformation Workshop!

Learn how Fortune 100 companies are actually deploying AI agents at enterprise scale. Join my exclusive live session "Fortune 100 Playbook: GenAI Transformation" this Friday, September 19th at 9AM PDT.

What You'll Master: Discover how leading enterprises identify high-ROI AI use cases, architect multi-agent systems that actually scale, and deploy production-ready solutions that go beyond impressive demos. We'll unpack real transformation playbooks, examine why most AI projects fail in production, and show you the frameworks Fortune 100 companies use to achieve measurable business impact.

🚀 Join Our Free Live Session - Fortune 100 Playbook and join 1000+ enterprise professionals who are implementing AI transformation the right way. No hype, just proven strategies from companies who've cracked the code on GenAI at scale! 🚀

✍️ What Is an AI Blog Post Agent?

Here's the Github Link

The AI Blog Post Agent represents the future of research-driven content marketing, a sophisticated system that transforms how content teams create authoritative, well-researched articles at scale.

Unlike simple text generators, this agent follows a systematic seven-step process: gathering strategic bookmarks from various sources, intelligently filtering for the most relevant and authoritative content, scraping and analyzing source material for key insights, conducting deep research to fill gaps and provide unique perspectives, generating comprehensive articles with research-backed insights, managing text length and chunking for optimal readability, and finally publishing to platforms like Notion with proper formatting and team notifications.

This complete pipeline revolutionizes content creation by replacing random research with structured methodology, ensuring every article is built on solid research foundations. The agent doesn't just write, it scrapes, analyzes, and synthesizes existing content to create unique, well-informed perspectives while maintaining consistent brand voice and professional writing standards throughout the entire process.

📺 Student Success Story: Newsletter Generation Agent

Here's a practical example of how our students Amina Javaid and Tommy Fountain built a Newsletter Generation Agent using the concept of our Blog Post Agent:

See video above!

🔧 How to Implement Your Blog Post Agent Successfully

The technical implementation follows a proven six-step workflow that our B2B technology clients use to produce authoritative content with 91% reduction in research time.

First, gather bookmarks by extracting HTML from popular bookmark pages and splitting individual bookmarks for processing.

Second, filter relevant bookmarks using LLMs to analyze and select the top 3 most relevant sources for your target audience. T

hird, scrape and analyze content by calling external workflows to extract article content and summarize key insights.

Fourth, create focused topics by analyzing all gathered content to generate compelling article angles.

Fifth, perform deep research by running additional Google searches and combining inspiration articles with research results.

Finally, generate and publish detailed articles using LLMs with all previous research, converting markdown to HTML for better formatting, and intelligently handling both short articles (direct publishing) and long articles (smart chunking before publishing).

This systematic approach ensures content teams can produce comprehensive 3,200-word articles with research-backed insights, achieving 187% increase in content authority scores and 89% increase in organic traffic due to better-researched, authoritative content that resonates with readers and search engines alike.

💡 Pro Tips for Blog Post Agent Success

Success with your blog post agent depends on three critical optimization strategies that separate high-performing content from generic AI-generated text.

First, optimize your bookmark strategy based on content type: focus on analyst reports and market research for industry analysis, emphasize documentation and case studies for technical content, prioritize expert opinions and trend analysis for thought leadership, and collect practical examples and tutorials for how-to guides.

Second, build robust research quality gates throughout your process, including source authority verification and credibility scoring, information recency checks and cross-reference validation, and expert opinion verification with proper quote attribution.

Third, implement intelligent content templates with structured formats for different article types: industry analysis with market data and expert insights, technical deep-dives with implementation examples, thought leadership with unique perspectives and predictions, and comprehensive guides with step-by-step instructions.

The compound effect of these optimizations creates systematic, research-driven content that drives organic authority, establishes thought leadership, and generates qualified engagement. Advanced enhancements include multi-source research integration with academic databases and industry reports, content performance analytics to measure research quality impact, collaborative features for team-based development, and multi-platform publishing capabilities extending beyond Notion to WordPress, Ghost, and LinkedIn with automated SEO optimization and social media integration.

You're receiving this email because you're part of our mailing list—and you've attended, registered for, or been invited to our MAVEN events. These emails are the only way to reliably receive updates from us. We don't spam or sell your information. If you prefer not to receive our messages, simply unsubscribe below and we'll respect your wishes.

This is a conversation space exclusively for subscribers—kind of like a group chat or live hangout. I’ll post questions and updates that come my way, and you can jump into the discussion.

Join chat

How to get started

1. Get the Substack app by clicking this link or the button below. New chat threads won’t be sent sent via email, so turn on push notifications so you don’t miss conversation as it happens. You can also access chat on the web.

Get app

2. Open the app and tap the Chat icon. It looks like two bubbles in the bottom bar, and you’ll see a row for my chat inside.

3. That’s it! Jump into my thread to say hi, and if you have any issues, check out Substack’s FAQ.