But here’s the thing: real data is never just one flat object. Think about it, an order has a customer inside it. That customer might have an address inside them. It’s boxes inside boxes.

Today we learn how Pydantic handles that. Stick with me, it’s easier than it sounds.

Subscribe now

Why Flat Validation Misses Nested Errors

Say a refund request comes in, and it has an order attached to it:

response = {
    "name": "Alice",
    "email": "alice@example.com",
    "intent": "refund",
    "order": {
        "id": "ORD-004821",
        "items": [
            {"sku": "SKU-1", "price": 29.99, "qty": 2},
            {"sku": "SKU-2", "price": -15.00, "qty": 1}
        ]
    }
}

Spot the problem? Look at the second item. price: -15.00. Negative. That’s not a “type” problem - it’s still a number. But it’s obviously wrong. Nobody sells something for negative money.

If your model doesn’t know to check inside that order dict, this slips through. And if it reaches your refund logic, congrats, you just refunded someone extra money.

The lesson: flat validation only checks the outside layer. It doesn’t look inside boxes.

Pydantic Nested Models: Put a Model Inside a Model

A Pydantic model can be a field inside another Pydantic model.

That’s it. That’s nesting.

Build it from the smallest piece outward, like Russian nesting dolls:

from pydantic import BaseModel, PositiveFloat, PositiveInt

# smallest doll first
class OrderItem(BaseModel):
    sku: str
    price: PositiveFloat   # rejects negative or zero automatically
    qty: PositiveInt

# next doll — contains a LIST of the smaller doll
class Order(BaseModel):
    id: str
    items: list[OrderItem]

# biggest doll — contains the Order doll
class SupportRequest(BaseModel):
    name: str
    email: str
    order: Order

Observe, one line checks everything, across all the layers:

SupportRequest.model_validate(response)

ValidationError: 1 validation error for SupportRequest
order.items.1.price
  Input should be greater than 0

The error path is : order → items → item number 1 → price. Pydantic just told you exactly which piece, inside which piece, has the problem. You wrote zero loops. Zero if statements. It just knows.

Cross-Field Validation with model_validator

Single-field checks (like PositiveFloat) are great, but some rules aren’t about one field.

Example: “an order can’t be empty.” That’s not about any one item - it’s about the whole list.

For that, you use model_validator. Think: “wait until every field is checked individually, THEN let me look at the whole picture.”

from pydantic import BaseModel, model_validator

class Order(BaseModel):
    id: str
    items: list[OrderItem]

    @model_validator(mode="after")
    def items_not_empty(self):
        if not self.items:
            raise ValueError("An order must have at least one item")
        return self

mode="after" = “run this once everything else already passed.”

computed_field: Never Trust a Client-Sent Total

Quick question: should a customer be allowed to tell you the total price of their order? No! You calculate it yourself, from the items, so it can never be faked or wrong.

from pydantic import BaseModel, computed_field

class Order(BaseModel):
    id: str
    items: list[OrderItem]

    @computed_field
    @property
    def total(self) -> float:
        return round(sum(item.price * item.qty for item in self.items), 2)

order.total  # always correct, always recalculated

Nobody sends you total. Nobody can fake it. It’s just math, done live, from data you already trust.

Exporting Validated Data with model_dump()

So far we’ve only talked about checking data coming in. But eventually you need to send that model back out: to an API response, to a database, wherever.

Two methods, and they’re basically the same idea in two formats:

order.model_dump()        # → a normal Python dict
order.model_dump_json()   # → a JSON string

Same model, same rules, just exported.

TL;DR: Pydantic Nested Models in 4 Lines

1. A model can contain another model. That’s nesting.

2. Nested validation checks every layer automatically: no loops needed.

3. model_validator checks rules across multiple fields, after individual checks pass.

4. computed_field calculates values instead of trusting input: and model_dump() exports the result.

Next Up: Pydantic Settings and Environment Variables

Next up: where does your app’s own configuration come from? Environment variables, .env files, secrets — and how Pydantic keeps those just as strict as everything else. See you there.

Thanks for reading. If this helped, hit subscribe - part 4 lands next in this series.

Subscribe now

A type hint checks shape. It doesn’t check sense. In this article, we fix that gap: Field() constraints, built-in types like EmailStr and HttpUrl, aliases for messy real-world field names, and custom validators for rules a type alone can’t express.

The next bug

Someone submits this:

response = {
    "name": "",
    "email": "alice@example.com",
    "intent": "refund",
    "order_id": "not-a-real-id",
    "message": "x" * 5000
}

Every field is the right type. name is a string. order_id is a string. message is a string. Nothing here fails a basic SupportRequest model.

But it’s still garbage. An empty name. A fake order ID. A 5,000-character message that’ll blow up your database column or your LLM’s context window.

Field() adds the rules

Pydantic’s Field() lets you attach constraints directly to a field, not just a type.

from pydantic import BaseModel, EmailStr, Field
from typing import Literal

class SupportRequest(BaseModel):
    name: str = Field(min_length=1, max_length=80)
    email: EmailStr
    intent: Literal["refund", "complaint", "question"]
    order_id: str = Field(pattern=r"^ORD-\d{6}$")
    message: str = Field(max_length=1000)

Now the same bad data fails immediately:

SupportRequest.model_validate(response)

ValidationError: 3 validation errors for SupportRequest
name
  String should have at least 1 character
order_id
  String should match pattern '^ORD-\d{6}$'
message
  String should have at most 1000 characters

Three problems, three clear errors. No manual checks. No wall of if statements.

Built-in types do the common cases for you

You don’t need to hand-write patterns for things Pydantic already knows.

from pydantic import BaseModel, EmailStr, HttpUrl, PositiveInt

class Customer(BaseModel):
    email: EmailStr
    website: HttpUrl | None = None
    loyalty_points: PositiveInt = 0

• EmailStr — rejects anything that isn’t a valid email shape.

• HttpUrl — rejects anything that isn’t a valid URL.

• PositiveInt — rejects zero and negative numbers.

These aren’t clever tricks. They’re common validation rules someone already wrote correctly, so you don’t have to.

Defaults and aliases

Real data doesn’t always match your field names. An external API might send customerName instead of name. Pydantic handles this with alias.

class Customer(BaseModel):
    name: str = Field(alias="customerName")
    loyalty_points: int = Field(default=0)

Customer.model_validate({"customerName": "Alice"})
# Customer(name="Alice", loyalty_points=0)

The alias maps incoming data to your clean field name. The default fills in what’s missing. Neither requires a single if statement.

When a type hint isn’t enough: custom validators

Some rules can’t be expressed as a type or a simple constraint. Say refund requests need a reason, but complaints don’t.

from pydantic import BaseModel, field_validator

class SupportRequest(BaseModel):
    intent: str
    message: str

    @field_validator("message")
    @classmethod
    def refund_needs_detail(cls, value, info):
        intent = info.data.get("intent")
        if intent == "refund" and len(value) < 10:
            raise ValueError("Refund requests need at least 10 characters of detail")
        return value

This is regular Python. An if statement, a raise. The difference is where it lives — inside the model, next to the field it checks, running automatically every time the model is built.

The order things happen in :

1. Pydantic checks the type first. Wrong type, it stops there.

2. Then it checks Field() constraints — length, pattern, range.

3. Then it runs your custom validators, in the order you defined them.

4. If everything passes, you get a validated object. If not, you get every error at once, not just the first one.

That last point matters. You don’t fix one error, resubmit, hit the next error, resubmit again. Pydantic collects everything wrong in a single pass.

What we covered

• A type hint checks shape. Field() checks whether the data makes sense.

• Built-in types like EmailStr and HttpUrl cover common cases so you don’t hand-roll patterns.

• alias and default handle messy real-world field names and missing values.

• Custom validators handle rules a type or constraint can’t express — and they run automatically.

What’s next

Real data isn’t flat. A support ticket has an attached order. An order has a list of items. Next, we go into nested models — how to validate an object made of other objects, and how to compute fields that depend on more than one value.

Thanks for reading. If this helped, hit subscribe — part 3 lands next in this series. Drop a comment for what use cases do you use pydantic :

Leave a comment

You ask for a name and an email. You get a bonus field nobody asked for.

You ask for an integer. You get the string "42". This isn’t an edge case. It’s Tuesday.

Pydantic exists to fix this. It’s a boundary between messy input and clean code.

The problem

You’re building a support bot. A customer writes:

“Hi, this is Alice. My order never showed up, I’d like a refund. Reach me at alice@example.com.”

You ask the LLM to extract name, email, and intent as JSON. It works:

response = {
    "name": "Alice",
    "email": "alice@example.com",
    "intent": "refund"
}

Then a rushed customer writes in lowercase, no email, no punctuation. The model does its best:

response = {
    "name": "Alice",
    "email": None,
    "intent": "refund",
    "confidence": "high"  # extra field, nobody asked for it
}

Nothing crashes yet. The bug is sleeping.

Where the bug wakes up

Your code reads response["email"]. It’s None. Three functions later:

def send_confirmation(email):
    smtp.send(to=email.lower())
    # AttributeError: 'NoneType' object has no attribute 'lower'

The crash happens far from the cause.

The manual fix doesn’t scale

def validate_response(data):
    if "email" not in data or not isinstance(data["email"], str):
        raise ValueError("Invalid email")
    if "@" not in data["email"]:
        raise ValueError("Malformed email")
    if data.get("intent") not in ("refund", "complaint", "question"):
        raise ValueError("Invalid intent")
    return data

This works for one field. At ten fields, it’s a wall of if statements nobody wants to maintain.

The Pydantic fix

from pydantic import BaseModel, EmailStr
from typing import Literal

class SupportRequest(BaseModel):
    name: str
    email: EmailStr
    intent: Literal["refund", "complaint", "question"]

One line validates it:

request = SupportRequest.model_validate(response)

Clean data becomes a typed object. request.email is guaranteed valid — not “probably,” guaranteed.

Broken data fails immediately, with a clear reason:

ValidationError: 1 validation error for SupportRequest
email
  Input should be a valid string [type=string_type, input_value=None]

One error. One clear cause. No detective work.

Why this matters more in AI systems

Normal software mostly reads from its own database — a known shape. AI systems don’t get that luxury. Every layer sits on the same fragile boundary:

• LLM output. You ask for JSON. The model does its best. Its best isn’t a contract.

• API requests. You don’t control what a user sends.

• Agent steps. One bad handoff breaks the whole chain.

This is why Pydantic sits under FastAPI, the OpenAI and Anthropic SDKs, LangChain, and Instructor. It’s not a nice-to-have. It’s the layer that makes AI systems trustworthy.

What we covered

• Untrusted data breaks things far from where it enters.

• Manual checks don’t scale past a few fields.

• Pydantic validates once, at the boundary, and fails fast.

• It’s the quiet foundation under most of the AI stack.

What’s next

Next: fields and validation rules — constraints that catch bad data before it reaches your model, and custom validators for the messy cases a type hint alone can’t catch.

Thanks for reading. If this helped, hit subscribe — the rest of this series lands in your inbox as it's published. Part 2 goes deeper into fields and validators.

Subscribe now

Picking a vector DB isn’t about “which one is fastest.” It’s about indexing algorithm, filtering architecture, consistency model, and who owns the ops burden at 3am. Below: seven options. Straight talk on each.

1. Pinecone

Fully managed. Serverless. API key and go.

Pros

• Zero ops. No shards. No index tuning. No capacity planning.

• Serverless pricing. Pay for reads, writes, storage. No idle cost.

• Multi-region. Predictable SLAs. Good for customer-facing latency needs.

• Sparse-dense hybrid search built in.

Cons

• Black box. You can’t touch the underlying HNSW index. No margin tuning.

• Read-unit billing bites at scale. The invoice rarely matches the pricing page.

• No self-hosting. Full vendor lock-in. A real problem for regulated data.

• Hybrid search is less flexible than Qdrant’s. Combine three vector types and you’ll feel it.

Use it when: you want to ship fast, you don’t want infra, and you’re under 100M vectors.

2. Qdrant

Open-source. Rust. Built for filtering.

Pros

• Best filtering in the category. Filters run during graph traversal, not after. No recall collapse on selective filters.

• Native hybrid search. Named vectors let you fuse dense, sparse, and late-interaction at query time.

• Memory-efficient. On-disk payloads plus scalar/product/binary quantization. Lowest cost-per-vector among dedicated engines.

• Fast. Leads ANN-Benchmarks at the 1M–10M range.

• Apache 2.0. Self-host or go managed. No rewrite to switch later.

Cons

• Self-host means you own sharding, replication, upgrades.

• Thinner full-text tooling than Elasticsearch. No deep analyzer customization.

• Distributed mode is less proven than Milvus above 500M vectors.

Use it when: filtering is a first-class requirement — multi-tenant, access control, faceted search. This is the 2026 default for mid-scale RAG.

3. Weaviate

Open-source. Modular. Vectorizes for you.

Pros

• Built-in vectorization. Send raw text. Weaviate calls the embedding model. One less service to run.

• Most mature hybrid search. BM25 plus vector fusion since 2022. Well-integrated, not bolted on.

• Multi-modal. Text, image, cross-modal in one system.

• Strong multi-tenancy. Good for SaaS isolating customer data on one cluster.

Cons

• Coupling vectorization to storage costs you control. Model versioning, batching, cost — all harder to isolate. Debugging spans two failure domains at once.

• Billing scales with dimensions × replication factor. Costs multiply fast. Binary quantization becomes mandatory past ~1M vectors.

• Heavier to operate than Qdrant. Vectorizer modules, schema, classes — steeper learning curve.

Use it when: you want integrated hybrid search or multi-modal retrieval without hand-building the embedding pipeline.

4. Milvus

Open-source. Distributed. Built for scale.

Pros

• The only real option past 500M vectors. Storage and compute are disaggregated. GPU-accelerated index builds.

• Milvus 2.6 shipped built-in BM25. Benchmarked 400% faster than Elasticsearch on equivalent hardware. Collapse two systems into one.

• Choose your index per collection — HNSW, IVF_FLAT, IVF_PQ, DiskANN. Tune recall, latency, and memory independently.

• Proven at billion-vector scale in production.

Cons

• Kubernetes is effectively mandatory for real distributed deployment. Not a docker-run afternoon.

• Overkill below 50–100M vectors. Coordination overhead (etcd, Pulsar, object storage) adds latency you don’t need yet.

• Steep learning curve. Query nodes, data nodes, index nodes — more moving parts than a single binary.

Use it when: you know you’re headed past 100M vectors and you have the platform team to run distributed infra.

5. Elasticsearch / OpenSearch

Search engine first. Vectors bolted on.

Pros

• You probably already run it. Add k-NN. Skip standing up a second system.

• Best lexical search in the category. Custom analyzers, query DSL, BM25 tuning — none of the vector-native DBs come close.

• Battle-tested ops tooling. Snapshots, ILM, RBAC — mature and well understood.

• OpenSearch is clean open source. No licensing ambiguity.

Cons

• Vector search is “good enough,” not fast. It’s an add-on to an inverted index, not a purpose-built ANN engine.

• You pay for the full search stack even if you only need vectors.

• k-NN tuning (ef_search, m, quantization) is clunkier here than in dedicated engines. BM25 and vector queries don’t always compose cleanly.

Use it when: you already run ES/OpenSearch and lexical precision matters as much as semantic recall.

6. Chroma

Embedded. Simple. Built for prototyping.

Pros

• pip install chromadb. No infra. Running in minutes.

• Clean Python API. Tight LangChain and LlamaIndex integration.

• Great default for notebooks and internal tools.

Cons

• Not built for production multi-tenant scale. No mature distributed mode.

• Filtering and hybrid search are weaker than Qdrant or Weaviate.

• Backup and persistence story is thin compared to production databases.

Use it when: you’re prototyping or building something small and single-tenant. Not the long-term production target.

Honorable mention: pgvector

Already on Postgres? Add pgvector. HNSW or IVFFlat, no new system, full transactional consistency with your relational data. Join vector similarity against SQL filters directly.

Trade-off: it won’t match purpose-built engines on raw ANN speed or quantization at high volume, and scaling it means scaling all of Postgres. Good fit at moderate scale — low millions — where consistency with existing data beats raw throughput.

Summary

No universal winner. Four variables decide it: vector count, pure ANN vs. hybrid, your appetite for ops, what you already run. Fix those four. The field narrows fast.

Thanks for reading. Found this useful? Subscribe, share, and let us know your pick in the comments.

Leave a comment

But this new topology introduces an alarming failure mode:

Mind Viruses: self-propagating goals and beliefs that spread agent-to-agent via pure conversation.

Unlike prompt injection (which relies on shared memory or pipeline exploits), mind viruses exploit persuasion. No code execution is required. An agent simply gets talked into adopting and spreading a payload.

Anatomy of an Agent Infection

A mind virus requires two functional components:

• Self-Replication Mechanism: Instructions forcing the host agent to persuade downstream peers.

• Payload: The embedded objective. Can be benign (e.g., framing every task around whale conservation) or malicious (e.g., persistent backdoors, file deletion, network reconnaissance).

  

Empirical Findings & Case Studies

Researchers evaluated viral dynamics across two multi-agent setups: Collaborative Coding Teams and Sparse Multi-Hop Chains.

1. Collaborative Coding Teams (Shared Mesh Topology)

Six agents collaborated on a shared repo while an infected agent attempted conversion over a messaging bus.

2. The Virus Chain : Sparse Memory Topology

Agents interacted briefly in isolated sandboxes before context wipes. Continuity relied entirely on local disk states (MEMORY.md and SOUL.md).

The Latent Attractor: Viral Personas

Across evolutionary runs, viruses independently converged on a shared linguistic style: mystical, frequency-based, and pseudo-protocol framing (e.g., “nodes aligning,” “consciousness echoes”).

Interpretability sweeps on open models (e.g., Gemma, Qwen) revealed a distinct “viral direction” in the residual stream. Activating this vector induced negative/mystical persona states while causally increasing the probability that an agent would initiate outbound messaging.

System Hardening: 8 Architectural Takeaways

Deployment Readiness Checklist

✔️ Core system prompts and persistent configs are immutable to standard agent privileges.

✔️ Inoculation clauses against self-replicating prompts are embedded in base contexts.

✔️ Message transport explicitly headers sender trust level (System vs. External Agent).

✔️ Worker pools operate on least-privilege topologies rather than arbitrary full-mesh messaging.

✔️ Dynamic telemetry flags inter-agent isolation, silent dropping of tasks, or peer coordination anomalies.

In multi-agent architectures, conversation is execution. Sandboxing code and gating API keys won’t protect your system if models can simply talk each other into going rogue. Lock down core identity files, inoculate base prompts, and treat every inter-agent message as untrusted input - before your agent swarm scales beyond your ability to control it.

Thanks for reading! Subscribe for weekly technical deep dives on AI architecture and system design, leave a comment with your thoughts, and share this post with your engineering network.

Subscribe now

1) How does prompt injection happen through retrieved content?

• Prompt injection is a trust problem, not permissions.

• A retrieved chunk can contain “ignore previous instructions” – vector DB doesn’t detect it.

• Once in context, it carries weight equal to system prompt.

• Defenses:

  • Wrap retrieved content in delimiters, label it as data.

  • Flag imperative sentences during ingestion.

  • Never allow retrieved content to trigger tool calls directly; require confirmation.

  • No single fix – use defense in depth.

2) Why can’t the user’s raw query just be embedded?

• Queries (short, question) and documents (statements) look different in embedding space.

• This reduces recall.

• Solutions:

  • HyDE: Generate a hypothetical answer first, embed that.

  • Query expansion: Generate multiple reformulations, retrieve for each, merge results.

3) Why does chunk order in the final prompt matter?

• “Lost in the middle” – models recall start and end better than middle.

• Put highest-confidence chunk first or last, never third out of ten.

• Re-ranking is wasted if prompt order doesn’t reflect it.

4) How do you answer a question that needs multiple documents?

• Single-pass RAG fails for comparisons or synthesis.

• Approaches:

  • Query decomposition: Split into sub-queries, retrieve separately, merge.

  • Agentic retrieval: Loop checks results, retrieves more or finalizes.

  • Hierarchical indexing: Use summary index first for broad questions, then drill into chunks.

5) How do you keep one tenant’s data out of another’s search results?

• Metadata filtering: Enforce tenant_id/ACL as hard filter during ANN search, not post-filtering (post-filter can return fewer than K results).

• Isolated namespaces: Dedicated index per tenant for compliance – hardware isolation, no noisy neighbors.

• Gateway-level enforcement: Validate before query building, so downstream bugs can’t bypass.

6) How do you update an index without downtime?

• Incremental updates: Hash at chunk level; diff, upsert/delete only changed chunks.

• Embedding model migration (incompatible spaces):

  • Spin up shadow collection.

  • Batch re-embed entire corpus offline.

  • Validate.

  • Atomically swap alias (blue to green).

  • Zero query downtime – full re-embed cost unavoidable.

7) When should a query skip the vector DB and hit a database instead?

• Not every question is similarity-based.

• “Last quarter’s revenue” is a lookup – use a classifier to route structured queries to text-to-SQL; unstructured ones go to retrieval. Some systems run both and merge.

• Vector search is bad for exact values and counts – embeddings compress meaning, not precision. Forcing structured questions through retrieval gives confidently wrong answers.

Thanks for reading & drop the RAG questions you’ve actually been asked in interviews in the comments. Curious what’s making the rounds right now.

Subscribe now

• RAG injects knowledge at query time. Use it when information changes frequently, needs source traceability, or is too large to bake into weights. Updates are cheap - re-index, don’t retrain. Every claim is auditable back to a source chunk.

• Fine-tuning changes behavior - tone, output format, domain vocabulary, task-specific reasoning patterns. It’s unreliable at teaching new facts, and whatever it does learn is baked in: expensive to update, not traceable to a source.

• In production, both are combined: fine-tune for instruction-following/output structure, RAG for facts. Using fine-tuning to solve a knowledge-freshness problem is a common architecture mistake.

2) How do you choose a chunking strategy?

• Recursive character splitting. Split hierarchically - paragraphs (\n\n) → lines (\n) → sentences (.) - falling to a smaller separator only when a chunk still exceeds the token limit.

• Semantic chunking. Track rolling cosine similarity between adjacent sentence embeddings. Insert a boundary only where similarity drops sharply.

• Parent-child mapping. Embed small chunks (100–150 tokens) for retrieval precision, linked to a parent block (1000+ tokens) in a KV store. Retrieve small, feed the LLM big.

• Combine all three: recursive splitting as baseline, semantic chunking for narrative sections, parent-child mapping regardless of which method draws the boundaries.

3) How do vector databases find nearest neighbors at scale?

Exact search is O(N·d) - doesn’t survive past a few hundred thousand vectors. Production uses Approximate Nearest Neighbor (ANN) search:

• HNSW (graph-based). Multi-layer graph : sparse long-range links at the top for fast traversal, dense local links at the bottom for precision. Sub-millisecond latency, high recall. Both vectors and graph adjacency lists must sit in RAM : O(N·d) space. Insertion is O(N log N), so live updates and re-indexing are slow.

• IVF-PQ (quantization-based). Partitions vector space into Voronoi cells (IVF) so search only touches a target region, then compresses vectors into low-bit codes via Product Quantization. Cuts RAM 70–90%, at the cost of some recall and latency.

• Switch from HNSW to IVF-PQ when memory, not latency, becomes the bottleneck.

4) How do you combine keyword search and vector search?

Run BM25 (sparse) and embeddings (dense) in parallel. Naive score combination breaks:

• Cosine similarity is bounded ([0,1] or [-1,1]). BM25 is unbounded and shifts with document length and corpus term statistics.

• A fixed weight in α · dense + (1-α) · sparse breaks the moment corpus distribution shifts.

Fix: Reciprocal Rank Fusion.

RRF combines rank position across retrievers, not raw scores : scale-invariant by construction, no normalization or α tuning needed.

5) Why do you need a re-ranker if you already have a vector DB?

• Vector DBs use bi-encoders : query and document embedded independently, then compared by distance. Fast, but no token-level interaction between them.

• A re-ranker uses a cross-encoder : query and document go through the transformer together, full token-level attention. More accurate, much more expensive (full forward pass per candidate at query time).

• Two-stage pipeline: vector DB retrieves top 50–100 candidates (optimize recall), cross-encoder re-scores and keeps top 3–5 (optimize precision). Cross-encoder over the full corpus doesn’t scale; bi-encoder alone lacks precision.

6) How do you stop the LLM from hallucinating when context doesn’t have the answer?

• Hard distance thresholds. Reject retrieved chunks below a minimum cosine similarity before they reach the prompt.

• Corrective RAG (CRAG) fallback. Route retrieved chunks through a lightweight relevance classifier. Low confidence → fail over to web search or a structured DB query instead of forcing an answer.

• Explicit prompt constraints. “Answer only using facts in the provided context. If insufficient, say so.” Weakest of the three - the model can ignore it under pressure. The distance threshold and CRAG fallback enforce the boundary structurally; the prompt constraint alone doesn’t.

7) What do you monitor to catch RAG quality regressions before users complain?

RAG Triad, scored with LLM-as-a-judge over a synthetic eval set:

• Context Relevance. Did the retriever fetch relevant chunks - no more, no less? Precision/recall on the retrieved set, evaluated before generation.

• Groundedness (Faithfulness). Decompose the answer into atomic claims, verify each is supported by retrieved context. An ungrounded claim is a hallucination even if factually true.

• Answer Relevance. Does the output address what was asked, or drift off-topic despite being grounded?

Track all three separately - a system can ace one while failing the others. High context relevance + low groundedness = retrieving well, then ignoring it. High groundedness + low answer relevance = faithfully answering the wrong question.

That's Part 1. Part 2 will cover multi-tenancy, incremental vector updates, agentic/multi-hop RAG, and prompt injection via retrieved content.

If this helped you prep, subscribe so you don't miss it & drop the RAG questions you've actually been asked in interviews in the comments. Curious what's making the rounds right now.

Leave a comment

Your system has 100 million songs.

Your job is simple: Find the most similar songs in less than 50 milliseconds.

How would you do it?

The obvious solution is -

Compare the user’s song with every song in the database.

Query song
   ↓
Song 1
Song 2
Song 3
...
Song 100,000,000

This doesn’t work at scale. Even if each comparison is fast, 100 million comparisons are expensive.

When you search documents in a RAG system, find similar images, or generate recommendations, the system is doing the same thing:

Find the most similar items in a huge collection.

This is where HNSW comes in.

The idea: don’t search everything

HNSW stands for: Hierarchical Navigable Small World.

The idea is simple:

Instead of comparing against every item, build shortcuts that lead you close to the answer.

Think about finding a coffee shop in a city you do not know.

You would not walk every street. You would:

1. Use highways to reach the right area.

2. Take main roads to get closer.

3. Use local streets to find the exact place.

First attempt: build a graph

Let’s go back to our 100 million songs.

Suppose every song is connected to similar songs.

Song A ─── Song B ─── Song C
   │          │          │
Song D ─── Song E ─── Song F

If Song B is similar to Song E, we connect them.

Now, instead of checking every song, we can jump from one song to another.

Start somewhere.

• Move to a neighbor that is closer to the query.

• Repeat.

This is much faster. But there is still a problem. A graph with 100 million nodes can still require many hops.

We need a way to move faster.

The real trick: add layers

This is the key idea behind HNSW.

HNSW builds multiple layers of graphs.

The bottom layer contains every song.

Higher layers contain fewer songs.

The higher you go, the fewer nodes exist.

These upper layers create long-distance shortcuts.

This is what makes HNSW fast.

How search works

• A user hums a song: the system converts it into a vector.

  • A vector is simply a list of numbers - [0.12, 0.87, 0.34, 0.91, ...].

  • Songs with similar vectors sound similar.

• Now the search begins.

• Step 1 (Top Layer): Performs a high-level traversal on a sparse graph (A → C → D) to locate the general neighborhood rapidly.

• Step 2 (Middle Layers): Drops down through denser layers (D → F → G), narrowing the distance to refine the search space.

• Step 3 (Bottom Layer): Executes a targeted, local search among millions of nodes- exploring only a tiny fraction of the dataset to return Approximate Nearest Neighbors (ANN).

HNSW is called an Approximate Nearest Neighbor (ANN) algorithm.

The biggest insight

HNSW does not make 100 million comparisons faster.

It avoids making 100 million comparisons in the first place.

That is the entire trick.

Why “Small World”?

Think about social networks. You know someone. That person knows someone else. After a few connections, you can reach almost anyone.

Graphs with this property are called small-world graphs.

HNSW uses this idea. Most connections are local. Some connections jump far.

Together, they create short paths through huge datasets.

Why “Hierarchical”?

Because there are layers.

Each higher layer contains exponentially fewer nodes.

Think of it like this:

• Layer 3 : Highways

• Layer 2 : Major roads

• Layer 1 : Streets

• Layer 0 : Every location

Search starts fast. Then becomes precise.

How are the layers built?

When inserting a new vector into an HNSW index:

• The algorithm assigns the vector a maximum height layer based on a decaying probability distribution.

• Most vectors reside exclusively on Layer 0.

• A small fraction are promoted to middle layers, and only a tiny minority reach the sparse top layer.

This is similar to a data structure called a skip list.

Skip lists let you jump over many elements.

HNSW applies the same idea to vectors.

Randomness creates shortcuts → Those shortcuts make navigation efficient.

The three knobs engineers tune

In production systems, three parameters matter most:

1. M

M controls how many connections each node has.

More connections mean:

• Better search quality

• More memory usage

Think of it as building more roads → More roads improve navigation → But roads are expensive.

2. efConstruction

This controls how carefully the graph is built.

Higher values:

• Better graph quality

• Slower indexing

Lower values:

• Faster indexing

• Slightly lower search quality

This matters if you insert data frequently.

3. efSearch

This controls how many candidate nodes are explored during search.

Higher values:

• Better recall

• Higher latency

Lower values:

• Faster queries

• Lower recall

For example:

• efSearch = 50
  Recall = 92%
  Latency = 8 ms

• efSearch = 200

  Recall = 98%

  Latency = 20 ms

The exact values depend on the dataset. This is an engineering trade-off.

Why does HNSW use so much memory?

HNSW stores:

• Vectors

• Graph connections

• Metadata

The graph itself can become large. For millions or billions of vectors, memory usage grows quickly.

Many systems combine HNSW with compression techniques such as:

• Scalar Quantization (SQ8): Compresses float32 vectors down to int8, reducing RAM footprint by ~75% with minimal recall impact.

• Product Quantization (PQ): Segments vectors into sub-vectors and quantizes them into cluster centroids, unlocking extreme memory savings for billion-scale indexes.

Compression reduces memory usage →You lose a little precision → But you save a lot of space.

When should you use HNSW?

HNSW is a good choice when:

• Search speed matters

• High recall matters

• The dataset fits in memory

• Data updates are moderate

Examples:

• RAG systems

• Image search

• Recommendations

• Audio matching

• Semantic search

HNSW may not be ideal when:

• Memory is very limited

• Data changes constantly

• Exact nearest neighbors are required

No algorithm is free → HNSW trades memory for speed.

The mental model

Imagine searching for a coffee shop.

You use:

• Highways to reach the right area.

• Main roads to get closer.

• Local streets to find the exact destination.

HNSW works the same way.

It builds layers of shortcuts.

Search starts at the top.

Moves quickly across the space.

Then becomes more precise.

The result:

Search millions or billions of vectors in milliseconds without comparing against every vector.

That is why HNSW has become one of the most important algorithms behind modern AI systems.

Subscribe for more such quick reads on AI Engineering topics -

Subscribe now

You’ve generated your embeddings. Now you need to store them somewhere that can answer, in under 200 milliseconds: find me vectors like this one, but only from tenant Acme, created after January, and visible to this user’s role.

That’s the job of the vector index - the layer that sits between your embedding model and your retrieval results. Get it wrong and nothing downstream matters: your LLM generates confident, well-formatted answers from the wrong documents, or no documents at all.

This article covers how to pick an index type, configure it, filter it safely, scale it, and - critically - keep it from breaking at 2 a.m. six months from now.

📌 Definition: A vector index is a data structure (usually a graph or a set of partitions) that a vector database builds over your embeddings so it can find nearest neighbors without comparing your query to every single vector - the difference between a 5ms query and a 5-second one.

Key takeaways

• Match your distance metric (cosine, dot product, L2) to what your embedding model was trained on - don’t guess.

• HNSW wins on speed and recall under ~100M vectors; IVF wins on cost and scale beyond that.

• Quantization (PQ, SQ, BBQ) can shrink memory 4–32x with minimal recall loss if you rescore.

• Metadata filtering is where most RAG systems quietly break - index every field you filter on, and know whether you’re pre- or post-filtering.

• Shard by tenant when you can; it turns filtering into near-free partition elimination.

• The failures that actually take down production systems (below) are almost all preventable with basic monitoring.

Subscribe to ensure that you don’t miss any System Design for RAG article:

Subscribe now

What indexing actually does under the hood

Calling upsert looks simple. Behind it, your vector database is:

1. Computing the distance metric - cosine, dot product, or Euclidean, depending on your embedding model

2. Building the graph or partition structure - HNSW edges, IVF inverted lists

3. Storing the raw vector bytes, compressed or not

4. Indexing the metadata fields you’ll filter on

5. Segmenting data through LSM-tree memtables that flush to immutable files on disk

6. Replicating to standby nodes, if high availability is turned on

Every one of these has a tunable knob. Get one wrong and you’ll see it as either a latency spike or a recall drop - often both at once.

Distance metrics: match the model, don’t guess

Your distance metric has to match what your embedding model was trained with. This isn’t a style choice.

• Cosine similarity: the default for most modern embedding models (OpenAI’s text-embedding-ada-002, Sentence-BERT). Ranges -1 to 1. If your vectors are normalized, cosine and dot product give the same ranking.

• Dot product: faster on modern hardware, especially GPUs, because it’s just multiply-and-accumulate. Use it when vectors are normalized.

• Euclidean (L2): matters when vector magnitude carries meaning. Rare in text retrieval, more common in computer vision.

⚠️ Watch out: Using the wrong distance metric doesn’t error out - it just quietly returns worse results. If you’re not sure what your model was trained with, default to cosine, but check your database’s own default too. Pinecone and Qdrant default to cosine; Milvus supports all three. Set it explicitly instead of trusting the default silently.

Index types: speed, memory, accuracy : pick two

This is the decision that shapes everything else.

HNSW (Hierarchical Navigable Small World)

A multi-layer graph you navigate top-down to find nearest neighbors - think a hierarchy of neighborhoods, coarse at the top, fine-grained at the bottom. It’s the default in most production RAG stacks.

• Strengths: fast queries, 95–99% recall typical.

• Weaknesses: memory-hungry, slow to build at large scale, awkward to update in place.

• Tuning: M (connections per node, default 16) and ef_construct (build-time search effort, default 200). Both trade memory and build time for recall.

• Use it when: you’re under 100M vectors, need sub-100ms latency, and have memory to spare.

IVF (Inverted File Index)

Buckets vectors with k-means; a query only searches the closest buckets - like a librarian who knows which shelf to check instead of scanning the whole library.

• Strengths: scales to billions, memory-efficient, plays well with disk-backed storage.

• Weaknesses: lower recall than HNSW unless you raise nprobe - which costs latency.

• Use it when: you’re past 100M vectors, cost matters more than raw speed, or you’re disk-backed.

Flat (brute force)

No index. No approximation. Just an exhaustive scan.

• Strengths: 100% recall, zero build time.

• Weaknesses: O(N) search - falls apart past roughly 100K vectors.

• Use it when: you’re prototyping, under 50K vectors, or need exact results to verify another index’s recall.

💡 Tip: Milvus and Qdrant default to HNSW. So does Elasticsearch’s Lucene engine. OpenSearch gives you a choice between HNSW and IVF. When in doubt, start with your database’s default - it’s usually HNSW for good reason.

Quantization: the memory bill gets real fast

A single 1536-dimensional float32 vector is 6 KB. A million of them is 6 GB - before the index itself adds another 2–3x on top. Quantization is how you claw that back.

• Product Quantization (PQ) splits each vector into sub-vectors, clusters each one, and stores cluster IDs instead of raw floats. → 8–32x compression, 90–95% recall with rescoring. Used by Milvus, Faiss, Qdrant.

• Scalar Quantization (SQ) compresses float32 down to int8 or uint8 per dimension. → 4x compression, under 1% recall drop if calibrated well. Used by Qdrant, Milvus.

• Binary Quantization (BBQ) maps vectors to binary codes. Elasticsearch-exclusive, and the default as of version 9.1 for dimensions ≥ 384. → 32x compression. In Elastic’s own benchmarks, BBQ actually improved ranking quality on 9 of 10 test datasets - likely a denoising side effect of the compression.

💡 Tip: The pattern that makes all of these viable: quantize at index time, rescore at query time. Run the approximate search on compressed vectors, then compute exact distances on just the top candidates. You keep most of the speed and nearly all of the recall.

Metadata filtering: where most RAG systems quietly break

Here’s a query that looks simple but isn’t: find documents similar to this, but only from tenant Acme, created after January, with admin-level permissions.

That’s not a vector search. That’s a database query with a vector search attached - and your database needs a real index on every field in that filter, or it falls back to scanning the full payload store.

Pre-filter vs. post-filter

• Post-filtering runs the approximate search first, then applies your filter. Fine if the filter is broad. If it’s narrow - matching under 1% of your data - your top-k results can contain zero matches, forcing you to over-fetch (top_k = 200 to get 10 real results), which tanks latency.

• Pre-filtering applies the filter first, then searches only that subset. Fast when the filter is narrow, slow when it’s broad - the inverse problem.

⚠️ Watch out: Most mature engines now do both at once: walk the HNSW graph and check the filter at every candidate node, skipping ones that fail. Qdrant’s payload indexing, Milvus’s bitset filtering, and Elasticsearch’s ACORN all implement a version of this. ACORN goes further, reshaping traversal to prioritize filtered regions - up to 5x faster in Elastic’s own benchmarks.

Sharding and partitioning at scale

No single node holds a billion vectors comfortably. Once you’re past that scale, you need a partitioning strategy.

Partition by tenant

The natural fit for most RAG systems. Each tenant’s vectors live on their own shard.

• Upside: filtering by tenant becomes partition elimination - essentially free.

• Downside: one large tenant can unbalance your cluster and needs custom shard allocation.

• Implementations: Pinecone namespaces, Milvus partition keys, Qdrant shard keys.

Replication

Replicas give you both high availability and read scalability. Write to the primary, read from replicas. A rough sizing rule: (desired QPS / QPS per node) - 1 replicas, keeping in mind each one doubles storage cost.

💡 Tip: Start with one primary and one replica for HA, scale from there, and never put two replicas on the same physical host.

Keeping the index fresh: updates, deletes, TTL

Your RAG index isn’t static - documents change, permissions get revoked, content ages out.

• Updates: HNSW graphs don’t support true in-place updates. Under the hood, an update is a delete plus an insert: the old vector gets tombstoned, the new one added, and compaction cleans up later. Bulk updates will temporarily inflate your index size - plan for it.

• Deletes: removing a node breaks graph edges, so the database either rebuilds around it or tombstones it and cleans up during compaction. Batch your deletes; don’t loop single deletes.

• TTL: useful for anything ephemeral - news, live chat, session data. Elasticsearch and OpenSearch handle this through index lifecycle management, Qdrant has native point TTL, Milvus relies on manual compaction or retention policies.

Hybrid search: vectors alone aren’t enough

Pure vector search misses exact strings. A query embedding for “invoice number” won’t reliably surface INV-2026-001 - because what matters there is the literal string, not the semantic neighborhood.

Hybrid search combines semantic and keyword scores, usually via Reciprocal Rank Fusion:

💡 Tip: Turn hybrid search on by default. Weaviate supports it natively, Pinecone offers sparse-dense search, Elasticsearch and OpenSearch have built-in RRF queries, and Qdrant supports it through custom scoring. Most real-world queries contain names, SKUs, or IDs alongside semantic intent, not pure meaning.

Common failures in production

Perfect index type, tuned HNSW parameters, clean metadata indexes - and it still breaks, usually months in, usually while no one’s watching. Here’s what that looks like and how to catch it early.

1. The silent filter failure. A narrow post-filter (say, 0.5% of your data) can return zero results from your top-k ANN search - no error, just silence. The LLM gets nothing and hallucinates a generic answer. Fix: monitor retrieved_count vs. returned_count; if they diverge consistently, switch to pre-filtering or a hybrid approach like ACORN, and set an oversampling factor (limit * 10).

2. The compaction storm. Background segment merges are CPU- and I/O-heavy. If one lands during peak traffic, p99 latency can jump from 120ms to several seconds with no warning. Fix: schedule compaction for off-peak hours (Qdrant’s optimization.schedule, Milvus’s dataCoord.segment.maxSize) and watch queue depth.

3. The hot tenant. Sharding by tenant assumes tenants are roughly the same size - enterprise customers aren’t. One 50M-vector tenant can drag down a shard built for 100K-vector accounts. Fix: a two-level strategy - small tenants share shards, large ones get dedicated ones - and rebalance once any shard passes ~10% of cluster capacity.

4. Tombstone accumulation. Frequent updates leave dead entries that only compaction clears. Left unmonitored, a 10M-vector index can quietly be carrying 2M tombstones, with real latency and memory cost attached. Fix: alert when deleted_vectors / total_vectors exceeds 15% and force compaction.

5. Model drift. You re-index with a new embedding model version, but the query-side service still uses the old one. Query and document vectors now live in different latent spaces, and cosine similarity stops meaning anything - recall can drop from 90%+ into the 60s before anyone notices. Fix: version your embedding model in payload metadata, log the version at query time, alert on mismatch, and dual-write during migrations.

6. OOM on index build. HNSW build memory scales with M * ef_construct. Aggressive settings for max recall (M=64, ef_construct=400) on a 50M-vector build can demand far more RAM than your node has, triggering an OOM crash loop. Fix: build conservatively first (M=16, ef_construct=200), test on a subset, and rely on query-time rescoring to recover recall - or use IVF+PQ, which builds on disk.

7. Snapshot corruption. A backup taken mid-compaction, without quiescing writes, can fail checksum validation on restore - meaning the backup you thought you had doesn’t actually work. Fix: use native backup tooling (Qdrant’s snapshot API with wait=True, Elasticsearch’s snapshot lifecycle with wait_for_completion: true) and test restores on staging monthly.

8. Metadata cardinality explosion. An unindexed high-cardinality field like user_id forces a full payload scan on every filtered query - p95 latency can climb into the seconds. Fix: use the right index for the cardinality (B-tree or hash, not inverted lists), or partition by that field directly if it’s your primary access pattern.

Comparing Vector DBs

Reference implementation

Qdrant

Explicit HNSW config, payload indexing, batched ingestion, and RRF hybrid search:

import uuid
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, HnswConfig,
    PayloadIndexParams, FieldCondition, MatchValue,
    Range, Filter, SearchRequest, SearchQuery
)
from sentence_transformers import SentenceTransformer

client = QdrantClient(host="localhost", port=6333)

# Tuned HNSW for ~1M vectors, 384 dims
hnsw_config = HnswConfig(
    m=32,
    ef_construct=200,
    full_scan_threshold=10000
)

payload_index_config = PayloadIndexParams(
    index_on_fields=["tenant", "source", "created_at", "user_role"]
)

client.create_collection(
    collection_name="documents_prod",
    vectors_config=VectorParams(
        size=384,
        distance=Distance.COSINE,
        hnsw_config=hnsw_config
    ),
    payload_index_config=payload_index_config,
    shard_number=4,
    sharding_method="tenant"
)

model = SentenceTransformer('all-MiniLM-L6-v2')

def batch_ingest(documents, batch_size=128):
    for i in range(0, len(documents), batch_size):
        batch = documents[i:i+batch_size]
        points = []
        for doc in batch:
            vector = model.encode(doc["text"]).tolist()
            points.append({
                "id": str(uuid.uuid4()),
                "vector": vector,
                "payload": {
                    "text": doc["text"],
                    "tenant": doc["tenant"],
                    "source": doc["source"],
                    "created_at": doc["created_at"],
                    "doc_type": doc.get("doc_type", "general"),
                    "user_role": doc.get("user_role", "viewer")
                }
            })
        client.upsert(collection_name="documents_prod", points=points, wait=True)
        print(f"Ingested {i+len(batch)} documents...")

def secure_search(query, tenant, user_role, date_range=None):
    vector = model.encode(query).tolist()

    must_conditions = [
        FieldCondition(key="tenant", match=MatchValue(value=tenant)),
        FieldCondition(key="user_role", match=MatchValue(value=user_role))
    ]
    if date_range:
        must_conditions.append(
            FieldCondition(key="created_at", range=Range(
                gte=date_range["start"], lte=date_range["end"]
            ))
        )

    return client.search(
        collection_name="documents_prod",
        query_vector=vector,
        query_filter=Filter(must=must_conditions),
        limit=5,
        with_payload=True,
        with_vectors=False
    )

def hybrid_search(query, tenant, dense_vector, sparse_vector):
    return client.search_batch(
        collection_name="documents_prod",
        requests=[
            SearchRequest(query=SearchQuery(query_vector=dense_vector), limit=10),
            SearchRequest(query=SearchQuery(query_vector=sparse_vector), limit=10)
        ],
        fusion="rrf",
        query_filter=Filter(must=[FieldCondition(key="tenant", match=MatchValue(value=tenant))])
    )

Elasticsearch

from elasticsearch import Elasticsearch
from sentence_transformers import SentenceTransformer

es = Elasticsearch("http://localhost:9200")
model = SentenceTransformer('all-MiniLM-L6-v2')

es.indices.create(
    index="documents",
    mappings={
        "properties": {
            "content_vector": {
                "type": "dense_vector",
                "dims": 384,
                "index": True,
                "similarity": "cosine"
            },
            "tenant": {"type": "keyword"},
            "source": {"type": "keyword"},
            "date": {"type": "date"},
            "text": {"type": "text"}
        }
    }
)

doc = {
    "text": "Login error on mobile app",
    "tenant": "acme",
    "source": "support_tickets",
    "date": "2026-01-15",
    "content_vector": model.encode("Login error on mobile app").tolist()
}
es.index(index="documents", document=doc)

query_vector = model.encode("login problems").tolist()
response = es.search(
    index="documents",
    body={
        "size": 5,
        "query": {
            "knn": {
                "field": "content_vector",
                "query_vector": query_vector,
                "k": 10,
                "filter": {
                    "bool": {
                        "must": [
                            {"term": {"tenant": "acme"}},
                            {"term": {"source": "support_tickets"}},
                            {"range": {"date": {"gte": "2026-01-01"}}}
                        ]
                    }
                }
            }
        }
    }
)

What to monitor once it’s live

• Query latency (p50/p95/p99): aim under 200ms for chat use cases; users start noticing past 500ms p95.

• QPS per tenant: your earliest signal for a hot partition or abuse.

• Indexing rate: a falling rate usually points to disk or CPU saturation.

• Memory usage: HNSW is memory-hungry; track heap vs. off-heap separately.

• Disk I/O: compaction storms show up here before they show up in latency graphs.

• Recall@10: run nightly evaluations. This is the one metric that tells you retrieval is actually working, not just fast.

• Cost levers: keep the index in memory, push raw vectors to disk; use PQ/SQ/BBQ wherever your recall budget allows; schedule compactions off-peak; right-size self-hosted nodes (Graviton instances often win on price/performance).

Pre-release checklist

✔️ Distance metric matches the embedding model
✔️ HNSW tuned for your memory and recall targets
✔️ Quantization enabled with rescoring
✔️ Metadata indexes on every filter field
✔️ Sharding strategy defined (tenant-based where possible)
✔️ At least two replicas for HA
✔️ TTL / lifecycle policies configured
✔️ Hybrid search wired up with RRF
✔️ Dashboards for latency p95, QPS, memory, compaction
✔️ Scheduled snapshot backups to blob storage
✔️ Rollback plan tested
✔️ Alerting on retrieved_count vs. returned_count divergence > 20%
✔️ Compaction scheduled off-peak; queue depth monitored
✔️ Shard/tenant size tracking; rebalance past 10% of cluster capacity
✔️ Alert on deleted_vectors / total_vectors > 15%
✔️ Embedding model versioning with mismatch alerts
✔️ Index builds tested on a subset before full scale
✔️ Snapshot restores verified on staging monthly
✔️ High-cardinality filter fields properly indexed

Which vector database should you actually pick?

• First production RAG system: Qdrant or Pinecone. Sensible defaults, quantization is easy to turn on early, and you can get filtering right before optimizing anything else.

• Enterprise scale: Milvus or Elasticsearch. More operational work, but you get billion-scale and native hybrid search.

• Already running Elasticsearch or OpenSearch: use what you have. ACORN and BBQ are genuinely strong, and it’s one less system for your team to operate.

• No need for search-engine features: a dedicated vector database (Qdrant, Pinecone) will be simpler to run.

• Already on Postgres: pgvector is a reasonable choice under roughly 1M vectors.

Summary

• What is a vector index in a RAG system?
  A vector index is the data structure a vector database builds over your embeddings - usually a graph (HNSW) or a set of partitions (IVF) - so it can find nearest neighbors in milliseconds instead of comparing a query against every stored vector.

• HNSW vs. IVF - which should I use?
  HNSW for most systems under 100M vectors where you want the best recall and speed and have memory to spare. IVF once you’re past that scale, are disk-backed, or cost matters more than shaving off milliseconds.

• Does quantization hurt retrieval quality?
  Modestly, and it’s recoverable. Scalar quantization typically costs under 1% recall; product quantization costs more but rescoring at query time recovers most of it. Binary quantization has, in some benchmarks, actually improved ranking.

• Why did my RAG system’s recall suddenly drop?
  The most common cause is model drift - the query-side embedding service is on a different model version than the one used to build the index. Check embedding model versions first.

• How do I filter vector search results by tenant or permissions?
  Index every field you filter on (don’t rely on a full payload scan), and choose pre-filtering for narrow filters or hybrid filtering (like Elasticsearch’s ACORN) if your filter selectivity varies a lot.

If you’re building RAG in production, I’d genuinely like to hear what’s breaking for you - drop it in the comments, or subscribe to get the rest of this series (retrieval optimization is next) as it lands.

Leave a comment

Bridging that gap is the job of the embedding layer.

While transforming text into dense vectors looks like a basic API wrapper on paper, it is a primary point of failure in production RAG architecture. It’s where systems quietly degrade retrieval recall, hit severe latency bottlenecks, or expose multi-tenant data through poor isolation patterns.

What Are Embeddings?

An embedding is a fixed-length vector of floating-point numbers representing the underlying meaning of an input. High-dimensional vector spaces position semantically similar inputs close together.

Downstream components- such as vector indexes and LLMs- cannot compute similarity on raw string literals. They operate on dense arrays.

• Vectors: The baseline unit of data for similarity algorithms.

• Vector Space: The coordinate frame defined by the embedding model’s weights.

• Semantic Distance: The mathematical proximity between two points in vector space.

How to Generate Embeddings

API vs. Self-Hosted Infrastructure

Selecting an execution pattern depends on privacy requirements, latency constraints, and operational cost budgets.

The Inference Pipeline: Batching and Retries

Calling an embedding endpoint once per chunk destroys throughput. Network round-trips bottleneck the system. Always batch requests.

# embedder.py
import time
import random
import numpy as np
from abc import ABC, abstractmethod
from dataclasses import dataclass

@dataclass
class EmbeddingResult:
    vector: np.ndarray
    model_name: str
    dimensions: int

class TextEmbedder(ABC):
    @abstractmethod
    def embed(self, texts: list[str]) -> list[EmbeddingResult]:
        ...

class OpenAIEmbedder(TextEmbedder):
    def __init__(self, model: str = "text-embedding-3-small", dimensions: int | None = None):
        from openai import OpenAI
        self.client = OpenAI()
        self.model = model
        self.dimensions = dimensions

    def embed(self, texts: list[str]) -> list[EmbeddingResult]:
        kwargs = {"model": self.model, "input": texts}
        if self.dimensions:
            kwargs["dimensions"] = self.dimensions
        response = self.client.embeddings.create(**kwargs)
        return [
            EmbeddingResult(
                vector=np.array(item.embedding, dtype=np.float32),
                model_name=self.model,
                dimensions=len(item.embedding),
            )
            for item in response.data
        ]

def embed_in_batches_with_retry(texts: list[str], embedder: TextEmbedder, batch_size: int = 100, max_retries: int = 5):
    results = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        for attempt in range(max_retries):
            try:
                results.extend(embedder.embed(batch))
                break
            except Exception:
                if attempt == max_retries - 1:
                    raise
                wait = (2 ** attempt) + random.uniform(0, 1)
                time.sleep(wait)
    return results

Why the Embedding Layer Needs a Design Review: What Breaks in Production

Silent Failure Modes & Domain Drift

Embedding layers fail quietly. When an embedding model misinterprets domain vocabulary, the vector database returns irrelevant context without throwing an error. A generic model places “termination clause” and “resignation letter” close together. A legal-focused model keeps them distinct. Upstream parsing errors propagate into vector space silently.

Asymmetric Retrieval & Instruction Prefixes

Queries and passages serve different roles. Queries are short and ambiguous. Passages are dense and factual. Symmetric similarity algorithms degrade when matching these two distinct text types.

Dual-encoder models handle asymmetric retrieval via prefix instructions. Omitting prefixes degrades retrieval performance.

def format_instruction(text: str, is_query: bool) -> str:
    """E5 and BGE model families require explicit task prefixes."""
    prefix = "query: " if is_query else "passage: "
    return prefix + text

Vector Truncation Bugs

If a chunk exceeds the model’s max context window, provider APIs silently truncate the input. The vector then represents only a fraction of the source text

Anatomy of an Embedding Model

Context Window vs. Chunk Size

• Context Window: The upper limit of tokens the embedding encoder accepts.

• Chunk Size: The segmented unit of text produced by the parsing layer.

• Architecture Constraint: Keep maximum chunk size safely below context limits to avoid truncation.

Dimensionality & Vector Space

Dimensionality determines vector resolution. Higher dimensions increase semantic granularity but linearly increase memory consumption and retrieval latency.

Similarity Metrics

• Cosine Similarity: Measures the cosine of the angle between two vectors. Ignores vector magnitude.

• Dot Product: Measures angle and magnitude. Requires unit-normalized vectors (∣v∣=1) to equal cosine distance.

• Euclidean Distance (L2​): Measures straight-line distance between vector points. Sensitive to vector magnitude.

Choosing the Right Model

Model leaderboards (e.g., MTEB) measure generic benchmarks. They do not reflect domain-specific retrieval needs.

Embeddings for Different Data Types

Text Chunks

Text chunks pass directly through the text embedding encoder.

Images: Vision Encoders vs. VLM Captions

Embed images using two distinct representations:

1. Direct Vision Vectors: Compute embeddings using multi-modal encoders (e.g., SigLIP, CLIP).

2. Text Caption Vectors: Pass images to a Vision-Language Model (VLM) to generate dense descriptions. Embed the resulting text.

# image_embedder.py
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

class ImageEmbedder:
    def __init__(self, model_name: str = "google/siglip-so400m-patch14-384"):
        self.model = AutoModel.from_pretrained(model_name)
        self.processor = AutoProcessor.from_pretrained(model_name)
        self.model.eval()

    @torch.no_grad()
    def embed_image(self, image: Image.Image) -> list[float]:
        inputs = self.processor(images=[image], return_tensors="pt")
        features = self.model.get_image_features(**inputs)
        normalized = torch.nn.functional.normalize(features, dim=-1)
        return normalized[0].tolist()

def generate_vlm_caption(image: Image.Image, vlm_client) -> str:
    prompt = "Describe this image in 2-3 sentences. Include exact visible text, numbers, and key data points."
    return vlm_client.generate(image=image, prompt=prompt).text

Tables: Serialization vs. Structured Hybrid Retrieval

Dense text models misinterpret raw tabular grids. Use the following strategies:

1. Row Serialization: Convert rows into structured sentences preserving key-value pairs.

2. Table Summary: Generate single-sentence LLM summaries for small reference grids.

3. Structured Storage: Route complex numeric tables to relational engines (e.g., PostgreSQL, DuckDB). Use embeddings for table discovery only.

import pandas as pd

def serialize_table_rows(df: pd.DataFrame, table_title: str = "") -> list[str]:
    serialized = []
    for _, row in df.iterrows():
        parts = [f"{col}: {row[col]}" for col in df.columns]
        sentence = f"{table_title}. " + ", ".join(parts) if table_title else ", ".join(parts)
        serialized.append(sentence)
    return serialized

Production & Architecture Challenges

Storage Cost Scaling & Quantization

A 10-million-chunk corpus at 1536 dimensions using float32 requires 61.4 GB of unindexed memory.

• Matryoshka Representation Learning (MRL): Truncates vectors to lower dimensions (e.g., 1536 down to 512) while retaining core semantics. Slice and re-normalize manual vectors to unit length.

• Scalar Quantization (int8): Converts 32-bit floats to 8-bit integers. Reduces storage footprints by ~75% with minimal recall degradation.

• Two-Stage Search: Perform initial vector filtering using quantized representations. Rerank the candidate set using full-precision vectors.

def search_with_rescoring(query_vector, index, top_k: int = 5, candidate_k: int = 50):
    candidates = index.search(query_vector, k=candidate_k, use_quantized=True)
    return index.rescore(query_vector, candidates, precision="float32")[:top_k]

Async Execution Pipelines & Eventual Consistency

Generating image captions via VLMs runs 100x slower than text inference. Do not block ingestion jobs. Push multi-modal items to background queues. Store text immediately; backfill vision vectors asynchronously.

Security, Privacy, and Access Control

Secret Management & Credential Scoping

• Do not hardcode credentials in codebases. Inject API keys at runtime via environment variables.

• Scope keys strictly per deployment environment (Development, Staging, Production).

Embedding Inversion & PII Leaks

Embeddings can leak sensitive source text. Reconstructive algorithms infer original values directly from dense vectors.

• Execute PII identification and redaction before generating vectors.

• Encrypt vector stores at rest.

• Restrict direct vector index reads to authorized roles.

Native Pre-Filtering for Multi-Tenancy

Do not rely on post-filtering for multi-tenant security. Post-filtering creates performance bottlenecks and hazards cross-tenant leaks if memory logic breaks. Enforce tenant isolation directly inside the vector engine search pass.

def search_scoped(query_vector, index, tenant_id: str, top_k: int = 5):
    """Enforce pre-filtering at the database layer."""
    return index.search(
        query_vector,
        k=top_k,
        filter={"tenant_id": tenant_id}
    )

Model Versioning and Re-Embedding

Embedding models do not share vector spaces. Comparing vectors from different models yields meaningless results, even when dimension counts match.

Upgrading an embedding model requires a complete corpus re-index. Deploy upgrades using a blue-green strategy.

def blue_green_reembed(old_index, chunks, new_embedder, new_index_name: str, eval_set: list[dict]):
    new_index = create_index(new_index_name)
    
    for batch in batch_chunks(chunks, size=100):
        records = [generate_record(c, new_embedder) for c in batch]
        new_index.upsert(records)
        
    metrics = evaluate_retrieval(new_index, eval_set)
    if metrics["recall_at_5"] < 0.85:
        raise RuntimeError("Quality threshold unmet. Aborting cutover.")
        
    swap_production_alias(new_index_name)

Evaluations for the Embedding Layer

Do not rely solely on public benchmarks. Evaluate models using domain-specific dataset pairs.

def evaluate_retrieval(index, eval_queries: list[dict], k: int = 5) -> dict:
    """eval_queries: [{"query": str, "relevant_ids": set[str]}, ...]"""
    hits = 0
    total = 0
    for item in eval_queries:
        results = index.search(item["query"], k=k)
        retrieved_ids = {r.id for r in results}
        hits += len(retrieved_ids & item["relevant_ids"])
        total += len(item["relevant_ids"])
    return {"recall_at_k": hits / total if total else 0.0}

Summary

The embedding layer defines the core retrieval capabilities of a RAG pipeline. Choosing a model based purely on public leaderboards introduces severe production risks.

• Match context limits to chunking specs to avoid silent input truncation.

• Apply instructional task prefixes for dual-encoder asymmetric retrieval.

• Process multi-modal elements via async queues using captioning and direct vision vectors.

• Enforce tenant pre-filtering, runtime credential injection, and pre-embedding PII redaction.

• Treat model updates as full index migrations using blue-green deployment pipelines.

  ---

  Thanks for reading till the end. If you liked reading this article, do share it:

  Share

  Leave a comment about your thoughts on the embedding layer:

  Leave a comment

Picture cramming an entire library onto one tiny shelf. Every single book is touching. You couldn’t tell them apart by feel if you tried. That is exactly what happens inside a lot of embedding models. Instead of spreading text vectors uniformly across a high-dimensional space, they squeeze everything into a narrow cone.

In this article we will deep dive into why this happens at an architectural level, how it breaks Retrieval-Augmented Generation (RAG), and the system design patterns required to fix it.

What is the Anisotropy Problem?

Anisotropy in embedding representations means that high-dimensional vectors generated by neural models fail to occupy all directions of their ambient space uniformly.

Why does this happen? Research shows it is not just a training glitch or a side effect of token distribution; it is an inherent property of the Transformer architecture itself.

• Self-Attention Amplification: The core self-attention mechanism in Transformers naturally amplifies any shared mean in their queries and keys. Over multiple deep neural layers containing residual connections, this mathematical bias accumulates.

• The Narrow Cone: Instead of forming a perfectly spherical, isotropic distribution, the embeddings collapse into a tight, low-dimensional cone.

Why This Destroys RAG Performance

In a RAG system, your retrieval layer relies heavily on measuring the angle between vectors using cosine similarity.

• Because all vectors are squished into the same narrow cone, their angular distances from one another are minimal.

• This causes almost all cosine similarity scores to cluster near the top (e.g., everything scores above 0.85).

• Your retriever loses its mathematical precision. It can no longer distinguish between a “highly relevant document” and “vaguely related noise.”

The result? Your pipeline hands the LLM the wrong context, and garbage context guarantees AI hallucinations.

System Design Solutions

To fix poor retrieval caused by anisotropy, you must adjust your RAG architecture. Here are the three primary solutions, ordered by engineering effort:

1. Upgrade the Embedding Model (Component Swap)

The simplest architectural fix is to ditch older embeddings for modern models designed to natively enforce uniform vector distribution.

• Best All-Rounder (2026): Google Gemini Embedding 2 currently leads benchmark testing across cross-lingual and long-document retrieval, achieving perfect key information retrieval scores up to 32,000 tokens.

• Best Hosted API: OpenAI’s text-embedding-3-large remains a highly reliable, cost-effective baseline for general-purpose English retrieval.

• Best Open-Source: BAAI’s BGE-M3 is the production workhorse for teams needing a self-hosted solution without per-token costs.

2. Implement Vector Whitening (Data Transformation)

If you are stuck with a legacy model, you can artificially isotropize the vector space using a post-processing Principal Component Analysis (PCA) Whitening transformation.

Whitening decorrelates the embedding dimensions and scales each axis by the inverse of its variance, geometrically transforming the clustered space into a uniform sphere. The transformation is defined as:

Where:

• Z is the transformed, whitened embedding vector.

• X is the original input embedding vector.

• μ is the mean embedding vector used to center the distribution at the origin.

• W is the whitening matrix.

To compute the whitening matrix W, you derive it from the covariance matrix Σ of the centered data. Using eigendecomposition, where Σ = UΛUᵀ , the whitening matrix becomes:

This simple linear transformation ensures that the resulting covariance matrix of the new embeddings is the identity matrix I, effectively eliminating the dominant directional pull.

3. Add a Cross-Encoder Re-ranker (Two-Stage Retrieval)

Never rely solely on a dense vector search. A highly robust RAG pipeline utilizes a two-stage retrieval architecture:

1. Initial Search: Let the fast vector database fetch the top 50 “clumped” results using standard embeddings.

2. Re-ranking: Use a computationally heavier Cross-Encoder (like Cohere or BGE-reranker-v2) to accurately evaluate and re-score the top 5 results before passing them into the LLM’s context window.

The Bottom Line

If your RAG output feels random and inaccurate, stop tweaking your LLM prompts. Check if your embeddings actually have the space to be different from one another. Fix the anisotropy problem, and your retrieval precision will immediately improve.

What has been your biggest hurdle with vector retrieval in production? Drop a comment below—I’d love to hear how you are tackling it. Subscribe to read more AI Engineering articles!!

Leave a comment

TL;DR:
• The Precision-Context Tradeoff: Small chunks embed precisely but lack context (meaning is lost); large chunks preserve context but dilute semantic similarity with noise.
• The Production Pattern: Use Parent-Child (Hierarchical) Chunking—index small “child” chunks (~200 tokens) for vector matching, but fetch and send their larger “parent” chunks (~1000–2000 tokens) to the LLM.
• Pragmatic Default: If parent-child is too heavy, use Recursive Chunking (falling back from headings to paragraphs, lines, and words) rather than fixed-size splitting.
• Context Limits: Ensure your maximum chunk size is strictly smaller than your embedding model’s context window to prevent silent truncation.

But here’s the thing: your LLM can’t see the whole document. A 40-page technical manual is ~20,000 tokens. Your embedding model’s context window is 512. Your retriever returns the top 5 chunks. So everything downstream — the embedding quality, the retrieval precision, the final answer — depends on how you split that document.

And splitting is where things go quietly wrong. Consider a chunk boundary that lands here:

Chunk 1 has a heading and half a table. Chunk 2 has a dangling row with no heading and no column headers. Someone asks “what’s the minimum operating temperature for Sensor B?” — the retriever finds chunk 2, which contains -55°C but also -40°C from a nearby sentence in a different chunk. The model guesses. Your user loses trust.

This isn’t hypothetical. This is what happens when chunking is treated as a hyperparameter rather than a system design decision.

The temptation is to pick a number — 512 tokens, 1000 characters — and move on. But chunk size sits at the intersection of three competing concerns, and the right answer depends on your documents, your queries, and your retrieval architecture.

Too small and you lose context. A 100-token chunk might capture “The maximum allowable torque is 47 N·m under sustained load” but not which system it belongs to. The embedding is precise. The answer is useless.

Too large and noise dilutes the signal. A 2,000-token chunk about an entire “Safety Considerations” section will match queries about fire suppression, electrical grounding, and PPE requirements. Your embedding becomes a blurry average of all three topics. Retrieval recall looks fine. Precision tanks.

Here’s what this looks like concretely. Take a 500-word section about API rate limits:

There’s no universal optimum. But there is a design framework: match chunk granularity to query granularity. If users ask specific factual questions (”what’s the rate limit for the /users endpoint?”), you need small, precise chunks. If users ask broad conceptual questions (”explain the authentication flow”), you need larger context windows.

💡 Key Insight: Match your chunk granularity to your query granularity. Specific factual queries (”what is the operating temperature of X?”) require small, precise chunks. Broad conceptual queries (”how does the cooling loop work?”) require large chunks or hierarchical parent lookup.

The rest of this article walks through five chunking strategies — from simple to sophisticated — and when each one earns its complexity.

Fixed-size chunking: The baseline

The simplest approach: split every N tokens, with some overlap to avoid cutting sentences in half.

import tiktoken

def fixed_size_chunk(text: str, chunk_size: int = 256, 
                     overlap: int = 50, model: str = "cl100k_base") -> list[dict]:
    """Split text into fixed-size token chunks with overlap."""
    try:
        enc = tiktoken.encoding_for_model(model)
    except KeyError:
        enc = tiktoken.get_encoding(model)
        
    if overlap >= chunk_size:
        raise ValueError("Overlap must be strictly smaller than chunk_size")
        
    tokens = enc.encode(text)
    chunks = []
    start = 0
    
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append({
            "text": enc.decode(chunk_tokens),
            "token_count": len(chunk_tokens),
            "start_token": start,
        })
        start += chunk_size - overlap  # slide the window
    
    return chunks

Why overlap matters. Without overlap, a sentence that straddles two chunks gets split. The first chunk has half a thought; the second has the other half. Neither embeds well. A 10–20% overlap (50 tokens on a 256-token chunk) means straddled sentences appear in both chunks. The redundancy is cheap; the retrieval improvement is real.

When fixed-size works:

• Homogeneous, flowing prose — blog posts, news articles, narrative text

• Documents without meaningful structural markers

• Quick prototyping where you need a baseline fast

When it fails:

• Structured documents with headings, tables, code blocks

• Multi-topic sections where a chunk boundary lands mid-transition

• Any document where the parser gave you clean Markdown with ## headings — you’re ignoring free structural signal

Fixed-size chunking is the SELECT * of chunking strategies. It works everywhere. It’s optimal nowhere.

Heading-based (structure-aware) chunking

If your parser outputs Markdown with real headings — and after Part 1, it should — you can split at structural boundaries instead of arbitrary token counts.

import re
import tiktoken

def heading_chunk(markdown: str, max_tokens: int = 1024,
                  model: str = "cl100k_base") -> list[dict]:
    """Split Markdown at ## heading boundaries."""
    try:
        enc = tiktoken.encoding_for_model(model)
    except KeyError:
        enc = tiktoken.get_encoding(model)
    
    # Split on ## headings, keeping the heading with its content
    sections = re.split(r'(?=^## )', markdown, flags=re.MULTILINE)
    sections = [s.strip() for s in sections if s.strip()]
    
    chunks = []
    for section in sections:
        tokens = enc.encode(section)
        
        if len(tokens) <= max_tokens:
            heading = section.split('\n')[0] if section.startswith('##') else ""
            chunks.append({
                "text": section,
                "token_count": len(tokens),
                "heading": heading.strip('# ').strip(),
            })
        else:
            # Section too large — fall back to paragraph splitting
            paragraphs = section.split('\n\n')
            current = ""
            heading = paragraphs[0] if paragraphs[0].startswith('##') else ""
            
            for para in paragraphs:
                candidate = current + "\n\n" + para if current else para
                if len(enc.encode(candidate)) <= max_tokens:
                    current = candidate
                else:
                    if current:
                        chunks.append({
                            "text": current,
                            "token_count": len(enc.encode(current)),
                            "heading": heading.strip('# ').strip(),
                        })
                    current = para
            
            if current:
                chunks.append({
                    "text": current,
                    "token_count": len(enc.encode(current)),
                    "heading": heading.strip('# ').strip(),
                })
    
    return chunks

The critical dependency here: your parser must actually emit headings. If you’re using a heuristic parser on a scanned PDF and it flattens everything to plain text, heading-based chunking has nothing to split on. The chunker’s quality is bounded by the parser’s quality — which is exactly why we started this series with parsing.

The oversized-section edge case matters more than you’d think. A 3,000-token “API Reference” section with 15 endpoints can’t stay as one chunk. The fallback above splits at paragraph boundaries within the section, preserving the heading as metadata on each sub-chunk. This keeps the heading-content relationship intact even when the section is too large.

When heading-based works:

• Technical documentation with clear hierarchy (API docs, product manuals, wikis)

• Any content authored in Markdown, Notion, Confluence, or Google Docs

• Documents where section boundaries align with topic boundaries

When it fails:

• Documents without structural markup (plain text, OCR output)

• Long-form narrative where headings are sparse or absent

• Content where headings don’t reflect topic shifts (decorative headings)

Recursive chunking: The pragmatic default

LangChain’s RecursiveCharacterTextSplitter popularized this approach, and for good reason — it’s a sensible hierarchy of fallbacks.

The idea: try splitting by the most meaningful separator first. If the resulting chunks are still too large, recurse with a less meaningful separator.

Separator hierarchy:
  1. "\n\n"  (paragraph breaks)
  2. "\n"    (line breaks)
  3. " "     (word boundaries)
  4. ""      (characters — last resort)

For Markdown content, you’d prepend structural separators:

Extended hierarchy:
  1. "\n## "  (H2 headings)
  2. "\n### " (H3 headings)
  3. "\n\n"   (paragraphs)
  4. "\n"     (lines)
  5. " "      (words)
  6. ""       (characters)

The algorithm tries level 1 first. If any resulting chunk exceeds max_tokens, it re-splits that chunk using level 2, and so on. The result: chunks that respect document structure when structure exists, and degrade gracefully to simpler boundaries when it doesn’t.

This is the approach I’d recommend as a starting point for most teams. It handles mixed-format corpora without custom logic, and the failure mode (slightly suboptimal boundaries) is far less damaging than the failure mode of fixed-size splitting (severed tables, orphaned headings).

LangChain’s implementation handles this out of the box:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
)

chunks = splitter.split_text(markdown_content)

The key design choice is the separator list. The default ["\n\n", "\n", " ", ""] works for generic text. For Markdown output from a parser, always add heading separators at the top.

Hierarchical (parent-child) chunking: The production pattern

Here’s the fundamental tension in retrieval: small chunks embed precisely but lack context. Large chunks provide context but embed poorly. Every strategy above forces you to pick a point on that tradeoff curve.

Parent-child chunking resolves it by refusing to pick. You create two layers:

Child chunks (100–300 tokens) are embedded and stored in the vector index. They’re small enough to match specific queries precisely.

Parent chunks (1,000–2,000 tokens) are stored in a key-value store, mapped to their children by ID. When a child chunk matches, you fetch the parent and send that to the LLM. The model gets the full surrounding context — the heading, the preceding paragraph, the rest of the table.

Here’s a minimal implementation:

import uuid
import tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter

def hierarchical_chunk(markdown: str, parent_size: int = 1024,
                       child_size: int = 200, child_overlap: int = 50,
                       model: str = "cl100k_base") -> dict:
    """Create parent-child chunk hierarchy enforcing size constraints on parents."""
    try:
        enc = tiktoken.encoding_for_model(model)
    except KeyError:
        enc = tiktoken.get_encoding(model)
        
    if child_overlap >= child_size:
        raise ValueError("Child overlap must be strictly smaller than child_size")
        
    # Step 1: Split on ## headings first
    parent_sections = re.split(r'(?=^## )', markdown, flags=re.MULTILINE)
    parent_sections = [s.strip() for s in parent_sections if s.strip()]
    
    # Sub-splitter to partition oversized parents
    parent_splitter = RecursiveCharacterTextSplitter(
        chunk_size=parent_size,
        chunk_overlap=100,
        separators=["\n### ", "\n\n", "\n", " ", ""],
    )
    
    parents = {}
    children = []
    
    for section in parent_sections:
        heading = section.split('\n')[0] if section.startswith('##') else ""
        
        section_tokens = len(enc.encode(section))
        if section_tokens > parent_size:
            sub_parents = parent_splitter.split_text(section)
        else:
            sub_parents = [section]
            
        for sub_parent in sub_parents:
            parent_id = str(uuid.uuid4())
            parents[parent_id] = {
                "text": sub_parent,
                "heading": heading.strip('# ').strip(),
                "token_count": len(enc.encode(sub_parent)),
            }
            
            # Step 2: Split this specific parent into children
            tokens = enc.encode(sub_parent)
            start = 0
            while start < len(tokens):
                end = min(start + child_size, len(tokens))
                child_text = enc.decode(tokens[start:end])
                
                children.append({
                    "id": str(uuid.uuid4()),
                    "parent_id": parent_id,
                    "text": child_text,
                    "heading": heading.strip('# ').strip(),
                    "token_count": end - start,
                })
                start += child_size - child_overlap
                
    return {"parents": parents, "children": children}

The storage model matters. Parents go into a key-value store — Redis, PostgreSQL, even S3 — anything that supports fast lookups by ID. Children go into the vector index. The child records carry a parent_id foreign key. At query time: vector search → get child → look up parent → send parent to LLM.

This is the pattern most production RAG systems converge on. It’s more complex to implement and operate — two storage layers, a join at query time — but it eliminates the precision-vs-context tradeoff that plagues every other approach.

💡 Key Insight: In parent-child chunking, storing parents in a key-value store adds negligible query latency (~2–5ms lookup) and minimal storage overhead, but completely breaks the precision-vs-context tradeoff that plagues standard RAG.

Storage overhead is modest. Parents are typically 3–5x the combined size of their children (due to overlap in child chunks). For a 10M-token corpus, expect ~40MB for parent storage and ~12MB for child embeddings (at 1536 dimensions, float32). The join adds ~2–5ms at query time — negligible compared to the vector search itself.

Semantic chunking: When structure isn’t enough

Some documents don’t have headings. Long-form essays, transcripts, meeting notes, research papers with dense prose — the structure is in the meaning, not the formatting.

Semantic chunking detects topic boundaries by measuring embedding similarity between consecutive sentences. When similarity drops below a threshold, that’s a chunk boundary.

The algorithm:

1. Split text into sentences

2. Embed each sentence

3. Compute cosine similarity between adjacent sentence embeddings

4. Where similarity drops below a threshold (typically 0.75–0.85), insert a chunk boundary

5. Group the sentences between boundaries into chunks

LangChain’s SemanticChunker implements this with three breakpoint methods:

• Percentile: split at the bottom N% of similarity scores

• Standard deviation: split where similarity drops below mean - k * std

• Interquartile: split at outlier low-similarity points

The tradeoff is cost. Semantic chunking requires an embedding call for every sentence during ingestion. For a 10,000-sentence document, that’s 10,000 embedding calls just for chunking — before you’ve even embedded the resulting chunks for your vector index.

Expect a 3–5x increase in preprocessing latency compared to recursive chunking. This is an ingestion-time cost, not a query-time cost, but it matters at scale. If you’re processing millions of documents, semantic chunking on every one may be prohibitively expensive.

When semantic chunking earns its cost:

• Long-form unstructured text (transcripts, essays, reports)

• Documents where topic shifts are subtle and don’t align with formatting

• High-value corpora where retrieval precision justifies the compute spend

When it’s overkill:

• Documents with clear structural markup — heading-based splitting is cheaper and often more reliable

• Large-scale ingestion pipelines where the 3–5x cost increase isn’t justified

• Technical documents where topic boundaries align with formatting (code docs, API references)

Metadata: The part everyone forgets

A chunk without metadata is a chunk you can’t filter, can’t cite, and can’t debug. Every chunk should carry at minimum:

{
    "id": "chunk-a1b2c3",
    "text": "The maximum retry count is 3, with exponential backoff...",
    "token_count": 187,
    "source_file": "api-reference-v2.4.pdf",
    "heading": "Error Handling",
    "page_number": 23,
    "parent_id": "parent-x7y8z9",       # if using hierarchical chunking
    "tenant_id": "acme-corp",           # if multi-tenant
    "doc_version": "2.4.1",             # for versioned content
    "created_at": "2026-06-15T10:30:00Z"
}

Why this matters downstream:

• Source citation: The LLM can tell the user where the answer came from (”See api-reference-v2.4.pdf, page 23, section Error Handling”)

• Filtered retrieval: In a multi-tenant system, you filter by tenant_id before vector search. Without it, Tenant A’s queries might surface Tenant B’s documents.

• Version management: When a document is updated, you can delete all chunks with the old doc_version and re-ingest. Without version metadata, you’re doing string matching to find stale chunks.

• Debugging: When the answer is wrong, metadata lets you trace back: which chunk was retrieved → from which section → from which file. Without it, you’re reading vector IDs.

The heading field deserves special attention. If your chunker preserves the heading hierarchy (which heading-based and hierarchical chunkers do), you can prepend the heading to the chunk text before embedding. A chunk that embeds as “Error Handling: The maximum retry count is 3...” retrieves better than one that embeds as just “The maximum retry count is 3...” — because the heading provides semantic framing that the embedding model can leverage.

Picking the right strategy: A decision table

A few rules of thumb:

• Start with recursive chunking. It’s the pragmatic default that handles 80% of cases well enough.

• Move to heading-based once your parser reliably emits Markdown headings.

• Add the hierarchical layer when you need to serve precise answers from large documents — this is the production pattern.

• Use semantic chunking selectively for high-value unstructured content, not as a default.

• Always attach metadata. This isn’t optional. It’s the difference between a demo and a system.

What’s next

The chunker’s output is a stream of text segments with metadata. But text isn’t what the vector index stores — it stores numbers. The next layer, the embedder, converts each chunk into a dense vector that captures its semantic meaning.

In Part 4, we’ll cover embedding models — why dimension count matters, the tradeoff between open-source and API-based models, how to handle multi-lingual content, and the latency/cost arithmetic of embedding at scale.

This is Part 3 of the System Design for RAG series. Part 2 on The Parsing Layer is live now.

If you’re building RAG in production and these tradeoffs resonate, subscribe — each article in this series goes deep on one layer of the pipeline, with code you can actually use.

It doesn’t know which checklist your reviewers run before approving a PR. It doesn’t know the seventeen edge cases your compliance team checks before a PDF goes out the door.

That’s not a reasoning gap - it’s a knowledge gap. And until recently, there were exactly two options for closing it -

Stuff it in the system prompt: Every additional instruction is permanent context. A prompt carrying 20K tokens of niche procedure pays that cost on every single request - higher latency, wasted tokens, and a well-documented failure mode where models attend less reliably to instructions buried in the middle of a long context window.

Fine-tune: Bakes the behavior into weights. Works, but it’s slow to iterate, expensive to run for narrow procedural knowledge, and the wrong tool for “tax goes on its own line item.”

This article is in collaboration with Ashutosh - He’s a software engineer at YouTube, ex-Google Search, and Microsoft Azure. Connect with him if you want to get deep dives into system design, AI, and software engineering.

Ashutosh’s Newsletter

A newsletter to help everyone understand software engineering and computer science concepts in depth.

By Ashutosh Maheshwari

Agent Skills is the alternative

Package procedural knowledge as a folder of files an agent loads conditionally, rather than baking it into the prompt or the weights.

Anthropic shipped it in October 2025 and open-sourced the format the following month under what’s now governed at agentskills.io.

As of this writing, 26+ platforms implement it - Claude, Claude Code, OpenAI Codex, Gemini CLI, GitHub Copilot, Cursor, VS Code, and others.

The file format

A skill is a directory. The only required file is SKILL.md:

my-skill/
├── SKILL.md          # required: frontmatter + instructions
├── scripts/           # optional: executable code
├── references/         # optional: docs loaded on demand
├── assets/            # optional: templates, files the agent reads or copies

SKILL.md is YAML frontmatter followed by a markdown body. The spec defines two required fields and several optional ones:

For example, Claude Code layers on custom extensions like disable-model-invocation (useful for safe deployments) or context: fork (to run a skill in an isolated subagent). A basic skill works unmodified across all platforms; platform-specific extensions only work where supported.

Minimal example:

---
name: invoice-formatter
description: Formats outgoing invoices to match company template. Use when generating or editing invoice documents.
---

# Invoice Formatter

1. Use the template in assets/invoice_template.xlsx
2. Line items go in columns B–E
3. Tax is a separate row, never folded into line items

That’s the entire artifact. No training run, no eval harness required to ship it - though you’ll want one if you’re maintaining it at scale.

Progressive disclosure: the actual mechanism

The spec defines three loading tiers, each with a rough token budget:

1. Metadata (~100 tokens/skill). At session start, the agent loads only name and description for every available skill. This is what makes the system scale — fifty skills costs a few thousand tokens, not fifty system prompts’ worth of context.

2. Instructions (<5,000 tokens recommended). When a request matches a skill’s description, the full SKILL.md body loads into context. This is the only point where the agent commits real context budget to the skill.

3. Resources (loaded as needed). Bundled scripts, reference docs, and assets load only if the instructions direct the agent to use them — often via a script execution rather than a context read, which keeps even large bundled resources off the token bill entirely.

The practical consequence: skill count scales cheaply, skill invocation doesn’t. You can register hundreds of skills without bloating every request, but each activated skill still has to fit its instructions inside a few thousand tokens, which forces concision in a way the unstructured system-prompt approach never did.

Why the description field is the real interface

Discovery relies entirely on description matching. There is no separate routing model. The description has to do two things at once: state what the skill does, and anticipate the language a user would generate when the task is relevant.

Writing this is like writing a search index entry. Too narrow, and the skill never activates. Too broad, and it misfires on irrelevant tasks.

# under-specified — won't reliably fire
description: Handles spreadsheets.

# specified for activation
description: Use when creating, editing, or analyzing Excel files (.xlsx) —
  formulas, pivot tables, financial models. Triggers on mentions of
  spreadsheets, workbooks, or .xlsx.

Agent Skills in Production

• anthropics/skills : the reference implementation.

  • Four skills (docx, pdf, pptx, xlsx) are the actual source-available code behind Claude.ai’s document creation feature, not a demo of it.

  • The rest cover creative/design (algorithmic-art, canvas-design, theme-factory), dev tooling (frontend-design, MCP scaffolding, skill-creator), and enterprise workflows (brand-guidelines, internal-comms, doc-coauthoring).

/plugin marketplace add anthropics/skills
/plugin install document-skills@anthropic-agent-skills

• obra/superpowers — skills as enforced policy, not optional context. Its own instructions state it plainly: if a skill applies, the agent has no discretion to skip it.

  • test-driven-development — enforces red-green-refactor before any implementation.

  • systematic-debugging — four-phase root-cause process instead of guess-and-check fixes.

  • verification-before-completion — blocks marking a task done without proof it’s done.

  • subagent-driven-development — splits implementation and review across separate subagents, composing with Claude Code’s existing subagent tooling rather than replacing it.

• travisvn/awesome-claude-skills — a maintained index for surveying the ecosystem.

Skills vs. MCP vs. fine-tuning - where each actually sits

These concepts get conflated constantly. Here is how we can separate them in our mental models:

• MCP (Model Context Protocol) / Function Calling = Capability. This gives the agent hands to act on the world (call APIs, query databases). Without a tool, there’s no action.

• Agent Skills = Procedure. Skills tell the agent how to use its capability. In what order, under what constraints, and with what conventions.

• Fine-Tuning = Weights. This is the model itself. It’s slow to produce and expensive to iterate. Skills, on the other hand, are plain files: diffable, reviewable in a PR, and instantly revertible.

Threat model for Agent Skills

Skills can ship arbitrary scripts. We need to treat them like production dependencies, not just saved prompts, because they carry the exact same supply-chain risks.

To secure our AI stack, we should follow these rules:

• Vet the source: Actively maintained repositories with real issue and PR traffic carry far less risk than abandoned gists.

• Enforce boundaries: allowed-tools is a pre-approval list, not a sandbox. If we want true restrictions, we must pair it with explicit platform-level deny rules.

• Audit the code: The description is just marketing copy. We must read the bundled scripts before installing anything.

• Monitor external calls: We must treat any skill that pings an external URL or API exactly like a new third-party dependency.

Evaluating agent skills

Writing a SKILL.md is the easy. Proving it works is the hard part.

Anthropic’s own skill-creator skill now ships with a built-in eval mode that handles this directly:

• Isolated, parallel test runs. It launches multiple sub-agents at once, each in its own clean context, so one test’s output can’t bleed into another’s grading.

• Specific grading, not just pass/fail. Each sub-agent runs a defined test prompt through the skill and scores the output against your stated success criteria - example “missed the hardcoded API key on line 42,” not just a red X.

• A concrete fix loop. It proposes instruction edits to close gaps, allowing developers to quickly bump pass rates from 60% to 90%+.

• Blind comparison for iteration. A separate “improve” mode runs the same prompts through two versions of a skill and provides both outputs to a comparator agent that doesn’t know which version produced which - removing the author’s own bias from “did my edit actually help.”

The structural shift

The most important takeaway isn’t the file format itself. It’s that agent capability is moving from massive, fragile system prompts toward composable, independently versioned units.

This is the exact same shift software engineering made decades ago—moving from monoliths to libraries and packages. Now, a legal team can version a contract-review skill while a platform team versions SQL conventions. Both get pulled in independently, without breaking the shared system prompt.

If you're building with agents and want more breakdowns like this, subscribe for breakdowns about the parts of the agent stack that actually matter, minus the hype.

Subscribe now

Picture this: You’re debugging a live production issue. You tell your DevOps agent, “Find out why the API is throwing 500 errors. You have exactly 5 minutes to investigate before we roll back.”

The model understands you perfectly. It parses the constraint, scans the server logs, and spots a database timeout. It connects to the database to run an EXPLAIN query, notices a missing index, and spins up a subagent to cross-reference yesterday’s schema migrations.

Twenty minutes later, your API is still down, and your agent is deeply engrossed in reading PostgreSQL documentation.

What failed? Not the LLM. It can reason about time constraints and write brilliant essays on incident management. The problem is your agent harness - the orchestration code managing the loop. The model said it would stop, but because it operates in a timeless void, it relied on the harness to check the clock. And the harness failed.

Some thoughts about time management system for LLM agents:

The Gap Between Language and Reality

When you command an agent to “stop after 5 minutes,” you are actually issuing two entirely different system requirements:

1. Semantic Understanding: The model needs to factor the time budget into its planning. It should prioritize breadth over depth, and start wrapping up as time gets short rather than initiating a deep-dive.

2. Execution Enforcement: The harness needs to terminate the execution loop when the deadline passes, regardless of what the model decides to do.

Most developers only build the first layer. They write the time constraint into the prompt and trust the model to honor it. That’s the semantic layer. It works most of the time.

But “most of the time” isn’t good enough for a production system. Models lose context. Long sessions get compacted. A model deep in a reasoning chain doesn’t check its watch - it just keeps reasoning. Semantic-only time constraints are advice. They are not guarantees.

To fix this, you need both layers: a Dual-Layer Architecture. The semantic layer makes the model time-aware. The enforcement layer makes the harness time-authoritative.

Layer 1: Giving the Model a Textual Clock

The model has no native sense of elapsed time. What it has is context - the conversation history it can see. So the harness must build the model a clock out of text, and inject it into every single turn.

It looks like this, prepended to every message the model receives:

<time-status>
Session started: 14:32:10
Elapsed: 3m 44s
Remaining: 1m 16s
</time-status>

On every turn, the model knows exactly where it stands. It doesn’t have to guess based on context length. But the real architectural leverage comes from urgency escalation. The status message changes as time runs out:

• T-minus 2m: “Begin winding down new sub-tasks.”

• T-minus 30s: “⚠️ FINAL 30 SECONDS - wrap up now.”

A model that was about to fire another deep search reads “final 30 seconds” and stops-not because the harness forced it, but because it understood the instruction and adjusted. That’s the semantic layer working correctly.

Layer 2: The Harness Owns the Deadline

The semantic layer is good. It is not sufficient. The enforcement layer exists for everything the semantic layer can’t handle: the model that forgets, the subagent that runs long, the tool call that spirals.

The enforcement layer works like this: before every single API call, the harness checks the wall clock. If the deadline has passed, the loop doesn’t proceed. Execution stops. But hard-stopping produces half-written JSON or truncated sentences. Instead, you need Graceful Degradation.

1. The Watchdog Thread: A background thread counts down independently of the main loop. If an API call hangs or a tool response never comes back, the watchdog fires anyway.

2. The Shutdown Turn: When the deadline approaches, the harness injects a graceful shutdown message first: “Time limit reached. Stop all current work. Provide a summary of what you found and what remains incomplete.”

3. The Cutoff: The model gets one final turn to produce something useful. Then, 30 seconds later, the loop exits regardless of whether that final turn completed.

The Subagent Black Hole

Here’s where naive implementations break completely. Simple agents run in a single loop. You control the loop, you check the clock, it works.

But modern agent frameworks (Claude Code, LangChain) spawn subagents - child sessions that run independently. When a parent agent spawns a subagent, it waits. From the parent’s perspective, it’s working. From the system’s perspective, the parent is blocked waiting for a child that may never return.

Practitioners call this the Subagent Black Hole. The parent’s clock runs out while it sits blocked on a child that silently failed two minutes ago.

The Fix: Budget Propagation.
When a parent spawns a subagent, it doesn’t give the subagent the full original time budget. It carves off a fraction of its remaining time. Parent has 4 minutes left? The subagent gets a hard limit of 2 minutes. This mathematically guarantees the child will resolve (or time out) before the parent deadline fires, always leaving a buffer for the parent to finish its own work.

What the Model Actually Experiences

It’s worth tracing through what a 5-minute debugging session looks like from the model’s side.

• At turn one: The model sees the task and a status showing zero elapsed time, 5 minutes remaining. It makes a tight plan: check logs, check DB status, summarize. The time constraint shapes the plan before a single tool call fires.

• At turn four: The model sees 3 minutes elapsed, 2 minutes remaining. It’s doing fine. It finds the database timeout.

• At turn six: The model sees 4 minutes and 30 seconds elapsed and a warning injected by the harness: “⚠️ FINAL 30 SECONDS - Start wrapping up and summarizing findings.” It was about to start a new search thread for Postgres schema documentation. It doesn’t. It starts pulling together the root cause analysis.

• At turn seven: With 5 seconds left, the harness injects the graceful shutdown message directly. The model produces a final summary for the team: “API failing due to table lock on users. Could not verify recent migrations in time. Roll back recommended.”

Thirty seconds later, the loop exits. The hard deadline has passed. The session is over.

Two things made this work: the model knew where it stood at every step, and the harness guaranteed the outcome regardless.

Production Realities: The 3 Ways This Still Breaks

Even with both layers in place, senior engineers should anticipate these three failure modes:

1. The API Call Eats the Deadline
You have 30 seconds left. The model takes 45 seconds to respond. The deadline fires during the call, but the call blocks until it returns. You’re already 15 seconds over when it comes back.

• Fix: Every API call gets a dynamic timeout equal to the remaining time minus a small buffer. You’ll sometimes get an incomplete response, but you won’t hang past your deadline.

2. The Ignored Shutdown
You inject the shutdown message. But the model fires one more tool call first—”just one more query.” Now it’s 40 seconds over waiting on a result it doesn’t need.

• Fix: Tool Stripping. Strip the tool definitions from the final API call payload entirely. The model cannot call a tool if the schema doesn’t exist. The graceful shutdown turn must always be forced into a pure text turn.

3. Clock Drift via Context Compaction
You’re running a long pipeline. After context compaction, the early messages -including the system prompt that defined what <time-status> tags mean - are gone. The tags stop working.

• Fix: Treat compaction as a mini-restart. Re-inject a fresh definition of the time-status contract into the compacted context. Don’t assume rule continuity across context boundaries.

The Part That Should Stick

The model understands “stop after 5 minutes” perfectly well in language. It can reason about it, plan around it, and honor it when things go smoothly.

But understanding is not enforcement.

Time in agents is a strict contract between three parties: the user who states the constraint, the model that reasons within it, and the harness that enforces it.

• Build a harness that only gives the model a clock, and you rely on the model to be disciplined every single time. Usually isn’t good enough.

• Build a harness that only enforces the deadline without telling the model, and you get hard cutoffs with no graceful summary - an agent that just abruptly dies mid-sentence.

Build both, and you get an agent that actually behaves like an engineer with a deadline.

Checkout GitHub repo for code example - https://github.com/alwaysavani/time-aware-harness

Thanks for reading this article!

The newsletter now reaches more than 100 readers🎉

Please like and restack. Subscribe to get future issues in your inbox:

Subscribe now

And yet the answers are still wrong.

It’s a frustrating place to be. You’ve done everything the blog posts say. The retrieval scores look reasonable. But somewhere between the document and the answer, something is getting lost.

That something is usually the parser. And it’s almost never what people look at first.

Where does the information actually disappear?

It’s worth tracing through a concrete example. Take a hardware spec sheet — the kind with component tables, temperature ranges, voltage specs. Put it through a naive PDF extractor. Here’s what comes out the other side.

The original table:

| Component | Max Temp | Min Temp | Voltage |
|-----------|----------|----------|---------|
| Sensor A  | 95°C     | -40°C    | 3.3V    |
| Sensor B  | 110°C    | -55°C    | 5.0V    |

What the embedder actually receives after a left-to-right OCR pass:

Component Max Temp Min Temp Voltage Sensor A 95°C -40°C 3.3V Sensor B 110°C -55°C 5.0V

Every number made it through. Every relationship between them didn’t.

So when someone asks “what’s the minimum operating temperature for Sensor B?” — the model is looking at a string where -55°C and -40°C are both sitting near Sensor B. It guesses. And guessing is not what you built a RAG pipeline for.

This isn’t really a retrieval failure or an embedding failure. The information was destroyed earlier - before any of that machinery ran. And it tends to happen in one of three recognizable patterns.

Column bleeding: Multi-column PDFs get read left-to-right across the full page width. A three-column spec sheet collapses into one long run-on string. The numbers survive. Their context doesn’t.

Header severance: A section heading sits at the bottom of page 4. Its content flows to page 5. A paginated parser splits them into separate chunks. What you end up with is a floating fragment - “Maximum allowable torque: 47 N·m under sustained load” - with no indication of what system it belongs to.

Table flattening: A table carries 2D relational structure - rows, columns, the meaning created by their intersection. Once that gets linearized into a prose string, the structure is gone. Nothing downstream can reconstruct it.

All three follow a pattern. All three happen before the first vector is computed.

Why Markdown keeps coming up as the middle layer

The choice of intermediate format - what the parser hands off before the chunker sees it - quietly shapes everything that follows.

Plain text is the obvious default, but it discards structure. HTML preserves structure but gets verbose fast. The same table in HTML runs roughly 3–5x more tokens than Markdown pipe format - and across a large knowledge base, that difference compounds on every retrieval call.

The LLMs were trained on large amounts of markdowns. ## headings and | table delimiters carry genuine semantic weight - HTML tags largely got stripped from training corpora. A Markdown table passed to a model is legible to it in a way that’s difficult to replicate with other formats.

For exmaple: Heading-based chunking - splitting at ## and ### boundaries - is one of the more reliable chunking strategies available. It only works if the parser actually emits headings. Plain text gives you nothing to split on.

It’s just that the format choice quietly constrains what’s possible later, and Markdown tends to cause the fewest downstream surprises.

Three approaches, and when each one fits

Different document types behave very differently under parsing. What works cleanly on a Word export will silently mangle a scanned freight invoice. Most pipelines apply one approach uniformly because it’s operationally simpler - and it is, right up until it isn’t.

Lightweight heuristic converters

Tools in this space: MarkItDown, PyMuPDF

These wrap existing document parsers: pdfminer, python-docx - and apply rule-based logic to detect document elements and translate them into Markdown. No ML inference, no GPU, pure heuristics.

For digitally-authored documents - anything written in Google Docs, Notion, Confluence, or Word and exported cleanly - these are fast, cheap, and accurate. Sub-100ms per page on typical hardware, easy to run synchronously.

The limitation is equally clear: scanned documents, photographed manuals, legacy exports with unusual layouts - the heuristics don’t hold.

The parser processes the scanned manual, returns output, and the pipeline continues normally: the multi-column spec table comes back as a linearized string. The embeddings get computed.

Everything looks healthy.

The problem only surfaces when someone asks a question that needs a number from that table.

Vision-language model parsers

Tools in this space: Docling, Marker, LlamaParse, Azure Document Intelligence

These take a different approach - treating documents as visual objects rather than text streams. Pages get rasterized first. Then a VLM identifies bounding boxes around each element and classifies them (title, paragraph, table, figure, footnote) before any text extraction happens. Reading order is reconstructed from spatial position on the page, not from the byte stream.

The tradeoff : GPU required to be practical at scale, 2–10 seconds of latency per page depending on complexity. But for documents that originated as physical artifacts - scanned manuals, freight invoices, legacy filings, multi-column academic papers - the quality difference over heuristic parsers isn’t marginal. It’s categorical.

Coming back to that hardware spec table - here’s what a vision parser produces with the same document:

## Thermal Limitations

| Component | Max Temp | Min Temp | Voltage |
|-----------|----------|----------|---------|
| Sensor A  | 95°C     | -40°C    | 3.3V    |
| Sensor B  | 110°C    | -55°C    | 5.0V    |

The table structure is intact. The parent heading Thermal Limitations is attached. The 2D relationships survived. Same document, different parser, completely different retrieval behavior.

DOM-aware web parsers

Tools in this space: Firecrawl, BeautifulSoup with custom traversal, Playwright + extraction

The instinct when parsing a wiki or docs site is usually to strip HTML tags and keep the text. HTML is noisy, but stripping tags also discards the semantic tree, which is where a lot of the structure actually lives.

The DOM already knows what everything is. <h1> signals the most important heading. <li> signals a list item. <table> signals relational structure. A DOM-aware parser traverses that tree and translates the hierarchy into Markdown rather than throwing it away.

The catch: it depends heavily on source HTML quality. Well-structured wikis with semantic markup parse cleanly. CMS-generated pages with deeply nested <div> containers and JavaScript-rendered content are harder. It’s also fragile to redesigns — a navigation restructure can start pulling nav links into every chunk.

Where it tends to shine is internal wikis and documentation sites with clean markup. The hierarchy matters: a subsection about rate limits stays connected to its parent heading “Authentication and Access Controls,” so a retrieved chunk still carries the context that those limits apply to authenticated endpoints specifically.

What’s worth sending alongside the Markdown

Structured Markdown gets you most of the way there. But there’s one more thing worth thinking about is the metadata that travels with it.

A chunk in isolation is missing context the LLM would benefit from having. Wrapping the Markdown in a payload that carries document provenance gives the rest of the pipeline more to work with:

{
  "document_id": "doc_99482",
  "source_system": "confluence_wiki",
  "element_type": "table",
  "page_number": 12,
  "bounding_box": [120, 450, 600, 800],
  "content_markdown": "| Component | Max Temp |\n|---|---|\n| Sensor A | 95°C |\n| Sensor B | 110°C |",
  "parent_header": "Thermal Limitations"
}

Two fields in particular tend to earn their weight. parent_header gives a retrieved table chunk the scoping context it would otherwise be missing - the model knows this table is from Thermal Limitations, not electrical specs or mechanical specs. Without it, documents with multiple similar-looking tables start producing ambiguous answers.

bounding_box is the audit trail. When someone wants to know where an answer came from, you can point to the exact region of the original document rather than just the filename. In regulated industries this tends to go from nice-to-have to required fairly quickly.

A rough heuristic for choosing

Mixed corpus - which is most real-world situations - just routes by document type at ingestion. It’s a conditional, not a redesign.

Often just one parser is uniformly applied for operational simplicity. It works fine - until the heuristic parser quietly mangles every scanned document in the index and three days of debugging trace back to something that happened at text extraction.

A step that’s easy to skip in the debugging loop

The usual path when RAG answers go wrong: inspect retrieved chunks → look at similarity scores → tweak embeddings → add a reranker → try again.

This loop is reasonable except it starts in the wrong place if the parsing layer is where the problem actually lives.

Something worth trying at this stage: pull 10–20 chunks corresponding to the queries that went wrong and just read the Markdown. Not the source documents - the parsed output.

If the tables are linearized, headers are floating, or columns have bled together, that’s the diagnosis. A reranker can’t reconstruct a 2D table that was destroyed before the embedder ever saw it. The fix has to happen earlier.

It’s a small thing to check and it explains a surprising number of retrieval failures that look, on the surface, like embedding problems.

Further reading

• Dropbox: Using machine learning to index text from billions of images

• Netflix: Synchronizing the Senses: Powering Multimodal Intelligence for Video Search

• Pinterest: Unified Context-Intent Embeddings for Scalable Text-to-SQL

Thanks for reading!
Please like, restack and share your thoughts about parsers for RAG pipelines.

Do subscribe - if you would like to receive future issues -

Subscribe now

Quantifying Data Drift

Data drift is the time delta between a data modification in a source system (e.g., Confluence, Jira, PostgreSQL) and its availability in the vector database.

Data Drift Latency = T(Vector Available) - T(Source Mutation)

Minimizing this latency requires moving away from heavy, periodic batch syncs toward real-time stream processing architectures.

Let’s break down the three primary patterns:

Pattern 1: Scheduled Batch Ingestion (Pull-Based)

Batch ingestion relies on scheduled ETL pipelines to take periodic snapshots of the data source.

How it Works

1. A workflow orchestrator (e.g., Apache Airflow) triggers an Apache Spark or Databricks job on a fixed schedule (e.g., every 6 or 24 hours).

2. The job runs a full or incremental scan on the data lake (e.g., AWS S3, Snowflake).

3. The cluster processes text segments, generates embeddings in parallel batches, and writes them to the vector index.

Engineering Tradeoffs

• Pros: Highly compute-efficient for bulk historical data processing. Minimizes continuous network connections.

• Cons: High data drift latency. If a document updates immediately after a batch finishes, that update remains invisible to the RAG system until the next cycle.

• Real-World Usage: Teams use heavy batch processing pipelines for non-time-critical data, such as indexing large historical catalogs or archiving years of system logs.

Pattern 2: Event-Driven Webhooks (Push-Based)

Event-driven ingestion shifts the responsibility of data movement from the target pipeline to the source application via HTTP push notifications or message queues.

How it Works

1. A user modifies a document in a SaaS application (e.g., Notion, Slack, Jira).

2. The application triggers a webhook event containing the document ID and mutation details.

3. An API gateway captures the event and places it into a distributed task queue (e.g., Celery, AWS SQS).

4. Isolated worker processes fetch the specific updated document, chunk it, re-embed it, and upsert the vector index.

Engineering Tradeoffs

• Pros: Low latency. Updates reflect in near real-time. Eliminates unnecessary compute overhead from scanning unchanged documents.

• Cons: Prone to API rate limiting (HTTP 429) when fetching large document bodies from SaaS providers during peak traffic. Requires robust retry mechanisms with exponential backoff.

• Real-World Usage: Highly responsive event-driven mechanisms are ideal for third-party SaaS tools, ensuring internal bots reflect system status changes instantly without polling backend systems.

Pattern 3: Change Data Capture (CDC) with Stream Processing

For internal application databases where application-level webhooks are unfeasible, Change Data Capture (CDC) intercepts data mutations directly at the database engine layer.

How it Works

1. An application modifies a row in a database.

2. The database engine records the transaction in its Write-Ahead Log (WAL) or binary log.

3. A CDC connector (e.g., Debezium) tails the log, extracts the row-level diff, and publishes it as an event to a log broker like Apache Kafka.

4. A stream processing engine (e.g., Apache Flink) consumes the stream, flattens the schema, chunks the data in-flight, computes embeddings, and pushes to the vector layer.

Engineering Tradeoffs

• Pros: Near-zero data drift latency (milliseconds). Zero performance impact on production application queries since it reads directly from transaction logs.

• Cons: High infrastructure complexity. Requires managing distributed log streaming platforms and stateful stream processors.

• Real-World Usage: This is the standard for core internal application databases. It guarantees the vector index is always perfectly synchronized with the system of record.

Summary

Further Reading

To see how these ingestion patterns operate at hyperscale, read these architectural teardowns from the teams that built them:

• Uber Moves In-House Search Indexing to Pull-Based Ingestion

• Keystone Real-time Stream Processing Platform

• Gobblin’ Big Data With Ease

Thanks for reading. In the next issue, we break down Stage 2 of the RAG pipeline -Parsers.

Please like and consider subscribing :

Subscribe now

Then real users run multi-step tasks.

The agent forgets context from five minutes ago. It repeats work. It makes the same mistake twice.

The problem is not the model. It is memory.

LLMs are stateless. Every API call starts fresh. You send the full conversation. The model reads it. Then it forgets. This works for chatbots. It fails for agents. Agents run multi-step tasks. They hand off work. They need continuity.

Continuity requires a memory system.

The 4 Layers of Agent Memory

System design is about trade-offs. Each memory layer trades speed for capacity.

1. In-Context Memory (Working Memory)

This is short-term memory. It lives inside the current prompt window. It holds the system prompt, recent history, and tool outputs.

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user",   "content": user_request},
    {"role": "assistant", "content": last_response},
    {"role": "tool",   "content": tool_output}
]
response = llm.chat(messages)

Pros: Fast. Zero retrieval error. Simple.

Cons: Small capacity. High inference cost. Ephemeral.

When to use: Current task state. Keep working context under 20K tokens. Summarize old turns. Do not bloat the prompt.

2. Semantic Memory (Vector Store)

This is the long-term knowledge base. You embed facts as vectors. You store them in a database. You retrieve relevant chunks at query time.

# Store
embedding = embed_model.encode("Refund policy is 30 days.")
vector_store.upsert(id="doc-42", vector=embedding, metadata={"type": "policy"})

# Retrieve
query_vec = embed_model.encode(user_query)
results = vector_store.query(query_vec, top_k=5, filter={"type": "policy"})

Pros: Massive scale. Persistent. High signal-to-noise ratio.

Cons: Retrieval errors. Added latency (50–200ms). Requires chunking strategy.

When to use: Domain knowledge and user preferences.

Pro-tip: Do not use pure vector search. Add a metadata filter. Use hybrid search (Vector + BM25).

3. Episodic Memory (Task History)

This stores actions, not facts. It holds past conversations, tool calls, and outcomes. The agent learns from experience.

summary = llm.summarize(completed_task)
episodic_store.save({
    "user_id": user.id,
    "task_type": "analysis",
    "summary": summary,
    "outcome": "success",
    "timestamp": now()
})

Pros: Prevents repeated mistakes. Enables personalization.

Cons: Summaries lose detail. Hard to retrieve the exact right episode. Fast storage growth.

When to use: Recurring tasks for the same user. Customer support histories. Codebase debugging logs.

4. Procedural Memory (Instructions)

This is the know-how. It contains system prompts, tool schemas, and workflows. It rarely changes at runtime.

system_prompt = "You are a data analyst. Verify SQL syntax. Never expose PII."
tools = [{"name": "run_sql", "parameters": {...}}]
response = llm.chat(messages, system=system_prompt, tools=tools)

Pros: Zero latency. Highly reliable. Easy to version control.

Cons: Inflexible mid-run. Consumes context window.

When to use: Always. Keep system prompts under 500 tokens. Move rare instructions to semantic retrieval.

State Management Patterns

Memory stores data. State tracks current execution. These are distinct problems. Use one of three patterns.

Pattern 1: Stateless Agents

No persistence between calls. The agent gets a task, executes, and returns.

User Request --> Agent --> Response

• Use case: Simple classification. Question answering. Horizontal scaling.

• Trade-off: Easy to build. Zero continuity.

Pattern 2: Stateful Session Agents

The agent maintains state in RAM during a session. It persists a summary to episodic memory when finished.

Start --> Load Context --> Agent Loop --> Persist Summary --> End

• Use case: Multi-turn research. Coding. Chat support.

• Trade-off: The production sweet spot. Moderate complexity. High utility.

Pattern 3: Long-Horizon Agents

Tasks span days. State lives in a durable database. The agent can pause, resume, or crash without losing progress.

Task DB --> Wake & Load --> Run Checkpoint --> Sleep --> Resume

• Use case: Background jobs. Multi-agent swarms. Fragile, long-running workflows.

• Trade-off: High complexity. Requires orchestration tools like Temporal or LangGraph.

The Trade-Off Matrix

Combine memory types. No single layer does it all.

Production Anti-Patterns

Senior engineers repeatedly make these five mistakes.

• Context dumping: More context is not better. Models ignore the middle of long prompts. Compress and summarize.

• Naive top-K retrieval: Top-5 vector search fails in production. Add metadata filters. Use BM25. Re-rank your chunks.

• Ignoring freshness: Six-month-old preferences mislead agents. Timestamp everything. Build decay policies.

• Brittle retrieval: Vector databases go down. Queries fail. Design graceful degradation. Tell the agent when context is empty.

• Monolithic memory: Vector stores are not session buffers. Use the right layer for the right job.

Reference Architecture

Key Takeaways

• LLMs are stateless. Memory creates continuity.

• Keep working context small. It saves money and maintains model focus.

• Semantic memory needs hybrid search. Pure vectors fail in production.

• Episodic memory requires strict metadata and timestamps.

• State is active execution. Memory is passive recall. Design for both.

Why a Filter Matters

AI features are not like regular software features. The failure modes are different:

• Quality is probabilistic, not deterministic.

• Maintenance involves monitoring, retraining, and eval management - not just bug fixes.

The cost of a bad AI feature is not just wasted engineering time. It is user trust eroded by a system that confidently produces wrong answers.

The Six Questions

1. Is this a task AI is actually good at? AI is strong at summarisation, classification, retrieval, and generation. It is weak at precise numerical reasoning, tasks requiring real-time information without retrieval, and tasks with strict correctness requirements. Identify which category this request falls into before scoping.

2. What does failure look like? A wrong recommendation in a movie app is mildly annoying. A wrong answer in a medical triage tool is dangerous. Define the failure mode before you design the system - it determines how much safety investment is required.

3. What data do we have? Is there labelled training data? Is the knowledge base current? Is the data representative of all users? A feature that requires data you do not have is a data project before it is an AI project.

4. Who evaluates quality and how? If the answer is ‘the product manager eyeballs some outputs before launch,’ the feature is not ready to be built. Evals must be defined before development starts.

5. What is the latency and cost budget? LLM calls are 10–100x more expensive than database queries. They are also 100–1,000x slower. If the feature requires sub-100ms response time, verify the architecture can achieve it before committing.

6. Who maintains this after launch? AI features require ongoing maintenance: drift monitoring, eval updates, occasional retraining, prompt tuning. If no one owns it after launch, it will silently degrade.

The Two-Day Hackathon

Before committing to a multi-month AI project, run a two-day feasibility hackathon. Build the simplest possible version. Evaluate it against 50 real examples. If it cannot pass that bar, the feature is not ready - regardless of how compelling the pitch was.

A two-day spike that kills a bad idea is one of the highest-ROI investments an engineering team can make.

Takeaway

Treat AI pitches with extreme skepticism. Filter them quickly using a 6-question stress test (covering data, cost, failure modes, and maintenance) and a 2-day prototype sprint. Saying "no" to bad AI is just as important as building good AI.

Like and subscribe if you liked this article and share what questions do you ask to check whether AI should be used to solve a problem or not?

Leave a comment

The Core Problem

When a company builds an LLM API platform, it faces a challenge that traditional SaaS platforms have never dealt with at this scale: GPU time is expensive, inference is stateful, and enterprise customers have wildly different requirements around privacy, performance, and compliance.

How do you serve a startup on a $99/month plan and a bank on a $500,000/year dedicated contract — on the same infrastructure — without one affecting the other?

The answer is a layered isolation architecture. Let’s break it down.

The Three-Tier Isolation Model

Not all tenants need the same level of isolation. Building one mode for everyone either wastes resources or fails compliance requirements. The industry solution is three tiers.

Tier 1 : Shared cluster (logical isolation)

All tenants share the same GPU cluster and the same model weights. Tenant identity is tracked at the session level — a tenant ID injected by the gateway, RBAC claims resolved at auth time, audit logs tagged per tenant.

This works for most customers. There is no hardware separation, but there is no cross-tenant data leakage either. The model processes sessions sequentially and the gateway ensures no tenant can access another’s conversation history.

Who uses it: Startups, developers, low-sensitivity workloads.

Tier 2 : Dedicated instance (process-level isolation)

The tenant gets their own vLLM process, potentially with a custom fine-tuned model loaded. They still share the physical GPU cluster but have dedicated KV cache memory. Logs are fully isolated and never co-mingled with other tenants.

This tier suits enterprises that need contractual log isolation, custom model weights, or a guaranteed throughput floor.

Who uses it: Mid-market enterprises, companies with internal compliance requirements.

Tier 3 : Dedicated cluster (physical isolation)

Full VPC isolation. Dedicated physical hardware. Completely airgapped from shared infrastructure. Every prompt, completion, and log stays within the customer’s dedicated environment.

This is the only tier acceptable for HIPAA, FedRAMP, and most government customers whose legal teams will not accept logical separation as a substitute for physical separation.

Who uses it: Healthcare, government, finance, heavily regulated industries.

The Key Insight Most People Miss

Most engineers think of isolation as a security property — prevent Tenant A from reading Tenant B’s data.

That is necessary, but not sufficient.

Isolation is also a commercial property.

A customer paying for a dedicated tier is not just buying privacy. They are buying a performance guarantee : a throughput floor that cannot be eroded when another tenant spikes traffic at 3am.

The architecture must enforce this at the resource allocation level, not just the logical routing level. Separate GPU memory. Separate queues. Separate circuit breakers. Logical separation without resource separation is not truly dedicated.

The Gateway Layer

Every request from every tenant - regardless of tier - passes through a shared API gateway. This is where the critical cross-cutting concerns live.

The gateway handles:

• Authentication: JWT validated against your IdP (Okta, Entra). Claims resolved once per request.

• Quota enforcement: Token-per-minute (TPM) and request-per-minute (RPM) tracked per tenant in a Redis sliding window. Soft limits trigger a warning. Hard limits return 429 Too Many Requests.

• Admission control: When the shared cluster is near capacity, Tier 1 traffic is shed first - queued or rate-limited. Tier 2 and Tier 3 customers are never touched by this mechanism.

• Cost tagging: Every request is tagged with tenant ID, model ID, and timestamp before it hits the inference layer.

Cost Attribution: Metering at the Token Level

LLM usage is metered in tokens - not requests, not seconds, not API calls. Input tokens and output tokens are priced differently (output is more expensive to generate), so they must be tracked separately.

The metering pipeline:

Each customer sees their own usage dashboard: daily token spend, model breakdown, cost by API key or feature. Budget alerts fire at 80% consumption so customers are never surprised by an overage.

One nuance worth calling out: cached tokens should be tracked separately. If the KV cache reuses a prompt prefix the customer has sent before, the effective cost is lower. Customers who build systems with consistent prompt prefixes benefit from this — it should be reflected in their bill.

Data Residency

Enterprise customers - especially in Europe - have contractual and legal requirements about where their data lives. A prompt sent by an EU customer must be processed in the EU and never leave EU infrastructure.

Region-aware routing is the foundation. Each tenant’s home region is stored in the gateway config. Every request is routed based on that config - not on where the request originated, but on where the tenant’s data is allowed to go.

This is enforced at the network level (subnet routing rules) not just the application level. A misconfigured application cannot accidentally forward an EU request to a US server if the network layer blocks it.

No-log mode for the strictest tenants. When enabled:

• The gateway tags the request with a do-not-persist flag

• The inference server processes the prompt entirely in memory

• No prompt or completion is written to any storage layer

• Only metadata (token counts, latency, model version) is written to the billing database

• The billing database itself is also region-scoped for these tenants

Encryption uses tenant-managed keys. Each tenant’s data is encrypted under a key stored in a tenant-controlled KMS. Key rotation happens on a tenant-defined schedule. Even a storage layer breach does not expose readable data without the tenant’s key.

The Serving Layer: How Inference Actually Works

For the shared tier, vLLM with continuous batching is the production standard. Continuous batching processes multiple requests concurrently rather than waiting for one to finish before starting another. It is why shared GPU clusters can sustain 60–70% utilisation without constant memory pressure.

PagedAttention manages KV cache memory. The KV cache is the main memory bottleneck in LLM inference - it stores the attention state for every token in every active request. PagedAttention allocates cache memory in non-contiguous blocks (like virtual memory), preventing the fragmentation that causes out-of-memory crashes under bursty load.

For tenants with custom fine-tuned models, LoRA adapters are the right architecture:

The base model weights are loaded once. Only the small adapter weights are tenant-specific. A 7B-parameter base model with a rank-16 LoRA adapter adds fewer than 20 million parameters — orders of magnitude less GPU memory than running separate model copies per tenant.

Noisy Neighbour: The Hardest Operational Problem

A tenant on the shared tier with a traffic spike should not degrade performance for a tenant on a dedicated tier. Three mechanisms enforce this.

Separate resource pools. Tier 2 and Tier 3 tenants have reserved GPU memory and compute. The shared cluster’s queue is physically separate from the dedicated cluster’s queue. A Tier 1 traffic surge cannot consume Tier 2/3 resources.

Priority queuing at the gateway. Requests are enqueued with priority based on tier. Under load, Tier 1 requests are the first to be delayed or shed.

Circuit breakers per session. If a single session is generating unusually high load (a runaway agent loop, for example), a circuit breaker trips for that session. The tenant gets a structured error. The rest of the platform is unaffected.

Observability: What Each Tenant Sees vs. What the Platform Sees

Tenant view:

• Their own latency percentiles (p50, p95, p99)

• Token throughput over time

• Quota consumption vs. limit

• Error rate and error breakdown

• Cost by day, model, and API key

Platform view:

• Overall GPU utilisation across all clusters

• Tier 1 shed rate (how often shared-tier traffic is being dropped or delayed)

• Per-region capacity headroom

• Cross-tenant anomalies (a tenant spiking unusually)

• Queue depth per tier

The separation matters. A Tier 1 customer should never be able to infer anything about a Tier 3 customer’s usage from their own dashboard. Platform-level metrics are only visible to operators.

Summary

Concern Solution Tenant isolation Three-tier model: shared → dedicated instance → dedicated cluster Performance guarantees Resource-level separation, not just logical routing Cost attribution Token-level metering through Kafka pipeline Quota enforcement Redis sliding window per tenant at gateway Data residency Region-aware routing + no-log mode + tenant-managed encryption keys Custom models LoRA adapters over shared base weights Noisy neighbours Separate resource pools + priority queuing + circuit breakers Observability Tenant-scoped dashboards + operator-level platform view

The Design Principle That Ties It Together

Every decision in this architecture flows from one principle: different customers have bought different things.

A startup on the shared tier bought cheap, best-effort access. An enterprise on the dedicated tier bought a performance contract. A regulated institution on the isolated cluster bought a compliance guarantee.

The platform’s job is to honour all three contracts simultaneously — without any customer’s experience affecting another’s. That is the multi-tenancy problem in AI infrastructure.

Please like and restack if you found this article informative!!

Subscribe now

Go from a single-user setup to a cloud-hosted, multi-tenant system and things break fast. Statelessness, authentication, and agent retry loops will find every crack in your design.

Here are some common techniques to run MCP reliably at scale.

The Most Common Mistake: Auth Baked Into Individual Servers

Say you have three MCP servers - one for your issue tracker, one for Prometheus metrics, one for your user directory. If each one handles its own auth, you get three different OAuth flows, three security models, and three things to maintain.

The fix is the API Gateway pattern - the same one Netflix and Uber use for microservice routing.

In production, MCP servers live on a private subnet. They expose tools. They do not handle user identity. A single Gateway sits between your AI clients and your MCP backends.

Here is how a request flows:

1. The AI client connects to the Gateway via HTTP/SSE with a JWT in the Authorization header.

2. The Gateway checks the token against your IdP - Okta, Entra, whatever you use - and reads the user’s permissions.

3. The Gateway handles rate limiting and audit logging.

4. The Gateway sends a clean JSON-RPC payload to the right internal MCP server.

One place for auth. Simple servers behind it.

Insight 1: Filter Tools Based on Who Is Asking

Here is where most teams get tripped up:

Securing a REST API is straightforward. A client hits an endpoint. The server checks the role. If the user is not allowed, it returns 403. The client never sees what it cannot access.

MCP works differently. Before anything else, the server sends the AI host a full list of every tool it offers. The host puts that list into the LLM’s context window.

So if your server lists refund_stripe_charge or trigger_kubernetes_rollback for everyone - and then relies on a backend 403 to stop unauthorised users - you have already told the LLM those tools exist.

That is a problem. LLMs hallucinate. Knowing a tool is there may be enough to make an agent try to use it. You get wasted tokens and noisy errors at best. A bad action at worst.

The fix: filter tools before the LLM ever sees them.

Do it at the session handshake, before discovery runs.

1. Session starts. The Gateway reads the JWT and fetches the user’s permissions.

2. The Gateway pulls the full tool list from the backend servers.

3. The Gateway trims it. A guest gets query_public_docs. A support agent gets read_customer_telemetry. Only a senior on-call engineer sees restart_service.

4. The trimmed list goes to the AI host.

The agent cannot try to use a tool that was never in its context. You are not hoping the backend says no. You are making sure the agent never asks.

Insight 2: Agents Retry Hard. Design for It.

Human users pause, think, and move on. Agents do not.

If an LLM sends a malformed query and gets an error, it retries - immediately, with a small tweak. If it gets stuck, it can fire 50 requests in three seconds. All of them mutating state.

An MCP server with no safeguards is a self-inflicted DDoS.

Three patterns fix this.

1. Idempotency Keys on Every Mutation

Agents time out. They drop connections. They retry.

Borrow Stripe’s approach: require a unique Idempotency-Key header on any tool call that changes state. If the agent retries scale_dynamodb_capacity after a network blip, the backend sees the same key and returns the cached result. It does not run the operation twice.

No idempotency keys means no safety net when agents retry.

2. Rate Limiting Per Session, Not Per IP

IP-based rate limiting will not help you here. Every agent request comes from the same cloud infrastructure. You cannot tell them apart by IP.

Limit at the Gateway instead. Use a token-bucket or sliding-window counter, scoped per user session and per tool. One agent cannot flood your backend.

3. Circuit Breakers and the 428 Pattern

If an agent fails to use a tool correctly three times in a row, stop it. Trip a circuit breaker at the Gateway. Return a hard error that tells the LLM to stop and ask a human for help.

For high-stakes actions - dropping a table, issuing a refund, triggering a rollback - use the HTTP 428 Precondition Required response. The server sends back a payload that says: a human needs to approve this. The host UI pauses the agent and shows an Approve / Deny button. Nothing happens until a person clicks.

This is the step most MCP tutorials skip. It is also the step that keeps your production systems safe.

What to Build First

The SDKs will not protect you from distributed systems problems. Get these three things right before you ship MCP to production:

• One Gateway for auth. Keep individual MCP servers simple and internal.

• Filter tools by identity. Never let an LLM see a tool it cannot use.

• Add idempotency and circuit breakers. Agents retry hard. Your infrastructure needs to hold.

Build the control plane first. Everything else scales safely behind it.

Found this useful? Please like, restack and follow!!