• The last of the four agent classes — the formatter that shapes what actually reaches a user

• A hard guarantee: a rejected answer’s raw text never appears in the delivered response

• Distinct, honest messaging for all four outcomes a query can reach

Why This Matters

Day 8’s critic can reject an answer, but rejecting something internally doesn’t automatically stop it from being shown — that only happens if whatever renders the final response actually respects the verdict. A pipeline can do everything right up through the critic and still leak a bad answer if the formatting layer doesn’t check the verdict before displaying anything. Today’s component is where that check either happens or doesn’t.

Core Concept

ResponseFormatter branches on the critic’s verdict label, not on the answer’s content. A rejected verdict never reaches the branch that formats answer.text at all — it’s routed to a fixed, generic safe message before the raw text is ever touched. The failure reason (which check failed) is preserved in an internal_debug field for troubleshooting, but that field lives only in the structured response, never in display_text.

This is verified directly, not just asserted: running the corrupted answer from Day 8 through today’s formatter and checking the output confirms "90 days" — the fabricated detail — appears nowhere in either display_text or the full channel_json response. The only thing that survives into the delivered response is grounded_in_source, the name of the check that failed, useful for debugging and unreadable as an answer.

Four distinct outcomes exist, and each gets its own honest message: a clean approved answer is delivered with its caveat appended if one exists; approved_fallback (retrieval found nothing) is delivered as the honest “no information found” message it already was; rejected gets the safe generic message with debug info hidden from the user; and out_of_scope gets a message that tells the user what the system can help with, not just what it can’t.

Architecture

ResponseFormatter is the final stop before delivery — everything upstream (Days 6 through 8) can be wrong in some way, and this component is where those failures either get correctly contained or don’t. It returns a FormattedResponse with both a display_text (what a chat interface would show) and a channel_json (what an API caller would receive), so the same verdict can serve multiple delivery surfaces consistently.

Implementation

def format(self, answer: DraftAnswer, verdict: CriticVerdict) -> FormattedResponse:
    if verdict.verdict_label == "rejected":
        failed_checks = [c.name for c in verdict.checks if not c.passed]
        return FormattedResponse(
            status="rejected",
            display_text=SAFE_REJECTION_MESSAGE,  # answer.text is never referenced below this line
            channel_json={"status": "rejected", "text": SAFE_REJECTION_MESSAGE,
                          "sources": [], "internal_debug": {"failed_checks": failed_checks}},
        )
    ...

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

start.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html. Each card shows the internal raw draft next to what was actually delivered — for the corrupted answer, these are visibly, confirmedly different. Tests: 6 passed, including two that check the corrupted text directly against both display_text and the full JSON response.

./stop.sh

Implementation Guide

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_09_package

Package contents

lesson_09_package/
├── lesson_code.py    # ResponseFormatter, FormattedResponse, generate_dashboard (+ Days 6-8's agents, Days 3-5's retrieval stack)
├── test_lesson.py    # 6 tests verifying all four outcome paths
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_09_package
chmod +x start.sh stop.sh
./start.sh

You should see dashboard.html generated, and 6 tests passed. Open the dashboard — each card shows a “RAW DRAFT” box (internal-only) next to a “DELIVERED TO USER” box; for the corrupted-answer card, these are visibly different.

Cleanup: ./stop.sh — also removes the generated dashboard.html.

Architecture, in more detail

ResponseFormatter has two entry points: format_out_of_scope() for queries Day 6 already routed away, and format() for everything that reached the critic. format() branches on verdict.verdict_label first, before touching answer.text at all in the rejected case — this ordering is deliberate. The rejected branch constructs its entire FormattedResponse from SAFE_REJECTION_MESSAGE and a list of failed check names (not their reasons, and never the answer text), so there’s no code path inside that branch where the raw answer could leak in.

FormattedResponse carries both display_text (a single string, what a chat UI would render) and channel_json (a structured dict, what an API response would return) so the same verdict produces consistent messaging across delivery surfaces — a chat interface and an API caller should never disagree about whether an answer was safe to show.

Why this guarantee is tested directly, not just implied by the code structure

Code that “looks like” it can’t leak the raw answer can still leak it after a well-intentioned edit — someone adds a debug line, a logging statement, or a “just show the original text so support can see it” shortcut, and the guarantee quietly breaks. test_rejected_answer_raw_text_never_appears_in_display_text and test_rejected_answer_raw_text_never_appears_in_channel_json check the actual string content of the output, not the code path that produced it — they’d fail immediately if a future change reintroduced the leak, regardless of how that change was written.

Reading the four outcomes

Test coverage

Working Demo Link:

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_09_package.

• Wondering why the dashboard itself shows the raw draft — the dashboard is an internal debugging tool built for this lesson, not the actual delivery surface; showing both views side by side is what makes the guarantee visible and checkable. The guarantee applies to display_text and channel_json — what a real user or API caller would receive — not to this lesson’s own diagnostic output.

• dashboard.html doesn’t open automatically — start.sh‘s auto-open is best-effort and silently skipped in headless environments; open the file manually.

Real-World Connection

This pattern — separating “what a component produced internally” from “what crosses the trust boundary to a user” — is standard practice anywhere a system needs to fail safely rather than just fail. It’s also exactly the kind of guarantee that’s easy to claim and easy to accidentally break with an innocent refactor; having a test that checks the corrupted text isn’t present in the output, rather than trusting the code path looks right, is what keeps the guarantee real over time.

Next Steps

Days 10 through 12 wire all four agents into one working Pipeline, run it end to end, and establish the baseline measurement every later phase — starting with evaluation engineering — will measure improvement against.

• A grounding check that verifies an answer’s text actually appears in the source it cites — not just trusts that it does

• A caveat-surfaced check catching cases where an entity mismatch was flagged but the warning never made it into the final draft

• A deliberately broken answer, injected on purpose, to prove the critic actually rejects something

Why This Matters

Day 7’s synthesis promises extractive answers: text taken directly from a source, never invented. A promise enforced by one component’s code isn’t the same as a promise verified by another. If a future change to synthesis introduces a bug — a truncation error, an off-by-one in string slicing, a well-intentioned “let me just clean this up” edit — nothing catches it unless something downstream actually checks. That’s what a critic is for: not trusting that upstream code does what it says, confirming it.

Core Concept

CriticAgent.review() runs three checks against every draft answer: the text isn’t empty, the text is verifiably grounded in whatever passage it cites, and if synthesis flagged an entity mismatch, the caveat explaining it actually survived into the final draft. All three must pass for approval.

The grounding check is the one worth sitting with: it doesn’t ask “did synthesis say this was extractive” — it directly checks whether answer.text appears verbatim inside the cited passage’s text. To prove this isn’t a rubber stamp, today’s code deliberately constructs a corrupted answer — text edited to say “90 days” and “guaranteed” where the real source says “30 days” and “no questions asked” — and runs it through the same critic. Verified by actually running it: the critic catches it, flags grounded_in_source as failed, and rejects the answer outright.

There’s also an honest middle case: when retrieval finds nothing relevant, synthesis returns a fallback message with no citations at all. That’s not a failure to catch — it’s the system correctly admitting it doesn’t know. The critic labels this approved_fallback, distinct from both a normal approval and a rejection, so the dashboard can show the difference between “this is a good answer” and “this is an honest non-answer.”

Architecture

CriticAgent sits after SynthesisAgent and before formatting, taking a DraftAnswer and a lookup of passage ID to full text. It returns a CriticVerdict: an overall approval boolean, a list of individual CheckResult objects (one per check, each with its own pass/fail and reason), and a verdict label distinguishing normal approval, honest fallback, and outright rejection.

Implementation

def review(self, answer: DraftAnswer, passage_lookup: dict) -> CriticVerdict:
    checks = []
    checks.append(CheckResult("non_empty", bool(answer.text.strip())))

    is_fallback = len(answer.source_passage_ids) == 0
    grounded = True
    if not is_fallback:
        for passage_id in answer.source_passage_ids:
            if answer.text not in passage_lookup.get(passage_id, ""):
                grounded = False
    checks.append(CheckResult("grounded_in_source", grounded))

    caveat_ok = answer.entity_match or bool(answer.caveat.strip())
    checks.append(CheckResult("caveat_surfaced", caveat_ok))

    approved = all(c.passed for c in checks)
    ...

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

start.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html. Four cards render: two clean approvals, one out-of-scope query skipped entirely, and one deliberately corrupted answer showing a red REJECTED badge with the specific failed check visible. Tests: 6 passed.

./stop.sh

Implementation Guide

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_08_package

Package contents

lesson_08_package/
├── lesson_code.py    # CriticAgent, CheckResult, CriticVerdict, generate_dashboard (+ Days 6-7's agents, Days 3-5's retrieval stack)
├── test_lesson.py    # 6 tests verifying each check independently
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_08_package
chmod +x start.sh stop.sh
./start.sh

You should see dashboard.html generated, and 6 tests passed. Open the dashboard — two green “approved” cards, one skipped out-of-scope card, and one red “REJECTED” card showing exactly which check failed.

Cleanup: ./stop.sh — also removes the generated dashboard.html.

Architecture, in more detail

CriticAgent.review() runs three independent checks and combines them with all() — every check must pass for approved to be True. Each check is recorded as its own CheckResult, not collapsed into a single boolean, specifically so the dashboard (and any future audit log) can show which check failed, not just that something did.

The is_fallback branch matters: when source_passage_ids is empty, there’s nothing to ground the answer against, so the grounding check is skipped rather than automatically failed. An honest “I don’t know” shouldn’t be penalized the same way a wrong answer is — this is a real design decision, not an oversight, reflected in the distinct approved_fallback label.

Why the corrupted-answer example matters more than the two clean approvals

Two clean approvals prove the critic doesn’t reject good answers. They don’t prove it catches bad ones — a critic that approves everything would pass those same two tests. The corrupted answer exists specifically to close that gap: it’s constructed by hand ("90 days, guaranteed, no matter what" in place of the real "30 days ... no questions asked"), cites a real passage ID, and is run through the exact same CriticAgent.review() call as everything else. The rejection isn’t scripted into the dashboard — it’s the actual, verified output of running the grounding check against text that genuinely doesn’t match its cited source.

Test coverage

Working Demo Link:

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_08_package.

• Wondering why the corrupted answer isn’t constructed via a synthesis bug — it’s built directly as a DraftAnswer in lesson_code.py‘s main block, simulating what a future bug’s output would look like, without needing to actually break SynthesisAgent‘s working code to demonstrate it.

• dashboard.html doesn’t open automatically — start.sh‘s auto-open is best-effort and silently skipped in headless environments; open the file manually.

Real-World Connection

This is the difference between a system that trusts its own components and one that verifies them — and it’s exactly the kind of check that catches problems no single-component test would, because it only fails when two components disagree about what actually happened. A production incident where an answer contradicts its own cited source is a genuinely hard failure to detect from the outside; verifying grounding at the point of generation is far cheaper than discovering it from a user complaint.

Next Steps

Day 9 builds the ResponseFormatter, which takes an approved answer and shapes it for delivery — the last component before Days 10 through 12 wire the full five-piece pipeline together.

• An extractive synthesis agent — answers built directly from retrieved text, never invented

• Source citations on every answer, traceable back to a specific passage ID

• An entity-mismatch check that adds a visible caveat instead of silently answering the wrong question

Why This Matters

Retrieval finds relevant passages; something still has to turn them into an answer. The obvious approach is to hand the passages to a model and let it write a response — but that reintroduces the exact problem Day 6 solved for classification: a fluent answer with no way to verify where any part of it came from. Today’s synthesis stays extractive on purpose: the answer is retrieved text, not a paraphrase of it, so grounding isn’t something to check later — it’s structurally guaranteed.

Core Concept

SynthesisAgent.synthesize() takes the top-ranked passage from retrieval and returns it as the answer, tagged with the passage ID it came from. No rewriting, no summarizing, no risk of the answer saying something the source passage doesn’t. That’s a real trade-off — extractive answers can read less naturally than generated ones — accepted deliberately for the traceability it buys.

The agent also checks something classification alone can’t: whether the entity Day 6 extracted from the query actually appears in what got retrieved. Ask about the refund policy for a “lifetime” plan, and retrieval correctly finds the closest passage about refunds — but that passage is about annual plans, because no passage mentions lifetime plans at all. Verified by running the code, not assumed: the agent catches this and adds an explicit caveat, "Requested plan type 'lifetime' was not found in the top result; showing the closest available match instead." The answer still ships, but it ships honestly.

Queries Day 6 already routed out_of_scope never reach this agent at all — synthesis is skipped entirely, visible on the dashboard as a distinct, deliberate state rather than an empty result.

Architecture

SynthesisAgent sits directly after the retrieval stack (Days 2–5) and consumes both its output and Day 6’s Intent object — specifically the extracted entities, which is the only piece of Day 6’s work this agent actually needs. The result is a DraftAnswer: text, source passage IDs, an entity-match boolean, and an optional caveat string.

Implementation

def synthesize(self, intent: Intent, retrieved: list[RetrievedPassage]) -> DraftAnswer:
    top = retrieved[0]
    plan_type = intent.entities.get("plan_type")

    entity_match = True
    caveat = ""
    if plan_type and plan_type not in top.text.lower():
        entity_match = False
        caveat = f"Requested plan type '{plan_type}' was not found in the top result; ..."

    return DraftAnswer(text=top.text, source_passage_ids=[top.passage_id],
                        entity_match=entity_match, caveat=caveat)

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

start.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html. Five query cards render: three clean extractive answers with source citations, one visible entity-mismatch caveat, and one out-of-scope query shown as skipped rather than answered. Tests: 6 passed.

./stop.sh

Implementation Guide

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_07_package/lesson_07_package

Package contents

lesson_07_package/
├── lesson_code.py    # SynthesisAgent, DraftAnswer, generate_dashboard (+ Day 6's QueryUnderstandingAgent, Days 3-5's retrieval stack)
├── test_lesson.py    # 6 tests verifying extraction, citations, and caveats
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_07_package
chmod +x start.sh stop.sh
./start.sh

You should see dashboard.html generated, and 6 tests passed. Open the dashboard — five query cards, three with clean matched answers, one showing a visible entity-mismatch caveat, one shown as skipped (out of scope).

Cleanup: ./stop.sh — also removes the generated dashboard.html.

Architecture, in more detail

SynthesisAgent.synthesize() takes exactly two inputs: an Intent (from Day 6) and a list of RetrievedPassage objects (from the Days 3–5 retrieval stack). It doesn’t call either component directly — the pipeline wiring in if __name__ == "__main__": is responsible for calling QueryUnderstandingAgent, checking in_scope, and only calling HybridSearcher and SynthesisAgent when the query passed Day 6’s gate. This keeps each agent testable in isolation, the same discipline established on Day 1.

The entity-match check only fires when intent.entities contains a plan_type — if the query has no extracted entity (like the plain “How do I cancel my subscription?” query), there’s nothing to check against, and entity_match defaults to True with no caveat.

Why extractive synthesis, specifically, and what it costs

An extractive answer is bounded to exactly what a source passage says — there is no code path in synthesize() that can produce text absent from retrieved. That’s the entire safety argument: verify the source passages are correct once, and every answer built from them inherits that correctness automatically.

The real cost: extractive answers can read awkwardly, especially when a query would ideally combine facts from two different passages into one sentence. Today’s agent uses only the single top-ranked passage — it doesn’t attempt to combine multiple passages into a smoother answer. That’s a genuine limitation, not hidden here: a more capable synthesis approach (bullet points from multiple sources, or eventually model-generated text with citations) is a real next step, deliberately deferred until Phase 1’s eval harness exists to measure whether a more complex approach is actually more correct, not just more fluent.

Reading the entity-mismatch example honestly

The “lifetime plan” query wasn’t picked because it was guaranteed to fail — several queries were tested first, and most matched cleanly. This one triggers the caveat because the mock passage set genuinely contains no passage mentioning “lifetime,” so retrieval correctly falls back to the closest related passage (an annual-plan refund policy) rather than returning nothing. The caveat exists precisely for this situation: retrieval did its job — it found the most similar available passage — but similarity isn’t the same as correctness, and the agent says so instead of hiding the gap.

Test coverage

Working Demo Link:

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_07_package.

• A query you try yourself doesn’t trigger a caveat you expected — the entity-match check only compares against the top retrieved passage; if a lower-ranked passage happens to mention the entity, that’s not currently checked. That’s a real, acknowledged limitation, not a bug.

• dashboard.html doesn’t open automatically — start.sh‘s auto-open is best-effort and silently skipped in headless environments; open the file manually.

Real-World Connection

Extractive-first synthesis, with generation reserved for later once evaluation and guardrails exist, is a common production pattern for exactly this reason — it’s far easier to audit a system that can only say what its sources say than one that can say anything at all. The caveat behavior mirrors a real support pattern too: a good agent, human or automated, says “I found something related but not exactly what you asked” instead of quietly answering an adjacent question as if it were the one asked.

Next Steps

Day 8 builds the CriticAgent, which reviews this draft answer before it reaches a user — the last checkpoint in the pipeline before formatting and delivery.

• A weighted, multi-category classifier replacing Day 1’s single-keyword stub

• Entity extraction that pulls structured facts out of a raw question

• An explicit confidence threshold that routes uncertain queries away from retrieval instead of guessing

Why This Matters

Day 1’s QueryUnderstandingStage picked the first keyword substring it found and called it done — no confidence, no way to say “I’m not sure.” That’s fine for a proof of concept; it’s a liability in production, where a system that confidently misclassifies a query has already made its first mistake before retrieval even runs. Today’s agent is the first component in the pipeline built to know the difference between “confident” and “guessing.”

Core Concept

Instead of a first-match rule, the agent scores every category against the query using weighted keywords, then normalizes those scores into a probability-like distribution that sums to 1.0. The category with the highest share wins — but only if that share clears a confidence threshold. Below the threshold, the agent doesn’t guess; it returns out_of_scope and the query never reaches retrieval at all.

This matters because ambiguity is real, not a bug to eliminate. Run today’s code on “Can I get a refund if I cancel my annual plan?” and the agent produces a genuine tie: refund_policy and cancellation score exactly 0.5 each. A more nuanced case — “What is the cost to cancel and get a refund on my purchase?” — splits three ways: 0.44 / 0.33 / 0.22. The confidence score isn’t a made-up number; it’s a direct, honest readout of how much the query actually leans toward one category over the others.

Entity extraction runs alongside classification, pulling out details like plan type (”annual,” “monthly,” “lifetime”) whenever they’re mentioned — structured facts the retrieval stage can use later, instead of re-parsing the raw text.

Architecture

QueryUnderstandingAgent.process() is the pipeline’s first real seam: raw text goes in, a structured Intent object comes out, carrying the winning category, its confidence, the full score breakdown across every category, extracted entities, and an explicit in_scope boolean. Everything downstream — retrieval, synthesis, the critic — can trust that boolean instead of re-deriving it.

Implementation

def process(self, query_text: str) -> Intent:
    category_scores = self._score_categories(query_text.lower())
    best_category = max(category_scores, key=category_scores.get)
    best_confidence = category_scores[best_category]

    if best_confidence < CONFIDENCE_THRESHOLD:
        return Intent(query_text=query_text, intent_label="out_of_scope",
                       confidence=best_confidence, in_scope=False, ...)

    return Intent(query_text=query_text, intent_label=best_category,
                   confidence=best_confidence, in_scope=True, ...)

The threshold (0.34) isn’t tuned against real data yet — there’s no eval harness to tune it against until Phase 1. Today it’s a reasonable default, made visible and adjustable in one place rather than buried in scattered if-statements.

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

start.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html. Five test queries render as cards — three clean classifications, one genuine tie, and one out-of-scope weather question correctly routed away from retrieval. Tests: 6 passed.

./stop.sh

Implementation Guide

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_06_package/lesson_06_package

Package contents

lesson_06_package/
├── lesson_code.py    # QueryUnderstandingAgent, Intent, generate_dashboard
├── test_lesson.py    # 6 tests verifying classification, entities, and scope routing
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_06_package
chmod +x start.sh stop.sh
./start.sh

You should see dashboard.html generated, and 6 tests passed. Open the dashboard — five query cards, each with category confidence bars, extracted plan-type entities where relevant, and a scope badge.

Cleanup: ./stop.sh — also removes the generated dashboard.html.

Architecture, in more detail

_score_categories() sums keyword weights per category, then divides each category’s raw score by the total across all categories — this normalization is what makes category_scores sum to 1.0 and behave like a confidence distribution rather than an arbitrary number.

_extract_entities() runs independently of classification — a query can be out_of_scope and still have an entity extracted, or in_scope with no entities found at all. The two operations don’t depend on each other.

process() combines both, then makes the single most consequential decision in the agent: compare the winning category’s confidence against CONFIDENCE_THRESHOLD (0.34) and decide whether the query proceeds to retrieval or gets labeled out_of_scope and stops here.

Why the confidence threshold matters more than the classification itself

Anyone can write a classifier that always returns its best guess. The harder, more valuable behavior is knowing when not to guess. With only three categories, a genuinely ambiguous query can score as low as 0.33 per category in a perfect three-way split — which is why the threshold sits at 0.34, just above an even three-way tie. A query that’s genuinely torn between all three categories gets routed to out_of_scope rather than picking one arbitrarily; a query that clearly leans toward one category, even without total certainty, still gets through.

This threshold is a placeholder default, not a tuned value — there’s no eval harness yet (that starts in Phase 1) to measure whether 0.34 is actually the right cutoff for real queries. Making it a single named constant, rather than scattering the logic across the codebase, is what makes it possible to tune later without a rewrite.

Test coverage

Working demo link:

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_06_package.

• A query you try yourself classifies unexpectedly — check CATEGORY_KEYWORDS directly; this is a small, hand-written keyword list, not a trained model, so it only recognizes what’s explicitly listed.

• dashboard.html doesn’t open automatically — start.sh‘s auto-open is best-effort and silently skipped in headless environments; open the file manually.

Real-World Connection

Systems that skip explicit out-of-scope handling tend to fail the same way: a user asks something the system was never built to answer, and instead of saying so, it confidently retrieves the closest-sounding passage and answers anyway. That’s a harder failure to catch than an outright error, because nothing in the response signals anything went wrong. An explicit confidence floor is one of the cheapest reliability wins available, and it’s why it comes this early in the pipeline.

Next Steps

Day 7 builds the SynthesisAgent, which takes the retrieval results this agent’s in_scope decision permits and turns them into a drafted answer.

• A bigram-overlap reranker — a phrase-level signal neither BM25 nor vector similarity can see

• A second pass over Day 4’s hybrid search candidates

• A visual dashboard showing exactly which passages moved, and why

Why This Matters

Both of Day 4’s scorers — BM25 and vector similarity — treat a passage as a bag of independent words. Neither can tell the difference between “cancel at any time” and “at any point you may cancel,” even though only one of those actually contains the phrase a user asked about. Reranking exists because word order carries real information, and it’s cheap to check once the candidate pool is already small.

Core Concept

Bigram overlap scores a query against a passage by comparing their two-word sequences, not their individual words. “Cancel at any time” produces the bigrams (cancel, at), (at, any), (any, time) — a passage containing that exact phrase shares all three; a passage with the same words scrambled shares none.

This is a real, order-sensitive reranking signal, distinct from anything in the retrieval stage. It’s also deliberately cheap enough to run on only the top handful of candidates — reranking every passage in a large corpus this way wouldn’t scale, but reranking Day 4’s top 3 costs nothing.

Running today’s code on the query “cancel at any time” produces a genuine, verified rank change: hybrid search ranks doc_cancel_p0 first and doc_refund_p1 second, because doc_cancel_p0 shares more individual words overall. But doc_refund_p1 contains the exact phrase “cancel at any time” verbatim, while doc_cancel_p0 only has “cancel your subscription at any time” — the phrase is broken up. Reranking correctly promotes doc_refund_p1 to first place. That’s not a scripted outcome; it’s what the bigram math actually produces on this query.

Architecture

Reranker.rerank() takes Day 4’s HybridSearcher output directly — it doesn’t touch the passage store or recompute anything from scratch. It scores each candidate’s bigram overlap against the query, re-sorts by that score, and records both the original hybrid rank and the new final rank, so the dashboard can show the movement explicitly.

Implementation

def bigram_overlap_score(query: str, passage_text: str) -> float:
    query_bigrams = bigrams(tokenize_ordered(query))
    passage_bigrams = bigrams(tokenize_ordered(passage_text))
    intersection = query_bigrams & passage_bigrams
    union = query_bigrams | passage_bigrams
    return len(intersection) / len(union)

Note tokenize_ordered() deliberately keeps stopwords that Day 3 and Day 4’s tokenizer dropped — bigrams need real word adjacency to mean anything; filtering out “at” before forming pairs would destroy the exact phrase this lesson depends on.

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

start.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html. Each passage renders with a rank badge — moved up, moved down, or unchanged — plus its bigram and hybrid scores side by side. Tests: 6 passed.

./stop.sh

Implementation Guide

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_05_package/lesson_05_package

Package contents

lesson_05_package/
├── lesson_code.py    # bigram scoring, Reranker, generate_dashboard (+ Day 4's HybridSearcher, BM25, embeddings)
├── test_lesson.py    # 6 tests verifying bigram scoring and rank changes
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_05_package
chmod +x start.sh stop.sh
./start.sh

You should see dashboard.html generated, and 6 tests passed. Open the dashboard — doc_refund_p1 should show a “moved up 1” badge, doc_cancel_p0 a “moved down 1” badge.

Cleanup: ./stop.sh — also removes the generated dashboard.html.

Architecture, in more detail

Two tokenizers exist in this lesson for a specific reason:

• tokenize() (from Day 3/Day 4) strips stopwords, used for BM25 and embeddings, where individual word presence is what matters.

• tokenize_ordered() (new today) keeps every word, including stopwords, because bigrams need genuine adjacency — dropping “at” from “cancel at any time” would turn it into “cancel any time,” destroying the exact phrase match this lesson depends on.

Reranker.rerank() takes HybridResult objects directly from HybridSearcher.search(), computes bigram_overlap_score() for each, and re-sorts. It records hybrid_rank (the incoming order) and final_rank (the outgoing order) on every result, which is what lets the dashboard show movement rather than just a final list.

Why this specific query demonstrates the point honestly

The query “cancel at any time” was chosen after testing several candidate queries and checking the actual output — not written first and assumed to work. Most queries tested left the hybrid order unchanged after reranking; this one produces a genuine flip because it’s constructed from an exact phrase that appears verbatim in one passage (doc_refund_p1: “...but you can cancel at any time”) and only in scrambled form in another (doc_cancel_p0: “...cancel your subscription at any time”). Reranking doesn’t guarantee a rank change on every query — it only matters when phrase structure and bag-of-words agreement genuinely diverge, which is exactly what happened here.

Test coverage

Working Demo Link:

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_05_package.

• No rank change visible for a query you try yourself — expected for most queries; the demo query was specifically chosen to show a genuine flip, not because reranking always changes the order.

• dashboard.html doesn’t open automatically — start.sh‘s auto-open is best-effort and silently skipped in headless environments; open the file manually.

Real-World Connection

This two-stage pattern — a cheap, broad first pass followed by an expensive, narrow second pass — is the standard shape of production retrieval systems, usually with a trained cross-encoder model instead of today’s bigram heuristic. The principle is identical either way: don’t pay the cost of precise scoring on every passage in the corpus, only on the handful that already made it through the first cut.

Next Steps

Day 6 begins Phase 0’s agent build: the QueryUnderstandingAgent, the first of the five components that will call this retrieval stack as part of a complete pipeline.

Most NLP tutorials stop at isolated notebooks: one file for tokenization, another for sentiment, a third for intent, none wired to HTTP, persistence, or observability. week_25_26_aiml_integrated_project is a self-contained capstone that consolidates classical text processing, transformer tokenization, embeddings, sequence modeling, sentiment analysis, intent routing, and response selection into one runnable system:

• An installable Python package (week2526_python) with a core engine, learning lab wrappers, and a product NLP pipeline

• A single FastAPI backend exposing a lab path for component exploration and a product path for analyze, predict, and experiment training

• A React dashboard with live telemetry, a support-assistant workflow, a pipeline studio, and an operations console

• Postgres-backed experiment metadata, PyTorch artifacts on disk, and production hardening (timeouts, structured logging, graceful ML degradation)

The engineer payoff is operational clarity: one /analyze call runs preprocess → intent → sentiment → response; training jobs become Run rows you poll from the dashboard; missing models fall back to lexicon and keyword rules instead of crashing the API.

Core Components

Each block below is a distinct capability inside this repository—not an external dependency.

Classical text pipeline

week2526_python.core.text_pipeline provides NLTK-backed normalization, tokenization, stemming, lemmatization, POS tagging, and TF-IDF feature extraction. nltk_bootstrap.py lazily downloads punkt and stopwords only when classical processing runs, keeping cold starts fast.

Subword tokenization

core.tokenization.distilbert and core.tokenization.subword wrap Hugging Face tokenizers (DistilBERT, BERT, GPT-2). Lab endpoints expose side-by-side token comparisons without loading full models on every health check.

Word embeddings

core.embeddings.glove loads GloVe vectors or falls back to a deterministic synthetic vocabulary for offline demos. glove_bootstrap.py caches vector files under GLOVE_CACHE_DIR on startup when WEEK2526_BOOTSTRAP_MISSING=true.

Character sequence model

core.sequence_models.char_lstm implements a character-level LSTM for text generation. Checkpoints live under artifacts/sequence/char_lstm.pt after a lab train call.

Sentiment stack

The sentiment module spans vocabulary building (vocabulary.py), Bi-LSTM architecture (model.py), training (training.py), lexicon scoring (lexicon.py), and inference (inference.py). Artifacts: artifacts/sentiment/best_model.pt, vocab.json, and metrics.json.

Intent classification

core.intent.classifier trains a DistilBERT head over six support intents. core.intent.lightweight provides keyword-based routing when no checkpoint exists or when WEEK2526_LIGHTWEIGHT_MODE=true. Checkpoint path: artifacts/intent/intent_classifier.pt.

Response selection

core.response_selection.selector maps intent + sentiment score to template replies, optionally reranking with embedding similarity when ML artifacts are present.

Product NLP engine

core.nlp_engine.pipeline implements analyze_text, predict_sentiment_product, system_status, and bootstrap_missing_assets. This is the orchestration layer the product API calls through app.services.model_loader.

Model registry loader

backend/app/services/model_loader.py lazy-loads sentiment and intent models behind a thread-safe registry, exposes readiness(), and never raises on missing artifacts—lexicon and lightweight intent are used instead.

Experiment and training orchestration

SQLAlchemy models (Experiment, Run, Artifact) plus async repositories manage product training. nlp_train_job.py schedules background jobs with asyncio.create_task and supports synchronous training via ?sync=true.

FastAPI surface and middleware

backend/app/main.py defines the only FastAPI() instance. Middleware includes request timeouts (30s default, 600s for long train routes), upload size limits (WEEK2526_MAX_UPLOAD_MB), JSON request logging with request_id and latency_ms, and environment-aware CORS.

React dashboard

The frontend (frontend/src/ui/) ships four views—Overview, Support Assistant, Pipeline Studio, and Operations—with sidebar navigation, live metric polling (4s global, 3s during active training runs), animated pipeline progress on analyze, and module cards grouped by topic (text processing, embeddings, sentiment, intent and response).

Architecture Overview

Surface table

http://localhost:3000

OperatorsMulti-panel NLP console proxied through Nginx

Architecture diagram

The diagram below shows how the dashboard, API layer, Python package tiers, and persistence volumes connect. External caches (NLTK, Hugging Face) feed the core package without blocking the product path in lightweight mode.

Read more

• A BM25 keyword scorer — the same ranking function real search engines use

• A hybrid ranker that blends it with Day 3’s vector similarity

• The fix for misranking, and an honest look at where hybrid search has its own blind spot

Why This Matters

Vector similarity alone ranked an unrelated passage above the actual refund policy for a refund question — a real failure, not a contrived one. Vectors capture fuzzy semantic overlap; they don’t guarantee the passage containing the exact term you asked about ranks first. Keyword search has the opposite problem: it can’t tell “cancel” and “terminate” mean the same thing. Today we stop choosing between them.

Core Concept

BM25 scores a passage against a query using term frequency, weighted by how rare that term is across the whole passage set — a term every passage contains tells you nothing; a term only one passage contains is a strong signal. This is the same core algorithm behind most production keyword search, not a simplified stand-in for it.

Blending BM25 with vector similarity isn’t as simple as adding the two scores — they live on different scales. So both get min-max normalized into a 0–1 range first, then combined with a weighted average controlled by alpha (0.5 splits the weight evenly). The result: a passage that wins decisively on either signal still ranks well, and a passage that’s mediocre on both doesn’t sneak to the top the way it could with vectors alone.

Run today’s code against the refund-policy query from Day 3, and the fix is visible directly: doc_refund_p0 — the actual refund policy — now ranks first, winning on both keyword match and vector similarity. That’s the win. But hybrid search isn’t a universal fix — it’s two imperfect signals combined, not one perfect one. A query built around a single rare word that happens to appear in an unrelated passage can still pull BM25 (and therefore the blend) in the wrong direction. Today’s code makes that failure mode visible too, not just the success.

Architecture

HybridSearcher owns both scorers: a BM25 index built once over the passage set, and Day 3’s embed_text() reused unchanged. search() scores a query against both, normalizes each score list independently, then blends them into one ranked list.

Implementation

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_04_package/lesson_04_package

def search(self, query: str, top_k: int = 3) -> list[HybridResult]:
    keyword_scores = self.bm25.score(query)
    vector_scores = [cosine_similarity(embed_text(query), v) for v in self.passage_vectors]

    norm_keyword = normalize(keyword_scores)
    norm_vector = normalize(vector_scores)

    combined = [
        self.alpha * norm_vector[i] + (1 - self.alpha) * norm_keyword[i]
        for i in range(len(self.passages))
    ]
    ...

alpha is a real tuning knob, not a formality — Phase 1’s eval harness (starting Day 13) is what eventually tells you the right value for your own corpus, rather than guessing.

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

start.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html — open it in your browser to see the result, not a terminal log. Each passage renders as a card with three color-coded score bars: combined (green), keyword/BM25 (orange), and vector (blue), ranked top to bottom.

For this lesson’s query, doc_refund_p0 — the actual refund policy — renders at the top with all three bars near full, while the two unrelated passages visibly trail behind it. Tests: 8 passed.

./stop.sh

Implementation Guide

Package contents

lesson_04_package/
├── lesson_code.py    # BM25, HybridSearcher, normalize, generate_dashboard (+ Day 3's embed_text, cosine_similarity)
├── test_lesson.py    # 8 tests verifying BM25, hybrid blending, and dashboard output
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_04_package
chmod +x start.sh stop.sh
./start.sh

You should see dashboard.html generated in the directory, and 8 tests passed. Open the dashboard in your browser — doc_refund_p0 should render at the top with the fullest bars across all three scores for the refund-policy query.

When you’re done: ./stop.sh — also removes the generated dashboard.html.

Architecture, in more detail

BM25.__init__() builds a document-frequency index once, over the whole passage set — how many passages contain each term. BM25.score(query) uses that index to score every passage against a query’s tokens, using the standard BM25 formula (k1=1.5, b=0.75 — the commonly used defaults).

HybridSearcher wraps a BM25 instance and a set of Day 3 embeddings computed once at construction time. search() scores both, min-max normalizes each list independently (so neither signal dominates just because its raw numbers happen to be larger), and blends them with alpha.

Why normalize before blending, specifically

BM25 scores and cosine similarities live on completely different numeric ranges — BM25 can produce values like 3.7, cosine similarity is bounded to [-1, 1]. Adding them directly would let BM25’s larger raw numbers dominate the blend regardless of alpha. Min-max normalization rescales each list to [0, 1] independently first, so alpha actually controls the blend the way it’s meant to.

Reading the real results honestly

For the refund-policy query, hybrid search works cleanly — doc_refund_p0 wins on both signals and ranks first. That’s a genuine fix for Day 3’s misranking, verified by running the code, not asserted.

It’s worth testing other queries yourself, because hybrid search isn’t a universal fix. BM25’s IDF weighting means a single rare shared term can still dominate a score the way hash collisions dominated Day 3’s vector similarity — the failure mode changes shape, it doesn’t disappear. This is exactly why Phase 1 (starting Day 13) builds a real eval harness instead of eyeballing a handful of example queries: with only 4 passages, any single example can look great or look bad by chance. A golden dataset and real pass-rate tracking is what actually tells you whether alpha=0.5 is the right choice for your corpus.

Working Demo Link

Test coverage

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_04_package.

• dashboard.html doesn’t open automatically — start.sh tries open/xdg-open as a best-effort convenience and silently skips if neither is available (common in headless/sandboxed environments); just open the file manually in any browser.

• Different rankings than expected for a query you try yourself — expected; this is a 4-passage toy corpus, and small corpora amplify exactly the quirks described above. Don’t read too much into any single query’s result at this scale.

• Wondering why alpha isn’t tuned automatically — that’s intentional. Automatic tuning needs an eval harness to measure against, which doesn’t exist until Phase 1.

Real-World Connection

This is the default retrieval architecture in most production search-and-answer systems for exactly this reason: neither signal alone is trustworthy enough to ship. A support system relying on vectors alone will occasionally miss the passage with the exact policy number a user asked about; one relying on keywords alone will miss a passage that says the same thing in different words. Combining them is standard practice, not an advanced technique.

Next Steps

Day 5, we add reranking — a second pass over the top candidates that can use more expensive, more accurate scoring now that hybrid search has already narrowed the field.

• A deterministic text embedder — no API call, no training, no cost

• A 256-dimension vector for every passage from Day 2

• The similarity search that Day 4’s retrieval stage will build on — including an honest look at where it falls short

Why This Matters

Yesterday’s passages are just text — comparing a question against them character by character finds nothing unless the words match exactly. To search by meaning, text has to become numbers first: a vector, positioned so that similar meaning means similar position. Today we build the simplest version of that idea that still works, using a technique real production systems actually use — not a toy simplification of one.

Core Concept

The hashing trick turns each word in a passage into a position in a fixed-length vector, using a hash function instead of a learned vocabulary. Every occurrence of a word nudges the same vector position by the same amount, so two passages that share vocabulary end up with vectors pointing in a similar direction — measurable with cosine similarity, a score from -1 (opposite) to 1 (identical).

This is a real technique, used at scale specifically because it needs no training step and no vocabulary file — you can embed text the moment you see it. It’s also genuinely weaker than a learned embedding model, and today’s run shows exactly why: with only a handful of passages and short text, shared common words and hash collisions can outweigh the one word that actually matters for relevance. Stripping obvious stopwords (”the,” “a,” “can”) helps, but doesn’t eliminate the problem — it’s a structural limit of the technique, not a bug in today’s code.

Architecture

Text goes in, a fixed-length vector comes out — deterministically, so the same passage always produces the same vector. Embedder.embed_passages() runs this over every passage from Day 2’s passage store and writes the result to embeddings.json. rank_by_similarity() previews what Day 4’s retrieval stage will do: embed a query the same way, then rank passages by cosine similarity.

Implementation

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_03_package/lesson_03_package

def embed_text(text: str, dim: int = 256) -> list[float]:
    vector = [0.0] * dim
    for token in tokenize(text):
        digest = hashlib.md5(token.encode()).hexdigest()
        hash_int = int(digest, 16)
        index = hash_int % dim
        sign = 1.0 if (hash_int // dim) % 2 == 0 else -1.0
        vector[index] += sign
    # L2-normalize so cosine similarity behaves correctly
    ...

Two properties are worth testing directly, because they’re provably true regardless of vocabulary quirks: identical text always produces similarity 1.0, and text with zero shared vocabulary always produces similarity 0.0.

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

Expected output:

Embedding 4 passages (dim=256)...
  doc_refund_p0 -> vector with 9 non-zero dims
  doc_refund_p1 -> vector with 10 non-zero dims
  doc_cancel_p0 -> vector with 7 non-zero dims
  doc_pricing_p0 -> vector with 10 non-zero dims
Embeddings saved -> embeddings.json

Query: "Can I get my money back on an annual subscription?"
  doc_cancel_p0  similarity=0.169
  doc_refund_p0  similarity=0.149
  doc_pricing_p0  similarity=0.124

== Running tests ==
6 passed

Notice the top result, doc_cancel_p0, isn’t obviously the best answer to a refund question — doc_refund_p0 (the actual refund policy) ranks second. That’s not an error; it’s the hashing trick’s real behavior on a small, short passage set, and it’s genuine motivation for Day 4.

./stop.sh

Implementation Guide

lesson_03_package/
├── lesson_code.py    # tokenize, embed_text, cosine_similarity, Embedder
├── test_lesson.py    # 6 tests verifying embedding correctness properties
├── requirements.txt  # pytest only — no API keys needed
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, embeddings.json, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_03_package
chmod +x start.sh stop.sh
./start.sh

You should see 4 passages embedded into 256-dimension vectors, an embeddings.json file written, a similarity ranking for a sample query, and 6 passed.

When you’re done: ./stop.sh.

Architecture

embed_text() is the core function — everything else calls it:

1. tokenize() lowercases the text, strips punctuation, splits on whitespace, and drops stopwords.

2. For each remaining token, MD5-hash it, use the hash to pick a vector index (hash % dim) and a sign (+1 or -1, from a different bit of the same hash).

3. Add that signed value into the vector at that index.

4. L2-normalize the final vector — divide every value by the vector’s magnitude — so cosine similarity is comparing direction, not raw word count.

Embedder.embed_passages() runs this over a list of passages and returns one PassageEmbedding per passage. rank_by_similarity() embeds a query with the same function and sorts passages by cosine_similarity() against it.

Reading the actual similarity results honestly

Running this lesson produces:

doc_cancel_p0  similarity=0.169
doc_refund_p0  similarity=0.149
doc_pricing_p0  similarity=0.124

for the query “Can I get my money back on an annual subscription?” — even though doc_refund_p0 is the passage that actually answers a refund question. This is real, reproducible behavior, not a bug to fix quietly. Two things are happening:

• With only 4 passages of 7–11 tokens each, a single shared word (annual, subscription) carries a lot of weight in either direction — there isn’t enough text for chance overlaps to average out.

• MD5-hashing into a 256-dimension space still has occasional collisions between unrelated tokens, and at this passage count, one lucky or unlucky collision can flip the ranking.

Both effects shrink as passage count and text length grow, and disappear entirely with a learned embedding model (which this lesson deliberately doesn’t use yet, to stay free and dependency-light). The two tests that matter for correctness — identical text scores 1.0, completely disjoint vocabulary scores 0.0 — hold exactly, because those are mathematical properties of cosine similarity and L2 normalization, not properties that depend on passage count.

Working Demo Link:

Test coverage

Troubleshooting

• Different similarity scores than the article — if you change MOCK_PASSAGES, EMBEDDING_DIM, or STOPWORDS, expect different rankings; the hashing trick is deterministic per configuration, not across configurations.

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside lesson_03_package.

• Wondering why we didn’t just use a real embedding API today — that’s intentional. Day 3 stays free and dependency-light on purpose; later phases introduce real API-backed embeddings once the eval harness (Phase 1) exists to catch regressions when you do.

Real-World Connection

This is exactly why production retrieval systems rarely trust a single vector-similarity score by itself. A system that only ranks by embedding similarity will occasionally surface a plausible-looking wrong answer with high confidence — and nothing in the score itself signals the mistake. That gap is a large part of why hybrid approaches exist.

Next Steps

Day 4, we add hybrid search: combining today’s vector similarity with keyword-based scoring, so a passage that shares the exact right word ranks correctly even when the embedding alone gets it wrong.

• A document ingestor that turns raw text into uniform, retrievable passages

• A fixed-size chunking strategy with overlap, so context doesn’t get cut mid-thought

• The passage store that Day 4’s retrieval stage will search over

Why This Matters

Yesterday’s RetrievalStage was a hardcoded dictionary — real answers to three fake questions. Today it becomes real, because real documents don’t arrive in question-answer pairs. They arrive as long, unstructured text: policy docs, support articles, transcripts. Before any search can work, that raw text has to become a set of small, consistent units a retrieval system can actually compare against a query. Get chunking wrong here, and every later phase inherits the mistake — no eval harness or LLM judge can fix an answer built from a badly cut passage.

Core Concept

A document is too big to search directly. If you embed and compare a 4,000-word support article against a 12-word question, the match is diluted — the one relevant sentence gets buried in three thousand irrelevant ones. So we split documents into passages: smaller chunks a retrieval system can score individually.

The chunking decision has a real trade-off. Chunks too large re-introduce the dilution problem. Chunks too small lose context — a passage that ends mid-sentence, or separates a term from its definition, can’t be understood on its own even if it’s retrieved correctly. Today we use fixed-size character chunking with overlap: each passage shares a small window of text with its neighbor, so an idea that spans a chunk boundary still appears whole in at least one passage.

Every passage also carries metadata: which document it came from, and where in that document it sits. This isn’t optional bookkeeping — Day 4’s retrieval stage needs it to return results a user can trust, and Day 3’s embeddings need a stable ID to attach a vector to.

Architecture

Ingestion sits before the pipeline, not inside it: it’s a one-time (or periodically re-run) process that produces the passage store Stage 2 (Retrieval) will query. Raw documents go in one end; a flat list of Passage objects — each with an ID, source document, character offsets, and text — comes out the other.

Implementation

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_02_package/lesson_02_package

Two data contracts, matching the discipline from Day 1:

@dataclass
class Document:
    doc_id: str
    source: str
    raw_text: str

@dataclass
class Passage:
    passage_id: str
    doc_id: str
    text: str
    char_start: int
    char_end: int

The ingestor does one job per method: load_documents() reads raw text in, chunk_document() splits one document into overlapping passages, and ingest_all() runs both across every document and returns the full passage store.

Build, Test, Verify

chmod +x start.sh stop.sh
./start.sh

Expected output:

Loaded 3 documents
Document 'refund_policy.txt' -> 3 passages
Document 'cancellation.txt' -> 1 passages
Document 'pricing.txt' -> 1 passages
Total passages ingested: 5
Passage store saved -> passages.json

== Running tests ==
5 passed

./stop.sh

Implementation Guide

Package contents

lesson_02_package/
├── lesson_code.py    # Document, Passage contracts + DocumentIngestor
├── test_lesson.py    # 5 tests verifying chunking behavior
├── requirements.txt  # pytest only
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, passages.json, caches, Docker image
└── README.md          # quick-start reference

Run it

cd lesson_02_package
chmod +x start.sh stop.sh
./start.sh

start.sh creates a virtual environment, installs requirements.txt, builds the Docker image if Docker is present, runs lesson_code.py, then runs test_lesson.py. You should see 3 documents ingested into 5 total passages, a passages.json file written to disk, and 5 passed.

When you’re done: ./stop.sh — removes the venv, the generated passages.json, all caches, and the Docker image if one was built.

Architecture

Two dataclasses carry the data:

Type Fields Produced by Document doc_id, source, raw_text load_documents() Passage passage_id, doc_id, text, char_start, char_end chunk_document()

DocumentIngestor.ingest_all() is the entry point: it calls load_documents() once, then chunk_document() once per document, and returns a flat list of Passage objects — the passage store.

Why fixed-size chunking with overlap, specifically

• chunk_size=220 characters keeps each passage small enough that a single topic dominates it, without cutting it down to a fragment.

• overlap=40 characters means each new chunk starts 40 characters before the previous one ended. If a sentence spans the boundary between chunk N and chunk N+1, it appears complete in chunk N+1 even though it was cut off in chunk N.

• Short documents (shorter than chunk_size) produce exactly one passage — there’s nothing to split, and the loop exits after the first pass since end == len(text).

This is deliberately the simplest correct chunking strategy. Production systems often chunk on sentence or paragraph boundaries instead of raw character counts — that’s a refinement worth making later, but it changes chunk_document()‘s internals only. The Passage contract and everything downstream stays the same, which is the entire point of the typed-boundary discipline from Day 1.

Working Demo Link:

Test coverage

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run tests from inside the lesson_02_package directory.

• Different passage counts than expected — passage counts depend on the exact chunk_size and overlap values in DocumentIngestor(); if you change them, the counts in the article and README will no longer match your run — that’s expected, not a bug.

• passages.json already exists from a previous run — start.sh overwrites it each run; stop.sh removes it during cleanup.

Real-World Connection

This is the step most teams skip past without thinking about — and the one that quietly determines whether retrieval works at all. A support-triage system that chunks a refund policy document badly will confidently retrieve a passage that’s missing the one sentence that mattered, and no amount of prompt tuning downstream fixes that. Chunking strategy is infrastructure, not a detail.

Next Steps

Day 3, we turn these passages into embeddings — numeric vectors that let the retrieval stage compare a question against thousands of passages by meaning, not just keyword overlap.

• The five-stage contract every later phase of this course plugs into

• A working, testable pipeline skeleton — no real model calls required

• The one architectural decision that determines whether you can add evals, audits, and cost controls later without a rewrite

Why This Matters

Most AI features start as one function: take a question, call a model, return the answer. It works in the demo. It becomes unmaintainable the first time something goes wrong, because there’s no place to insert a check, log a decision, or measure a cost — everything happens inside one opaque call. Today’s architecture decision is the one choice you can’t easily undo later, so we make it deliberately, on Day 1, before writing a line of logic.

Core Concept

A single-call AI feature has exactly one seam: the API call itself. If the output is wrong, you have one place to look and no way to tell whether the problem was understanding the question, finding the right information, or writing the answer. You’re debugging a black box.

The fix isn’t a bigger model — it’s more seams. We split the work into five stages, each with a narrow, explicit responsibility: Query Understanding turns a raw question into a structured intent. Retrieval finds the information needed to answer it. Synthesis drafts an answer from that information. A Critic checks the draft against the original intent before anyone sees it. Formatting shapes the final response for its destination — API, chat window, or app.

Five stages sounds like more surface area to break. It’s the opposite. Each seam is a place where next week’s eval harness can score an output, where a future audit log can record a decision, and where a cost tracker can attribute a dollar. A monolithic call gives you none of that. The stages don’t make the system more complex — they make the complexity visible, one boundary at a time.

The rule for today: every stage takes a typed input and returns a typed output. Nothing passes a raw string or a loose dictionary between stages. That discipline is what lets Phase 1 through Phase 5 attach reliability tooling to this pipeline without touching the pipeline itself.

Architecture

Data flows in one direction: a Query enters, and a FinalResponse leaves. Each stage owns one transformation and knows nothing about the stages before or after it beyond the shape of the data it receives and returns. The Pipeline object is the only piece that knows the full sequence — swap any one stage’s implementation later, and nothing else needs to change.

Implementation

GitHub Link

https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_01_package%20(1)/lesson_01_package

Start by defining the data contracts — these are the types every stage will pass along:

@dataclass
class Query:
    text: str
    session_id: str

@dataclass
class RetrievedContext:
    passages: list[str]

@dataclass
class DraftAnswer:
    text: str

@dataclass
class CriticVerdict:
    approved: bool
    reason: str

@dataclass
class FinalResponse:
    text: str
    approved: bool

Each stage is a class with one method — run() — that takes the previous stage’s output type and returns its own. Today, each stage’s internal logic is intentionally simple (deterministic, no API calls) — you’re proving the seams work before you make any single stage smarter. The Pipeline class calls all five in sequence and returns the final response.

Build, Test, Verify

Everything — install, build, run, test, verify — is one command:

chmod +x start.sh stop.sh
./start.sh

start.sh installs dependencies, builds the Docker image if Docker is available (skips gracefully if not), runs the pipeline, runs the tests, and prints a verification summary:

Query received: "What's the refund policy for annual plans?"
Stage 1 (Query Understanding) -> intent: refund_policy
Stage 2 (Retrieval) -> 2 passages found
Stage 3 (Synthesis) -> draft answer generated
Stage 4 (Critic) -> approved: True
Stage 5 (Formatting) -> response ready
Final: "Annual plans can be refunded within 30 days of purchase."

== Running tests ==
4 passed

When you’re done, clean everything up:

./stop.sh

Implementation Guide

Package contents

lesson_01_package/
├── lesson_code.py    # the five stages + Pipeline class
├── test_lesson.py    # 4 tests verifying the pipeline is wired correctly
├── requirements.txt  # pytest only — no API keys needed today
├── Dockerfile         # minimal python:3.11-slim image
├── start.sh           # install deps, build, run, test, verify — one command
├── stop.sh            # cleanup — removes .venv, caches, and the Docker image
└── README.md          # quick-start reference

Run it

cd lesson_01_package
chmod +x start.sh stop.sh
./start.sh

start.sh does five things in order: creates a virtual environment, installs requirements.txt, builds the Docker image if Docker is present (skips cleanly if not), runs lesson_code.py, then runs test_lesson.py. You should see 5 stage log lines, a final answer, and 4 passed.

When you’re done: ./stop.sh — removes the venv, __pycache__, .pytest_cache, and the Docker image if one was built.

Working Demo Link

Architecture

Five typed dataclasses define every handoff between stages:

From To Type caller Query Understanding Query (text, session_id) Query Understanding Retrieval Intent (query, intent_label) Retrieval Synthesis RetrievedContext (intent, passages) Synthesis Critic DraftAnswer (context, text) Critic Formatting CriticVerdict (draft, approved, reason) Formatting caller FinalResponse (text, approved)

Each stage class exposes exactly one public method, run(), that takes the previous type and returns the next. The Pipeline class is the only object that knows the full call order — this is what makes it possible to replace, say, RetrievalStage in Day 2 without touching SynthesisStage, CriticStage, Pipeline, or any test outside test_lesson.py.

Why each stage exists

• QueryUnderstandingStage — classifies intent with a simple keyword match today. This is the seam Phase 1’s eval harness will score against a golden dataset of real questions.

• RetrievalStage — looks up from a hardcoded dict today. Day 2 replaces the internals with real ingestion and search; the run(intent) -> RetrievedContext contract doesn’t change.

• SynthesisStage — picks the first matching passage today. This becomes the seam where Phase 1’s LLM-as-judge later scores answer quality.

• CriticStage — the seam Phase 1’s CI gate eventually wraps: nothing ships past this point without passing a check.

• FormattingStage — the seam Phase 3’s audit trail will log from — every FinalResponse becomes a queryable, replayable event.

Test coverage

Troubleshooting

• ModuleNotFoundError: No module named 'lesson_code' — run test_lesson.py from inside the lesson_01_package directory, not from a parent folder.

• Docker build step skipped — expected if Docker isn’t installed locally; start.sh falls back to a plain Python run automatically.

• Python version errors — this lesson uses list[str] generic syntax, which requires Python 3.9+; the Dockerfile pins 3.11 to avoid this entirely.

Real-World Connection

This pattern is why a well-run support-escalation system can tell you exactly which step misfired when a customer gets a wrong answer — was it a retrieval miss, a synthesis error, or a critic that should have caught it? Systems built as one opaque call can only tell you that something, somewhere, went wrong. That difference is the entire reason this course spends 90 days on it instead of shipping the one-call version and hoping.

Next Steps

Day 2, we build the first real stage: ingestion. You’ll take unstructured source documents and turn them into the retrievable units that Stage 2 will search over — the raw material the rest of the pipeline depends on.

• A multi-capability NLP Intelligence API that classifies intent, scores sentiment, and generates contextual responses — all served through a single REST endpoint.

• A modular PyTorch-based pipeline where each NLP component is independently testable and swappable without touching the rest of the system.

• A production-style deployment with Docker, a health-check endpoint, structured logging, and a minimal web demo interface.

Why This Matters

You’ve spent 176 days learning how individual pieces work — tensors, layers, training loops, embeddings, sequence models. What separates a student from a practitioner is the ability to compose those pieces into a system that solves a real problem end-to-end. Every major NLP product at companies like OpenAI, Cohere, or Hugging Face is not one model — it’s a pipeline of coordinated components: input normalization, classification, generation, post-processing, and serving. This capstone is your first time building that pipeline yourself, which means by the end of Day 180, you’ll have produced something you can legitimately show in a job interview or deploy for real users.

Core Concepts

1. The NLP Pipeline as a System, Not a Script

A pipeline is a directed graph of transformations. Each node takes in a well-defined input and produces a well-defined output. The moment you start building pipelines this way, your code becomes composable. You can swap a rule-based intent classifier for a neural one without touching the sentiment module. You can add a new component — say, entity extraction — without modifying the response generator.

In production systems at Stripe or Shopify, the ML inference path looks exactly like this: an API gateway hands the raw text to a normalization service, which passes the cleaned text to a classification model, which produces a structured prediction object, which routes to a generation or retrieval system, which formats the final response. Each handoff is a typed contract. Today you’re building that exact pattern.

2. Intent Classification with Lightweight Transformers

The dominant pattern for intent classification in production is fine-tuning a pre-trained transformer on a small labeled dataset. DistilBERT — a compressed version of BERT — gives you 97% of the accuracy at 40% of the size. This is the same trade-off that powers intent routing in voice assistants at Amazon and Google.

The key insight here is that the pre-trained model already understands language at a deep level. Your fine-tuning job is not to teach it language — it’s to teach it your categories. When you have only a few hundred labeled examples, this transfer learning approach outperforms training from scratch by a wide margin. Your final classifier is really just a linear layer sitting on top of a frozen or lightly-unfrozen transformer backbone.

3. Sentiment-Aware Response Generation

The response generation in this system is a hybrid: a retrieval layer picks a candidate response template based on intent and sentiment score, and a neural reranker scores each candidate against the original query context. This avoids the hallucination risk of pure generation while still producing contextually relevant responses.

This pattern — retrieval-augmented generation with a reranker — is exactly what systems like Perplexity AI and Bing Chat use at scale. The reranker is a cross-encoder transformer that takes (query, candidate) pairs as input and outputs a relevance score. You’re not building the full RAG stack today, but the architecture you implement mirrors the production pattern precisely.

4. API Design for ML Systems

Serving an ML model is fundamentally different from serving a database. Model inference has unpredictable latency, GPU resource contention, and serialization overhead. Three patterns matter most:

Request batching: Group multiple incoming requests together before sending to the model. A batch of 32 requests takes only 1.5× the time of a single request on a GPU, so throughput multiplies dramatically. Your system implements a simple synchronous batch endpoint.

Response caching: Identical or near-identical queries should return cached predictions. A simple Redis-backed LRU cache cuts GPU load significantly in production. The architecture includes a cache layer even if you implement it in-memory for the capstone.

Structured error envelopes: Every response — success or failure — should follow the same JSON schema. Clients must be able to rely on the response structure regardless of what happened internally. Your API will always return {status, data, error, latency_ms}.

5. Component Architecture

The preprocessor runs first and its output is shared by both the intent classifier and the sentiment scorer — they run in parallel, then their outputs converge at the response selector. This parallel fan-out pattern keeps latency bounded to max(t_intent, t_sentiment) rather than t_intent + t_sentiment.

                   NLP Intelligence API

   Input Text
       |
       v
   [Preprocessor]  -->  normalize, tokenize, truncate
       |
       +-----------------------------+
       v                             v
   [Intent Classifier]        [Sentiment Scorer]
   (DistilBERT + linear)      (LSTM / Day 176)
       |                             |
       +-------------+---------------+
                     v
            [Response Selector]
            (retrieve + rerank)
                     |
                     v
              [JSON Response]

Read more

Computer vision prototypes usually die in notebooks: one script for preprocessing, another for training, a third for evaluation, and nothing that tracks experiments or serves inference behind a stable HTTP contract. week_23_24_aiml_integrated_project is a self-contained capstone that consolidates the full vision pipeline into one runnable system:

• An installable Python package (week2324_python) with concept-named modules for preprocessing, CNNs, pooling, datasets, training loops, evaluation, export, and deployment

• A FastAPI backend exposing two API surfaces: a lab path for topic exploration and a product path for experiment lifecycle management

• VisionClassifier Platform — a React dashboard with a guided pipeline, component studio, live training monitor, and multipart image inference

• PostgreSQL-backed experiment metadata (async SQLAlchemy + Alembic), named Docker volumes for artifacts and dataset caches, and offline-safe synthetic fallbacks when CIFAR archives cannot be downloaded

The engineer payoff is traceability and boundaries. A quick CNN smoke test is not a stray script—it is train_simple_cnn behind a lab route with capped batches. Production training is not a blocking HTTP call—it is POST /api/v1/cv/experiments/{id}/train returning HTTP 202 while asyncio.create_task runs ResNet fine-tuning in a worker thread, with the dashboard polling run status until completion or failure.

Core Components

The stack is layered. Each topic below is a distinct building block; together they form one coherent image-classification platform.

Image preprocessing and normalization

image_pipeline.py handles OpenCV/PIL ingestion, ImageNet-style normalization profiles (CIFAR10_MEAN / CIFAR100_MEAN), and build_inference_transform for 224×224 upload tensors. preprocess_upload_bytes powers multipart inference on the product API.

Convolutional network architectures

SimpleCNN provides a pedagogical three-block classifier for CIFAR-10 smoke tests. ImageClassifier is a deeper three-block production-style CNN with checkpoint saving. build_resnet18_classifier wraps torchvision ResNet18 with a replaceable FC head for 100-way CIFAR-100 classification. inspect_feature_maps in cnn/debug.py exposes spatial activations for architecture demos.

Pooling strategies

Read more

• Wire together everything from Days 169–172 into a single end-to-end pipeline

• Build a production-style sentiment classifier on the IMDB movie reviews dataset

• Train, evaluate, and serve the model through a lightweight REST inference endpoint

• Measure real metrics — accuracy, F1 score, and confusion matrix — not just training loss

Why This Matters

Every product that handles user-generated text has a sentiment layer somewhere. Amazon uses it to surface trending complaints before they become PR crises. Airbnb uses it to flag toxic reviews in real time. Spotify uses it on social data to figure out which artists are gaining buzz before the charts reflect it. The pattern is always the same: preprocess text, encode meaning, classify intent, act on the signal. You have all three of those pieces now — tokenization from Day 169, embeddings from Day 170, and sequence modeling from Day 172. This project stitches them into a deployable system and teaches you the engineering discipline of closing the loop from raw data to a live prediction endpoint.

Core Concepts

1. The Full NLP Pipeline as One Unit

Up to now you have built each stage in isolation. A sentiment analyzer forces you to reason about the pipeline as a single contract: garbage in one stage poisons every stage after it. The input to your model is not text — it is a fixed-length integer sequence produced by a specific tokenizer trained on a specific vocabulary. That tokenizer must be saved alongside the model weights or the model is useless at inference time. This is the single most common mistake junior engineers make when deploying NLP systems: they checkpoint the weights and throw away the tokenizer.

Think of it like a lock-and-key pair. The model is the lock; the tokenizer is the key. Ship them as one artifact.

2. Bidirectional LSTMs for Sentiment

A standard LSTM reads a sentence left to right. But the word “not” near the beginning of a sentence can completely invert the meaning of a word that appears twelve tokens later. A Bidirectional LSTM runs two passes — one forward, one backward — and concatenates the hidden states at each timestep. The model can now “see” what comes after a word before it decides how to weight that word. For sentiment tasks, Bi-LSTMs consistently outperform unidirectional ones by three to five percentage points on benchmark datasets with no extra training data.

3. Pretrained GloVe Embeddings vs Learned Embeddings

You can initialize your embedding layer with random weights and let the model learn them from scratch. That works fine if you have hundreds of thousands of training examples. For smaller datasets — under fifty thousand examples — you are better off seeding the embedding layer with pretrained GloVe vectors (100d or 200d). GloVe was trained on billions of tokens; your model gets that knowledge for free and only needs to fine-tune it. The practical difference in validation accuracy on IMDB is typically three to eight points, and training converges two to three times faster.

4. Evaluation Beyond Accuracy

Binary sentiment on IMDB is balanced (50% positive, 50% negative), so accuracy is a fair metric here. The moment you move to a real production dataset — say, customer support tickets where 80% are neutral — accuracy becomes misleading. A model that always predicts “neutral” gets 80% accuracy while being completely useless. F1 score, which balances precision and recall, is the metric that actually tells you whether the model is doing something useful. Wire in scikit-learn’s classification_report from day one so you develop the habit of looking at the full picture.

5. The Inference Contract

A trained model sitting in a .pt file does nothing for a product team. What they need is an inference function with a clean, stable interface: string in, label and confidence score out. In production this function gets wrapped in a FastAPI endpoint, containerized, and deployed behind a load balancer. Building that wrapper is as much a part of the project as training the model. The function needs to handle edge cases — empty strings, non-ASCII characters, extremely long inputs — without crashing.

Read more

This isn’t a course on how to call an LLM API. It’s a masterclass in what happens after your AI feature works in the demo — evaluation, safe deployment, observability, and cost control at production scale. Most tutorials stop at “call the model, print the response.” Here, we start where they end. You will build a five-agent retrieval system, then spend the next 78 days making it trustworthy enough to run unattended in front of real users.

We dismantle the “ship it and hope” approach to AI systems, replacing it with an eval-gated, audit-logged, budget-enforced deployment pipeline. You will learn to wrestle with the specific failure modes of AI software — non-deterministic outputs, silent quality regressions, refusal spikes, and runaway API costs — the class of problems that don’t show up in a HTTP status code. By the end, you won’t just have a working system; you’ll know how to keep any AI system honest once it’s live.

AI Reliability Engineering is a reader-supported publication. Days 1–12 are free for everyone. To receive new posts and support this work, consider becoming a free or paid subscriber.

Each lesson is hands-on — working code, real tests, no pseudocode.

90 Lessons. 90 Days. One System, Built in the Open.

Subscribe Now

1. Course Details

Why This Course?

Standard AI tutorials teach you to call a model and display the output. That’s the whole lesson. It doesn’t survive contact with production, where the same input produces a different output every time, where a “successful” API call can still return something wrong, harmful, or off-topic, and where nobody finds out until a user complains. This course exists to teach the discipline that closes that gap: reliability engineering applied specifically to AI systems, not general software.

What You’ll Build

You will build a complete, five-agent retrieval system, then wrap it in a full reliability stack.

The System: A multi-agent pipeline — query understanding, retrieval, synthesis, a critic, and response formatting — working together as one coordinated flow.

The Eval Harness: A golden dataset, automated scoring, and a calibrated LLM judge that catches quality regressions before they ship.

The Safety Net: Deployment tooling that makes changing a prompt as safe and reversible as any other code change.

The Ledger: An audit trail that makes every decision the system made queryable and verifiable after the fact.

The Governor: Cost tracking and budget enforcement so API spend is predictable instead of a surprise on the invoice.

Who Should Take This Course?

Engineers who’ve shipped an AI feature and are now getting the harder questions: “how do we know it’s not getting worse?”

Backend engineers moving from CRUD APIs into AI-native systems.

Teams who need to explain, after the fact, exactly why their system made a given decision.

Anyone tired of finding out about quality problems from a support ticket instead of a dashboard.

What Makes This Course Different?

No black boxes. You build the eval harness, the audit trail, and the cost governor yourself — not a wrapper around someone else’s SaaS dashboard.

Reliability over demos. We care less about the model’s cleverest answer and more about what happens on its worst day.

Failure first. Every phase includes deliberately breaking the system — a bad prompt, a budget overrun, a corrupted audit record — to prove the safeguards actually catch it.

Key Topics Covered

Eval design: golden datasets, deterministic checks, LLM-as-judge with calibration.

CI gates that block a deploy when quality drops.

Safe prompt deployment: versioning, staged rollout, instant rollback.

Tamper-evident audit logging and full session replay.

Cost attribution, budget enforcement, and circuit breakers for runaway spend.

Multi-agent failure modes and how they differ from single-model failures.

Prerequisites

Comfortable writing and testing Python. Basic familiarity with calling an LLM API (OpenAI or Anthropic). A GitHub account. No prior evaluation, MLOps, or observability experience required.

2. Learning Objectives

By Phase 0 (Build): You will be able to stand up a working five-agent retrieval pipeline and articulate, in plain terms, why “it works in the demo” isn’t the same as “it’s reliable.”

By Phase 1 (Evaluation): You will be able to design a golden dataset, write deterministic and LLM-judge-based checks, and wire a CI gate that blocks a pull request when output quality regresses.

By Phase 2 (Deployment Safety): You will be able to deploy a prompt change through a feature-flag workflow with automatic rollback, instead of editing a string in production.

By Phase 3 (Observability & Audit): You will be able to build a tamper-evident log of every decision the system makes, and answer “why did it do that?” for any past session.

By Phase 4 (Cost Governance): You will be able to attribute API spend to a specific user and feature, enforce budget limits in code, and fail gracefully instead of blowing through a ceiling.

By Phase 5 (Advanced Reliability & Capstone): You will be able to reason about multi-agent-specific failure modes and ship a fully integrated system — eval-gated, audited, and cost-governed — as a portfolio piece.

3. Course Structure

The curriculum is divided into 6 phases across 90 days, mirroring the lifecycle of a production AI system: build it, prove it works, deploy it safely, watch it, control what it costs, then harden it.

Phase 0: Build the System (Days 1–12) — Free. Stand up the five-agent pipeline: architecture, retrieval, and a working baseline.

Phase 1: Evaluation Engineering (Days 13–31) — Paid. Golden datasets, automated scoring, calibrated LLM-as-judge.

Phase 2: Deployment Safety (Days 32–48) — Paid. Feature flags, versioned rollout, instant rollback.

Phase 3: Observability & Audit Trails (Days 49–66) — Paid. Structured logging, tamper-evident audit logs, session replay.

Phase 4: Cost Governance (Days 67–78) — Paid. Cost attribution, budget policy enforcement, circuit breakers.

Phase 5: Advanced Reliability & Capstone (Days 79–90) — Paid. Multi-agent failure modes, full system integration, final deployment.

4. The 90 Days, Lesson by Lesson

Days 1–12

Day 1: Architecture — Designing the Five-Component Pipeline

Day 2: Ingestion — Getting Real Data Into the System

Day 3: Embeddings — Turning Text Into Searchable Vectors

Day 4: Hybrid Search — Combining Keyword and Vector Retrieval

Day 5: Reranking — Improving on Raw Retrieval Order

Day 6: The QueryUnderstandingAgent — Parsing User Intent

Day 7: The SynthesisAgent — Generating Coherent Answers

Day 8: The CriticAgent — Catching Bad Outputs Before They Ship

Day 9: The ResponseFormatter — Shaping Output for the User

Day 10: Wiring the Full Pipeline Together

Day 11: End-to-End Testing Across All Components

Day 12: Baseline Measurement — How the System Fails Today

Days 13–90

Phase 1 — Evaluation Engineering (Days 13–31)

Day 13: Why AI Outputs Need Eval Cases, Not Unit Tests

Day 14: Designing the Golden Dataset Schema

Day 15: Writing Your First Golden Eval Cases

Day 16: Building the Eval Case Intake Process

Day 17: Deterministic Checks — The Fast, Cheap First Layer

Day 18: Building the Composite Scorer

Day 19: Building the Eval Runner

Day 20: Connecting the Runner to the Live Pipeline

Day 21: Tracking Pass Rate Over Time

Day 22: LLM-as-Judge — Evaluating What Structure Can’t Catch

Day 23: Designing Judge Quality Dimensions

Day 24: Building the Mock Judge Backend

Day 25: Connecting the Judge to a Real Model

Day 26: Human-Labeling a Calibration Set

Day 27: Computing Inter-Rater Agreement

Day 28: The Trust Coefficient — When to Trust the Judge

Day 29: Composite Scoring — Checks and Judge Together

Day 30: The CI Gate — Blocking Bad Merges Automatically

Day 31: Phase 1 Review — Full Eval Suite Integration Test

Phase 2 — Deployment Safety (Days 32–48)

Day 32: Why Prompt Changes Need the Same Rigor as Code

Day 33: Designing the Flag Data Model

Day 34: Flag Storage and Version History

Day 35: Building the Flag CLI

Day 36: The Flag Evaluation Engine

Day 37: Deterministic Rollout Assignment

Day 38: Testing Rollout Math at Scale

Day 39: Eval-Gated Promotion

Day 40: Building the Promotion Workflow

Day 41: Version History and Change Audit

Day 42: Rollback in Under 30 Seconds

Day 43: Testing a Bad-Variant Rollback End to End

Day 44: Running A/B Experiments Between Variants

Day 45: Comparing Variants on Quality and Cost

Day 46: Establishing Statistical Confidence

Day 47: Wiring Flags Into the Live Pipeline

Day 48: Phase 2 Review — Full Deployment Safety Integration Test

Phase 3 — Observability & Audit Trails (Days 49–66)

Day 49: Every Decision Is an Event Worth Recording

Day 50: Designing the Audit Event Schema

Day 51: Building the Event Writer

Day 52: Hashing Inputs and Outputs Instead of Raw Data

Day 53: The Async Write Path — Never Block a Request

Day 54: Buffering and Flush Strategy

Day 55: Structured Tracing Across the Pipeline

Day 56: Load-Testing the Async Queue

Day 57: Querying Audit Events by User and Session '

Day 58: Comparing Behavior Across Prompt Versions

Day 59: Replaying a Full Session, Decision by Decision

Day 60: Tamper Evidence — Hash Chaining

Day 61: Verifying Chain Integrity and Detecting Breaks

Day 62: Refusal-Rate Analysis by Topic

Day 63: Cost and Behavior, Side by Side

Day 64: Building the Unified Dashboard

Day 65: Anomaly Detection on Quality Trends

Day 66: Phase 3 Review — Full Observability Integration Test

Phase 4 — Cost Governance (Days 67–78)

Day 67: The Invoice Problem — Why P50 Estimates Are Wrong

Day 68: Building Cost Attribution Middleware

Day 69: Attributing Spend to User and Feature

Day 70: Reconciling Cost Records With the Audit Trail

Day 71: Testing Cost Accuracy End to End

Day 72: Designing Budget Policy Limits

Day 73: Enforcing Limits Before Every API Call

Day 74: Handling a Budget-Exceeded Error Gracefully

Day 75: Testing Budget Enforcement Under Load

Day 76: Circuit Breaker States — Closed, Open, Half-Open

Day 77: Serving a Fallback When Budget Is Exceeded

Day 78: Phase 4 Review — Full Cost Governance Integration Test

Phase 5 — Advanced Reliability & Capstone (Days 79–90)

Day 79: Failure Modes Unique to Multi-Component Systems

Day 80: Simulating Cascading Failures

Day 81: Chaos Testing the Pipeline Under Load

Day 82: Recovery Strategies for Partial Failure

Day 83: Stress-Testing the Full Reliability Stack

Day 84: Wiring All Four Systems Into One Deployment

Day 85: Writing System Documentation

Day 86: Assembling the Unified Observability Dashboard

Day 87: Performance Tuning and Latency Budget

Day 88: Clean-Machine Verification

Day 89: Writing Your “Definition of Done”

Day 90: Final Deployment and Capstone Demo

Subscribe to unlock full access to the private GitHub repository — every line of code, every test, for all 90 days.

Subscribe Here

What We Are Building Today

By the end of this lesson you will have done three things that actually matter:

• Understood why sequential data breaks standard networks — and what RNNs do differently at the architectural level.

• Decoded how LSTMs solve the vanishing gradient problem that crippled early RNNs, using three learned memory gates.

• Built a working character-level language model that generates text one character at a time — the same core idea behind every autocomplete you have ever used.

• Connected this to production systems at Google, Amazon, Apple, and Stripe that run LSTM-based models today.

Why This Matters

Yesterday you built word embeddings — dense vector representations of meaning. Today you learn how to process sequences of those vectors over time. Every autocomplete suggestion you have typed, every translation you have read, every voice command your phone understood — all of it traces back to the sequential modeling ideas you will master today.

Even in the age of transformers, understanding RNNs and LSTMs is not optional. Production systems at Apple (on-device Siri models), Amazon (Alexa keyword detection), and trading firms (time-series forecasting) still run LSTM-based models because they are computationally lighter and interpretable. Knowing why transformers replaced RNNs requires understanding what RNNs got wrong. That knowledge gap is what separates engineers who implement AI from engineers who design it.

Core Concepts

1. The Sequence Problem — Why Feedforward Networks Fall Short

A standard dense neural network treats every input as independent. Feed it “The cat sat on the ___” and it sees seven unrelated word vectors with no awareness of order. The network has no memory — it cannot know that “cat” modifies what comes after “sat.”

Think of it like reading a book where each page is shuffled randomly. You see the words, but the story is gone.

RNNs fix this by introducing a hidden state — a memory vector that gets updated at every time step. At step t, the network takes two inputs: the current word vector x_t and the hidden state from the previous step h_{t-1}. The output hidden state h_t encodes everything the network has seen up to position t.

h_t = tanh( W_hh · h_{t-1}  +  W_xh · x_t  +  b )

That shared weight matrix W_hh is identical at every time step — it is what makes RNNs parameter-efficient. A 10-step sequence and a 1,000-step sequence use exactly the same parameter count.

2. The Vanishing Gradient — The RNN’s Achilles Heel

Here is the bitter irony. Backpropagation through an RNN means multiplying gradients across every time step. If those gradients are smaller than 1.0 (which they almost always are after a tanh), they shrink exponentially. By the time you are 50 steps back, the gradient is effectively zero.

The network literally forgets what happened earlier in the sequence. Training an RNN on a long paragraph and asking it to remember the subject from sentence one is like whispering a secret across 100 people — by the end, the message is noise.

Key Insight — The Vanishing Gradient Problem This is the single biggest reason vanilla RNNs struggle with dependencies longer than roughly 10–20 tokens. Each time step multiplies gradients together, and repeated multiplication of values less than 1.0 sends them toward zero exponentially fast. LSTMs were invented specifically to fix this.

3. LSTMs — Learned Memory Management

Long Short-Term Memory networks (Hochreiter & Schmidhuber, 1997) solve the vanishing gradient problem with a second memory channel: the cell state C_t. Where the hidden state is your working memory, the cell state is long-term storage — and crucially, information flows through it additively, not multiplicatively.

LSTMs use three learned gates to control information flow:

Read more

Notebook lessons teach tensors, Keras Sequential, PyTorch nn.Module, and CIFAR-10 pipelines in isolation. A platform team needs the opposite: one service that runs curriculum demos for learning, a product API for experiments and async training, and a core library that both surfaces import—without duplicating model code across fourteen days.

This project composes Days 141–147 into a single stack: week2122_python (TensorFlow + PyTorch cores), FastAPI with Learning and Vision APIs, Postgres-backed runs, a React dashboard on port 3000, and HTTP 501 stubs for Days 148–154 until lesson code lands.

Takeaway: You will see how to unify two deep-learning frameworks behind one vision engine, expose them through intentional API layers, and keep long training off the request thread.

System Overview

Four bands carry the design:

• React + Nginx (port 3000) — unified pipeline, curriculum studio, operations monitor with live metrics polling.

• FastAPI (port 8000) — /api/v1/week2122/learn/day{N}/... for Days 141–147 demos; /api/v1/vision/... for experiments, Iris comparison, and CIFAR jobs; lesson aliases at /api/v1/learn/lesson-{N}/.

• week2122_python core — core/tensorflow/ (tensors, autodiff, Iris Sequential, MNIST, CIFAR-10 tf.data), core/pytorch/ (tensors, autograd, tabular nets), core/vision_engine/pipeline.py (product composition), core/runtime/write_gate.py (Day 145 freeze pattern).

• Postgres + Alembic — experiments, runs, artifacts; background CIFAR training via cifar_train_job.py and asyncio.to_thread.

Day 141–143 material surfaces as TensorFlow learning routes and MNIST smoke tests. Days 144–146 feed PyTorch learning routes and the Iris compare product endpoint. Day 147 drives CIFAR baseline training and experiment registration.

Engine / Core System Design

The core package owns tensors, training loops, and evaluation metrics. The backend owns HTTP, SQLAlchemy repositories, and job scheduling. Routes validate caps from settings (max epochs, batch limits) and delegate synchronously or enqueue work.

Read more

If you’re working through the 180-Day AI & ML Curriculum — or you’ve followed along this far — you already know how this newsletter works. Daily lessons, real code, no fluff. Lesson 180 of that curriculum wraps up on August 1, 2026.

Starting today, there’s a second track running alongside it: AI Reliability Engineering.

What it is

A 90-day build: a production-grade, multi-agent RAG system, built completely in the open, two lessons a week, every lesson shipping with working code.

The 180-day curriculum takes you from zero to being able to build an AI system. This course picks up from there — it’s about what happens after you ship something: how you evaluate it, deploy it safely, keep it observable, control its cost, and keep it reliable once real users are hitting it. That’s the gap most courses don’t cover.

Subscribe Now

What we build

Five coordinated agents — query understanding, retrieval, synthesis, a critic, and response formatting — wired into one working pipeline. Not a toy demo. A real system with real failure modes, measured and fixed in public.

Phase 0 (Days 1–12): Build the system from scratch. Ingestion, embeddings, hybrid search, reranking, the full multi-agent pipeline, and a baseline read on how it actually breaks. All 12 lessons are free.

After Day 12, the course goes deep on production concerns: evaluation engineering (Days 13–31), deployment safety (32–48), observability and audit trails (49–66), cost governance (67–78), advanced reliability patterns (79–83), closing with a full capstone deployment (84–90).

Every lesson includes:

• A working, tested code package — not pseudocode

• A full written breakdown of the design decisions and trade-offs

• Diagrams showing the architecture and where we are in the 90-day arc

The schedule

New lessons publish every Monday and Thursday, starting Day 1 shortly after this post.

• Days 1–12: free, permanently. No rush to catch up before a paywall closes.

• Days 13–90: paid.

Who this is for

• Anyone who finished the 180-day curriculum and wants the next step — the production-reliability side of AI engineering

• Engineers with a working RAG or multi-agent system who need it to hold up under real traffic, not just in a demo

• Anyone who learns better from working code and real failure logs than from slide decks

You don’t need to have finished the 180-day course to start this one — Phase 0 assumes no prior lessons from this newsletter, just general familiarity with building software.

A note on paid

Days 1–12 stay free forever. From Day 13, lessons are for paid subscribers, under the same plans as the rest of the newsletter:

Plan Price Includes Monthly $39/month - Full curriculum access, private GitHub repo

Annual $279.30/year - Same, plus free ebook + 1 month systemdrd.com portal access

Lifetime Access $699/year - Unlimited access to entire course library, past/present/future

Free$0 - Basic free lessons only, no private repo access

First lesson

Day 1 — the architecture lesson, why the system is shaped the way it is before a single line of implementation code — publishes shortly.

Subscribe (free through Day 12) and follow the code on GitHub. See you in Day 1.

• Understand why raw text is unusable by neural networks and what embeddings solve

• Explore the architecture of Word2Vec (Skip-gram and CBOW) and GloVe

• Implement embedding lookups, similarity search, and semantic arithmetic in Python

• Connect word vectors to production NLP systems at scale

Why This Matters

Every search engine, recommendation system, and large language model you have ever used runs on one foundational idea: words can be represented as points in high-dimensional space, and words with similar meanings cluster near each other. Without this, a model cannot know that “bank” and “financial institution” refer to similar concepts, or that “Paris” relates to “France” the same way “Tokyo” relates to “Japan.”

Word embeddings are the bridge between human language and mathematical computation. They are the reason modern NLP works at all.

Core Concepts

The Problem: Language Has No Natural Numbers

Neural networks process tensors — grids of floating point numbers. The letter “A” means nothing to a matrix multiplication. One-hot encoding is the naive fix: represent 50,000 vocabulary words as a 50,000-dimensional vector where only one position is 1. It works, but it is catastrophically wasteful and semantically blind. “Dog” and “canine” have zero overlap in one-hot space even though they mean the same thing.

Word2Vec: Learning Meaning From Context

Word2Vec, introduced by Google researchers in 2013, learns embeddings by training a shallow neural network on a deceptively simple task: given a word, predict its neighbors, or given neighbors, predict the center word.

Skip-gram — Takes a center word, predicts surrounding context words. Works well for infrequent words.

CBOW (Continuous Bag of Words) — Takes surrounding context words, predicts the center word. Faster, better for frequent words.

The key insight is that the network never actually cares about the prediction task. What matters is the weight matrix it builds while solving that task — a 300-dimensional vector per word that encodes semantic relationships the model learned purely from co-occurrence patterns in billions of text examples.

The “aha” moment. After training on billions of words, the geometry that emerges is astonishing. King minus man plus woman equals queen. Paris minus France plus Germany equals Berlin. These are not programmed rules. The geometry emerges entirely from statistics.

Read more

• Why raw text is useless to a neural network, and what preprocessing actually does

• The tokenization spectrum: from character-level splits to subword vocabularies

• How modern NLP pipelines (BERT, GPT, LLaMA) tokenize text at production scale

• Building a preprocessing pipeline from scratch: cleaning, normalization, tokenization, and encoding

• Where this sits in a live AI system’s data flow

Why This Matters

Yesterday you got the 10,000-foot view of NLP. Today you go one level down into the engine room. Every language model — whether it’s the BERT model powering Google Search or the GPT-4 backbone behind ChatGPT — starts with the exact same problem: text is just a sequence of Unicode characters, and neural networks only understand numbers.

Text preprocessing is the translation layer. It transforms messy, human-written language into clean, structured numerical representations that a model can actually learn from. Get this wrong and your model learns noise. Get it right and you’ve handed your model a consistent, information-dense signal. At companies like OpenAI, Hugging Face, and Cohere, entire teams are dedicated to tokenizer design — not because it’s glamorous, but because a bad tokenizer caps the ceiling on everything built on top of it.

Tokenizers are not utilities. They are model components.

Core Concepts

1. The Raw Text Problem

Neural networks are mathematical functions. They compute dot products, matrix multiplications, and nonlinear activations — all operations that require floating-point numbers. The sentence "The model crashed at 3am!!" has no numeric form until you create one.

Before you even think about tokenization, the text itself needs to be cleaned. Production text is full of HTML artifacts (&, <br/>), inconsistent Unicode, mixed case, multiple whitespace characters, and null bytes from encoding issues. These irregularities don’t just add noise — they fragment your vocabulary. "Apple", "apple", and "APPLE" would all become different tokens if you skip lowercasing. That’s three entries in your vocabulary table for the same semantic concept.

The cleaning phase handles:

• Lowercasing (where appropriate)

• HTML unescaping and tag stripping

• Unicode normalization (NFC vs NFD)

• Punctuation handling

• Whitespace collapsing

This is not optional housekeeping — it directly determines vocabulary size, which determines model parameter count.

2. Tokenization Strategies

Tokenization is the act of splitting text into discrete units called tokens. Three main strategies exist, and each makes a different tradeoff.

Word-level tokenization splits on whitespace and punctuation. Simple and interpretable — "the cat sat" becomes ["the", "cat", "sat"]. The fatal flaw: out-of-vocabulary (OOV) words. If your training data never contained "transformer", the model has no representation for it at inference time. Vocabularies also explode — English alone has 500,000+ words, and that’s before you add technical jargon, names, and typos.

Character-level tokenization splits into individual characters. No OOV problem by definition — every character is in your alphabet. But sequences become extremely long, and the model must learn to compose meaning from scratch, which requires vastly more training data and compute.

Subword tokenization is the practical solution that modern systems use. The insight is elegant: keep common words as single tokens, but split rare or unknown words into meaningful sub-pieces. The word "tokenization" might become ["token", "ization"]. The word "unhappiness" might become ["un", "happiness"]. The model sees morphemic structure rather than opaque character soup.

Two dominant subword algorithms:

Byte Pair Encoding (BPE) — used by GPT-2, GPT-3, GPT-4, and LLaMA. Starts with character-level tokenization and iteratively merges the most frequent adjacent pairs until a target vocabulary size is reached.

WordPiece — used by BERT. Same general idea as BPE, but uses a likelihood-based merge criterion instead of raw frequency. Results in a vocabulary of 30,000–100,000 tokens that covers essentially any input text without OOV issues.

3. Vocabulary and Encoding

Once you have tokens, you need to map them to integers — this is the vocabulary lookup. The vocabulary is a dictionary: {"the": 0, "cat": 1, ...}. Every token in your corpus gets a unique integer ID. Special tokens are added explicitly to handle structural positions in a sequence.

After encoding, each sentence is a list of integers. This is the actual input to the embedding layer — not the words themselves.

4. Attention Masks and the Padding Problem

Batching sequences requires them to be the same length. But sentences are not. The solution is to pad shorter sequences with [PAD] tokens up to the maximum sequence length in the batch, then provide an attention mask — a binary vector marking which positions are real tokens (1) and which are padding (0).

The transformer’s attention mechanism uses this mask to ignore padded positions during computation. This is architectural plumbing that most tutorials skip, but it’s fundamental to how batched inference works at production scale.

Component Architecture

The preprocessing pipeline has five distinct stages. Each stage is stateless except the Tokenizer and Encoder — both depend on a trained vocabulary file (typically a .json or .txt vocab file loaded at startup). In production systems, this vocabulary file is versioned alongside the model weights.

Raw Input Text
      |
      v
+---------------------+
|   Text Cleaner      |  --> strip HTML, normalize unicode, collapse whitespace
+---------------------+
      |
      v
+---------------------+
|   Normalizer        |  --> lowercase, remove/keep punctuation, handle numbers
+---------------------+
      |
      v
+---------------------+
|   Tokenizer         |  --> split into subword tokens using BPE/WordPiece vocab
+---------------------+
      |
      v
+---------------------+
|   Encoder           |  --> map tokens --> integer IDs via vocabulary lookup
+---------------------+
      |
      v
+---------------------+
|   Padder/Masker     |  --> pad to batch length, generate attention masks
+---------------------+
      |
      v
 Tensor [batch_size, seq_len]  <-->  Model Input

Critical production note: Swapping a vocabulary file without updating the corresponding model weights is a silent failure mode. The model receives a different token distribution than it was trained on and produces degraded outputs with no error thrown. Tokenizer and weights are always versioned together.

Real-World Connection

At Stripe, every merchant-facing text field — chargeback descriptions, dispute notes, fraud explanations — passes through a preprocessing pipeline before hitting the risk classification model. The tokenizer there is trained on financial domain text to ensure that terms like "ACH reversal" and "NSF fee" are preserved as meaningful units rather than split arbitrarily.

At Google, the search query tokenizer handles 8.5 billion queries per day, many in low-resource languages where subword tokenization is the only viable strategy. The design of that tokenizer — its vocabulary size, its handling of numerals and punctuation — directly affects ranking quality for billions of searches.

Tokenizers are not utilities. They are model components.

Implementation: Build, Test and Demo

Github Link:

https://github.com/sysdr/aiml/tree/main/day170/day170_tokenization

What you will build: A complete, five-stage NLP preprocessing pipeline in Python, ending with batched tensors ready to feed directly into a BERT or GPT-2 model. All code lives in lesson_code.py. The test suite (20 pytest tests) lives in test_lesson.py.

Expected output: a confirmation message and a listing of the five files created inside day170_tokenization/.

Step 1 — Set Up the Environment

Installs all pinned dependencies, and downloads the NLTK and spaCy data required for tokenization exercises.

Expected output: "Environment ready." followed by the activation command. If you see a warning about a missing Rust compiler, ignore it — the Hugging Face tokenizers library ships pre-compiled wheels.

With Docker instead: docker build -t day170 . && docker run --rm day170

Step 2 — Run the Lesson Demo

lesson_code.py runs all seven sections of the pipeline in sequence, printing labeled output for each stage. Read through the terminal output as it runs — every printed block corresponds to a Core Concept section above.

python lesson_code.py

Step 3 — Run the Test Suite

The test suite has 20 tests organized into six classes, each covering a distinct stage of the pipeline.

pytest test_lesson.py -v

Expected result: 20 passed in approximately 8 seconds. The first run will download the BERT and GPT-2 tokenizer files from Hugging Face (~500 MB total). Subsequent runs use the local cache.

To also see line coverage:

pytest test_lesson.py -v --cov=lesson_code --cov-report=term-missing

Step 4 — Key Exercises to Try Yourself

Exercise A — Compare tokenization of a medical vs. general sentence

from lesson_code import load_bert_tokenizer, bert_tokenize

tok = load_bert_tokenizer()
medical = "The patient presented with tachycardia and diaphoresis."
general = "The patient felt dizzy and was sweating."

print(bert_tokenize(tok, medical))
print(bert_tokenize(tok, general))

Pay attention to how the medical terms fragment into subwords. This is exactly why clinical NLP teams train domain-specific tokenizers.

Exercise B — Inspect the token distribution of a larger corpus

from lesson_code import load_bert_tokenizer, token_length_analysis

sentences = [
    "BERT uses WordPiece tokenization.",
    "GPT uses byte pair encoding.",
    "Both are subword tokenization methods.",
    "The quick brown fox jumps over the lazy dog.",
    "Tokenizers convert text into numbers.",
]

tok = load_bert_tokenizer()
print(token_length_analysis(tok, sentences))

Change max_length in bert_encode_batch to the max value from this analysis. Compare the number of padding tokens before and after.

Exercise C — Find where BERT and GPT-2 agree

from lesson_code import load_bert_tokenizer, load_gpt2_tokenizer, compare_tokenizers

words = ["running", "cats", "the", "AI", "unhappiness"]
result = compare_tokenizers(load_bert_tokenizer(), load_gpt2_tokenizer(), words)

for word, splits in result.items():
    match = splits['bert_wordpiece'] == splits['gpt2_bpe']
    print(f"{word:15s}  match={match}  bert={splits['bert_wordpiece']}")

Common English function words like "the" and "cats" produce identical single-token results in both tokenizers. Rare or morphologically complex words diverge.

Verification Checklist

Before moving to Day 171, confirm you can answer each of these from memory or by looking at lesson_code.py:

1. Why does skipping lowercasing inflate vocabulary size?

2. What does the ## prefix mean in a WordPiece token list?

3. What is the integer ID for [CLS] in bert-base-uncased? (Hint: check bert_tok.cls_token_id)

4. What shape does bert_encode_batch return for 3 sentences padded to max_length=32?

5. Why do BERT and GPT-2 produce different splits for the same word?

6. What does a 0 in the attention_mask indicate, and why does it matter?

7. If you swap the vocab.json file without updating model weights, what happens at inference?

Working Code Demo:

Read more