Agent quality cannot be judged by reading a few replies. This lesson builds a scoring service that grades outputs against golden cases with exact-match, containment, token-F1, and an offline LLM-as-judge heuristic. Scores aggregate in a thread-safe store and appear on a live dashboard. One demo request exercises all four scorers, so operators read real numbers.

Highlights

• Four scorers: strict, soft, lexical, and judged.

• Offline judge stub — no API key, deterministic in CI.

• Thread-safe running means, pass rate, bounded event ring.

• Dashboard polling /metrics every second.

• Containerised on port 8094 with health check.

What We Build

• eval/ package: cases.py, scorers.py, metrics.py, service.py.

• Routes /evaluate, /score, /demo, /metrics, /cases, /health, /dashboard.

• A CLI demo printing per-case scores and suite averages.

• Six golden cases: exact hits, soft matches, refusals, wrong answers.

Connection to Previous Lesson

Day 14 bounded history with token budgets. Bounded prompts are the precondition for fair scoring: otherwise a metric drop cannot be separated from silent truncation.

Enables Next Lesson

Day 16 adds a vector store and RAG. Retrieval quality only matters if measurable, so this becomes the harness for retrieved-context answers.

Architecture Context

• Where it sits: after the model response, before any release gate.

• Why it exists: manual review does not scale and cannot be regression-tested.

• Problem solved: converts text quality into numeric, alertable signals.

Integration

Integration Components

EvalCase is a Pydantic model holding id, prompt, expected, actual. Each metric is a pure function registered in METRIC_FNS. EvalService.evaluate() scores a suite and delegates aggregation to the shared METRICS store. Routes return results and a fresh snapshot together, so the dashboard needs no second request.

Module Objectives Alignment

The objective is provable, non-zero measurement. Dictionary registration means a new metric needs no change to the service, routes, or rendering loop.

Core Concepts

Strict versus soft correctness. exact_match scores 0.0 for a correct answer wrapped in a sentence; contains accepts the wrapping. Reporting both separates formatting failures from knowledge failures.

Token-F1. Precision is overlap over actual tokens, recall over expected tokens, F1 their harmonic mean. It grants partial credit for verbose-but-correct answers while penalising padding, since precision falls as unrelated tokens appear. Whitespace tokenisation avoids tokenizer churn.

LLM-as-judge. Production judges call a stronger model; this graded heuristic returns 0.98 exact, 0.92 contained, 0.45 partial, 0.15 miss, 0.05 refusal — same interface, deterministic and offline.

Aggregation as evidence. The store keeps cumulative sums, recomputes means on write, and retains 24 events: enough for a dashboard, bounded so memory cannot grow.

Production AI System Design Relevance

This is the gate between generation and release. Prompts change constantly upstream; users see results downstream. Scoring a fixed suite makes a model swap measurable rather than a guess.

Workflow

• Request flow: the dashboard polls GET /metrics; a demo click issues POST /demo.

• Execution flow: load the suite, score every case with every metric, mark pass or fail, record the batch.

• Data flow: scores become events; sums update cumulative and last-run means; the snapshot returns to caller and UI.

• State changes: cases_scored, eval_runs, pass_count, fail_count, demo_runs, and means advance per run.

Production Integration

• Architecture fit: one uvicorn worker, since counters live in process memory.

• Enterprise patterns: swap the judge stub for a model call behind a timeout and cache; version the dataset.

• Scalability: scoring is O(cases × metrics) and CPU-bound; shard suites, externalise state before replicating.

• Observability: export means, pass rate, run counts; alert on drops between builds.

• Security: no credentials committed, .env ignored, event details truncated.

Step-by-Step Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering-p/tree/main/lesson15/aiam-day15

cases.py defines the schema first, so malformed cases fail at construction rather than mid-run. scorers.py keeps metrics pure functions of one case, making units trivial. metrics.py guards mutation with a lock and recomputes means in the same critical section. service.py is the only place scoring and recording are sequenced, so API and CLI cannot drift. Pydantic rejects empty fields with HTTP 422; smoke checks assert no dashboard field stays zero. Failures surface as FAIL events.

Coding Highlights

Registering scorers by name keeps the pipeline extensible:

METRIC_FNS = {"exact": exact_match, "contains": contains,
              "token_f1": token_f1, "llm_judge": llm_judge}
scores = {name: fn(case) for name, fn in METRIC_FNS.items()}

Sums and derived means update under one lock:

with self._lock:
    self._data["exact_sum"] += exact_sum
    self._data["exact_mean"] = round(self._data["exact_sum"] / total_cases, 3)

Splitting these would expose a new sum with a stale mean — a metric that never occurred.

Working Demo Link:

Production Considerations

• Scalability: one worker in-memory; Redis when replicated.

• Security: authenticate mutating routes; never log prompts.

• Monitoring: alert on pass-rate deltas and on a stalled suite.

• Logging: case ids and scores only.

• Testing: per-scorer units, aggregation, API contracts.

• Failure handling: bad cases rejected at validation; empty suites no-op rather than divide by zero.

• Edge cases: empty strings, refusal stubs, case-only differences.

Validation

• Verification: /health healthy, /demo non-zero, /dashboard serves HTML.

• Testing strategy: 16 pytest cases across scorers, aggregation, and routes, plus live smoke checks.

• Success criteria: tests pass; no dashboard field stays zero after a demo.

• Expected outputs: exact≈0.167, contains≈0.667, token_f1≈0.281, llm_judge≈0.657, pass rate ≈66.7%.

• Benchmarks: the six-case suite scores in single-digit milliseconds.

• Production checklist: versioned cases, pinned dependencies, health check, regression alert.

Key Insights

A suite where everything passes cannot detect regressions, which is why two cases fail deliberately. The common mistake is shipping exact-match alone: it fails correct answers and pushes teams to fix formatting instead of substance. Judge fidelity trades against cost — semantic judges catch paraphrase but add latency and flaky builds.

Important Classes

• EvalCase — validated case schema.

• MetricsStore — locked counters, means, event ring.

• EvalService — orchestrates scoring and recording.

Key Methods

• score_case() — all metrics for one case.

• EvalService.evaluate() — scores a suite, returns averages.

• MetricsStore.record_eval() — updates aggregates atomically.

• MetricsStore.snapshot() — consistent read with throughput.

Success Criteria

Every dashboard metric is non-zero after one demo, tests pass without warnings, and the container reports healthy on port 8094.

Real-World Examples

• Model upgrade gating: a support team scores golden tickets against a candidate model. Exact-match falls while containment and judge scores rise, exposing a formatting change, not quality loss.

• Prompt regression in CI: a docs assistant scores its suite per pull request. An edit meant to shorten answers cuts token-F1 recall by eleven points, failing the build before release.

Parallel tool fan-out fills history faster than models can accept. This lesson bounds growth with token budgets, priority trimming, and sliding-window compaction. ContextWindow counts tokens, drops low-value turns, and keeps dialogue under available = max_tokens - reserved. MetricsStore records every fit; the dashboard polls /metrics. POST /demo runs both strategies so counters leave zero.

Highlights

• Stable chars//4 token estimates without deprecated tokenizer APIs.

• Priority trim that preserves high-priority system and policy messages.

• Sliding-window compaction that keeps the newest turns under budget.

• Live dashboard for tokens, utilization, trim events, and demo runs.

• Docker lifecycle on port 8093 via start.sh, demo.sh, and run_tests.sh.

What We Build

• context/ package: tokens.py, window.py, metrics.py, service.py.

• FastAPI (app.py): /fit, /sliding, /demo, /metrics, /health, /dashboard.

• CLI demo (main.py) comparing priority trim versus sliding window.

• Default overfull conversation: one system message plus eight user/assistant turns.

Connection to Previous Lesson

Day 13 overlapped independent tool I/O with asyncio.gather. Gathered results inflate prompts; Day 14 keeps that context inside token budgets.

Enables Next Lesson

Bounded prompts enable fair evaluation. Day 15 scores agent outputs under controlled windows so metric drift is not caused by silent truncation.

Architecture Context

• Where this component sits: between the conversation store and the model call site.

• Why it exists: unbounded history and tool payloads hit hard context limits or silently degrade quality.

• Problem solved: deterministic trim policies with operator-visible token and message counters.

Read more

This lesson turns independent agent tool calls into an async execution pipeline. Sequential await is the latency baseline; asyncio.gather overlaps I/O so wall time tracks the slowest tool, not the sum. MetricsStore records sequential and parallel runs, gather calls, concurrency, and speedup while the FastAPI dashboard polls /metrics. POST /demo runs both modes so every counter leaves zero.

Highlights

• Simulated async tools via asyncio.sleep with structured results.

• Sequential runner for baseline wall-clock latency.

• Parallel runner using asyncio.gather for overlapped tool I/O.

• Live dashboard for speedup, concurrency, and recent tool events.

• Docker lifecycle on port 8092 via start.sh, demo.sh, and run_tests.sh.

What We Build

• pipeline/ package: tools.py, runner.py, metrics.py, service.py.

• FastAPI (app.py): /run/sequential, /run/parallel, /demo, /metrics, /health, /dashboard.

• CLI demo (main.py) printing sequential vs parallel timings and speedup.

• Default five-tool workload: search, calculate, summarise, lookup, validate.

Connection to Previous Lesson

Day 12 made agent lifecycle edges explicit with an audited FSM. Day 13 keeps that control plane and overlaps tool I/O inside TOOL_CALL/PROCESSING.

Enables Next Lesson

Parallel tool fan-out grows context quickly. Day 14 manages context windows so gathered results stay inside token budgets.

Architecture Context

• Where this component sits: between the agent planner and tool adapters.

• Why it exists: serial tool awaits waste wall time when tools have no data dependency.

• Problem solved: measurable speedup with operator-visible sequential vs parallel counters.

Read more

This lesson models an agent as an explicit finite-state machine with audited transitions. Legal edges move through IDLE, VALIDATING, PROCESSING, TOOL_CALL, RESPONDING, DONE, and ERROR. Illegal shortcuts raise errors and increment blocked counters. MetricsStore records transitions, recoveries, and history; the FastAPI dashboard polls /metrics. POST /demo runs happy path, error recovery, and an invalid edge so counters leave zero.

Highlights

• Explicit TRANSITIONS map encodes allowed agent lifecycle edges.

• Error path recovers ERROR → IDLE with rising error_count and recovery metrics.

• Invalid transitions are blocked and surfaced as BLOCKED events.

• Live dashboard shows transitions, states visited, recoveries, and throughput.

• Docker lifecycle on port 8091 via start.sh, demo.sh, and run_tests.sh.

What We Build

• fsm/ package: states.py, machine.py, metrics.py, service.py.

• FastAPI (app.py): /transition, /scenario, /demo, /metrics, /health, /dashboard.

• CLI demo (main.py) printing happy-path, error-recovery, and blocked edges.

• DEMO_SCENARIOS covering tool-call loops and injection-style failure recovery.

Connection to Previous Lesson

Day 11 produced typed answers and entities. Day 12 advances only when validation and tool outcomes are well-defined, not free-form strings.

Enables Next Lesson

A correct FSM becomes the skeleton for Day 13’s async pipeline: each state can await I/O without losing auditability or recovery semantics.

Architecture Context

• Where this component sits: between validated structured output and tool/response handlers.

• Why it exists: agents without explicit states hide loops, retries, and failures in ad-hoc flags.

• Problem solved: deterministic lifecycle control with measurable transitions and operator-visible history.

Read more

This lesson turns raw model text into typed, validated objects with Pydantic v2 schemas. Entity and StructuredResponse enforce allowed entity types, confidence bounds, and optional follow-ups. parse_structured() uses model_validate_json so invalid JSON and schema violations fail loudly. MetricsStore tracks OK/FAIL parses, entities, follow-ups, and latency; the FastAPI dashboard polls /metrics. POST /demo runs five mixed payloads so every counter leaves zero after one click.

Highlights

• Pydantic v2 field_validator clamps confidence and rejects invalid entity types.

• model_validate_json replaces deprecated parse_raw paths.

• Demo mixes three valid and two invalid payloads for full validation coverage.

• Live dashboard shows parse OK/FAIL, entities, follow-ups, and throughput.

• Docker lifecycle on port 8090 via start.sh, demo.sh, and run_tests.sh.

What We Build

• structured/ package: models.py, parser.py, metrics.py, service.py.

• FastAPI (app.py): /parse, /demo, /metrics, /health, /dashboard.

• CLI demo (main.py) printing OK/FAIL outcomes for each payload.

• DEMO_RAW_PAYLOADS covering location, org/date, follow-up, bad type, and non-JSON.

Connection to Previous Lesson

Day 10 assembled prompts and routed categories. Day 11 assumes that text may be JSON and proves it against a contract before any downstream agent trusts fields.

Enables Next Lesson

Validated structures become reliable inputs for Day 12’s agent state machine: transitions consume typed answers and entities, not free-form strings.

Architecture Context

• Where this component sits: between raw LLM/text ingress and business handlers that need typed fields.

• Why it exists: production systems cannot treat model prose as trustworthy data.

• Problem solved: schema enforcement with measurable OK/FAIL rates and operator-visible events.

Read more

This lesson builds a support-message classifier using prompt templates, few-shot examples, and keyword routing — without calling an LLM. PromptTemplate renders {{placeholders}}; FEW_SHOT_EXAMPLES inject labeled pairs into every prompt; classify() routes messages to billing, technical, general, or urgent. MetricsStore records renders, categories, token estimates, and recent events; a FastAPI dashboard polls /metrics. POST /demo runs five messages so every counter moves from zero after the first click.

The project in aiam-day10/ runs via start.sh, demo.sh, and run_tests.sh.

Highlights

• Mustache-style {{key}} template rendering with unresolved-variable detection.

• Few-shot example formatting for consistent classifier prompts.

• Deterministic keyword routing — fast, testable, no API cost.

• Dashboard auto-refreshes; /demo exercises all four categories.

• Docker lifecycle scripts on port 8089.

What We Build

• prompt/ package: template.py, classify.py, metrics.py, service.py.

• FastAPI (app.py): /classify, /demo, /metrics, /health, /dashboard.

• CLI demo (main.py) printing inputs, categories, and token estimates.

• DEMO_MESSAGES — billing, technical, general, urgent — so every dashboard counter updates.

Connection to Previous Lesson

Day 9 sequences tool calls with ReAct thoughts. Day 10 shapes the prompts that would drive that planner: templates structure instructions, few-shot pairs set behavior, routing picks the right handler path.

Enables Next Lesson

Prompt engineering prepares structured output. Production swaps classify() for an LLM call; the template, few-shot injection, and metrics contract stay the same.

Architecture Context

• Where this component sits: between app.py (API) and prompt/classify.py (routing logic).

• Why it exists: consistent, auditable prompts before any model invocation.

• Problem solved: repeatable prompt assembly with measurable category distribution.

Integration

Integration Components

app.py validates ClassifyBody and calls PromptService.render_and_classify(). The service formats few-shot examples, renders SUPPORT_CLASSIFIER_TEMPLATE, runs classify(), and records metrics. /demo loops DEMO_MESSAGES then marks demo_runs.

Module Objectives Alignment

Prove template correctness, isolate routing from rendering, and expose per-category counters so operators confirm all paths executed.

Core Concepts

Templates separate prompt structure from runtime data. PromptTemplate.render(**kwargs) replaces {{examples}} and {{message}}; leftover placeholders raise ValueError — catching typos before production.

Few-shot examples prime classification behavior. Four labeled pairs (billing, technical, general, urgent) format into the examples block; the model (or router) sees consistent Message: / Category: pairs.

Keyword routing maps substrings to categories: "charge" → billing, "500" → technical, "urgent" → urgent. First match wins; default is general. Deterministic routing enables unit tests without API keys.

Token estimation uses len(prompt) // 4 — a rough budget check before sending to a paid endpoint.

Production AI System Design Relevance

Prompt engineering sits before every LLM call. Day 9’s ReAct planner will consume prompts built this way. Templates version in Git; few-shot sets A/B test in staging; routing pre-filters before expensive inference.

Workflow

• Request flow: POST /classify (one message) or /demo (five messages); Pydantic validates length.

• Execution flow: format examples → render template → classify → record metrics.

• Data flow: each classification becomes a recent_events entry (max 20); aggregates update category counts and timing.

• State changes: prompts_rendered, classifications, per-category counts, demo_runs, token averages advance.

Production Integration

• Architecture fit: stateless API; persist prompt versions externally.

• Enterprise patterns: template registry, few-shot versioning, category allowlists.

• Scalability: single uvicorn worker for in-memory metrics; externalize at scale.

• Observability: export category distribution, render latency, token estimates.

• Security: authenticate /classify; no secrets in templates; 500-char message cap.

Step-by-Step Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering-p/tree/main/lesson10/aiam-day10

template.py defines PromptTemplate and format_few_shot_examples(). classify.py holds CLASSIFICATION_RULES and classify(). metrics.py locks counters and returns deep-copy snapshots. service.py owns DEMO_MESSAGES and run_demo(). app.py serves the dashboard with auto-refresh.

Coding Highlights

Template rendering with validation:

for key, value in kwargs.items():
    text = text.replace("{{" + key + "}}", str(value))
if "{{" in text:
    raise ValueError(f"unresolved template variables: {missing}")

Few-shot formatting:

return "\n".join(
    f"Message: {message}\nCategory: {category}\n"
    for message, category in examples
)

Unresolved placeholders caught at render time prevent silent prompt corruption in production.

Working Demo Link:

Production Considerations

• Scalability: externalize metrics and prompt versions beyond one replica.

• Security: no API keys in repo; .env gitignored; message length capped.

• Monitoring: alert on skewed category distribution or zero classifications.

• Testing: test_prompt.py for template/routing; test_api.py for HTTP and metrics.

• Failure handling: unresolved template vars raise; unknown keywords default to general.

Validation

• Verification: /health green; /demo completes five classifications; dashboard serves.

• Testing strategy: 9 pytest cases across template, classify, service, and API.

• Success criteria: tests pass; after demo, all category counters and demo_runs are non-zero.

• Expected outputs: charge → billing; API 500 → technical; phone → general; URGENT down → urgent.

• Benchmarks: sub-second demo; ~73 tokens per rendered prompt.

• Production checklist: single replica, auth on /classify, alert on zero category coverage.

Key Insights

Prompt quality depends on template structure and few-shot diversity, not length alone. Demo messages must hit every category — a zero counter signals a broken path. Common mistakes: missing placeholder validation, stale few-shot examples, routing rules that overlap.

Important Classes

• PromptTemplate — renders and estimates tokens.

• MetricsStore — thread-safe counters and recent events.

• PromptService — API runs and demo orchestration.

Key Methods

• PromptTemplate.render() / estimate_tokens() — build and size prompts.

• classify() — keyword routing to category.

• MetricsStore.record_classification() / snapshot() — update and read dashboard state.

Success Criteria

aiam-day10/ runs from start.sh and demo.sh alone. All tests pass. Dashboard metrics non-zero after demo. Cleanup removes containers and artifacts.

Real-World Examples

• A support triage system renders few-shot prompts per queue, routes billing keywords to a payment specialist agent, and logs category distribution for SLA dashboards.

• A content moderation pipeline templates policy instructions, injects labeled violation examples, and pre-routes urgent keywords to human review before LLM scoring.

Integrated production console for model landscape selection, secure LLM API access, structured output extraction, and autonomous agent deliberation. This project unifies four core engineering domains into a single deployable system with a unified dashboard, REST API, activity logging, and Docker packaging.

What this project covers

This platform is the capstone integration for the Model Landscape & Prompting track. It combines concepts and patterns from four standalone reference implementations:

Dashboard tabs

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    React Dashboard (:3300)                   │
│  Overview │ Catalog │ Gateway │ Structured │ Autonomy        │
└──────────────────────────┬──────────────────────────────────┘
                           │ REST / WebSocket proxy
┌──────────────────────────▼──────────────────────────────────┐
│                   FastAPI Backend (:8300)                    │
│  catalog/ │ gateway/ │ structured/ │ autonomy/               │
└─────────────────────────────────────────────────────────────┘

Stack: Python 3.12 · FastAPI · Pydantic v2 · google-genai · Vite · React 19 · Recharts · Docker (nginx + slim images)

Read more

This lesson implements Reason+Act (ReAct): an agent alternates thoughts with tool calls until it answers or hits a step limit. Unlike one-shot prompting, every thought, action, and observation is visible for debugging. Three tools — search, calculate, lookup — sit behind a planner in react/loop.py; MetricsStore records each step; a FastAPI dashboard shows live counters. POST /demo runs three questions so metrics move from zero after the first click.

The project in aiam-day09/ runs via start.sh, demo.sh, and run_tests.sh.

Highlights

• Thought → Action → Observation with stopping on final_answer or max_steps.

• Tool dispatch via Action: name[argument] parsed by strict regex.

• AST-safe calculator — no eval().

• Dashboard polls /metrics; /demo exercises all three tools.

• Docker lifecycle scripts on port 8088.

What We Build

• react/ package: tools.py, loop.py, metrics.py, service.py.

• FastAPI (app.py): /run, /demo, /metrics, /health, /dashboard.

• CLI demo (main.py) printing steps, observations, and final answers.

• DEMO_QUESTIONS — search, calculate, lookup — so every dashboard counter updates.

Connection to Previous Lesson

Day 8 fanned typed work to workers in parallel. Day 9 sequences one tool per step: observe, then continue or stop with a Final Answer.

Enables Next Lesson

ReAct is the control loop prompt engineering will steer. Production swaps _plan_thought() for an LLM; the action/observation contract and metrics stay the same.

Architecture Context

• Where this component sits: between app.py (API) and react/tools.py (tool backends).

• Why it exists: multi-step questions need grounded facts before a final answer.

• Problem solved: bounded, observable tool use with measurable loop health.

Read more

This lesson shows how a supervisor fans typed work to specialized workers in parallel. Single-agent pipelines waste wall time when research, writing, and summarization can overlap. A shared dispatch path records completions, timing, and speedup for operators. A FastAPI dashboard makes those signals visible after each demo.

Highlights

• Supervisor/worker fan-out with ThreadPoolExecutor.

• Typed tasks: research, write, and summarise.

• Parallel wall-clock vs serial estimate speedup metrics.

• Live dashboard for dispatch, completion, and recent task events.

What We Build

• An orchestration package: workers, supervisor, metrics store, and service layer.

• HTTP APIs for /dispatch, /demo, /metrics, /health, and /dashboard.

• CLI demo that runs six parallel tasks across three worker specialties.

• Docker image, compose file, and lifecycle scripts for run, test, and cleanup.

Connection to Previous Lesson

Day 7 sandboxed untrusted code so agents could execute snippets safely. Day 8 assumes that guardrail and focuses on coordinating multiple agents without serializing every step.

Enables Next Lesson

Parallel dispatch is the substrate for a ReAct loop: the planner chooses tools, workers execute in parallel where safe, and the loop merges results before the next thought step.

Architecture Context

• Where this component sits: between the planner/API gateway and specialized worker agents.

• Why it exists: multi-step agent work stalls if every specialty runs one after another.

• Problem solved: concurrent typed dispatch with observable speedup and task outcomes.

Read more

This lesson isolates untrusted Python before it can touch the host. AI agents often emit runnable snippets; executing them raw is unsafe. A child process, blocked imports, and timeouts form a minimal sandbox. Outcomes feed a live dashboard so operators see OK, blocked, timeout, and error rates. The demo proves every metric moves when real cases run.

Highlights

• Subprocess isolation for each submitted snippet.

• Dangerous imports (os, subprocess, socket, and peers) denied at load time.

• Hard timeouts that kill infinite loops without hanging the API worker.

• FastAPI dashboard with demo-driven execution metrics.

What We Build

• A sandbox package: executor, metrics store, and service layer.

• HTTP APIs for /execute, /demo, /metrics, /health, and /dashboard.

• CLI demo covering safe, blocked, timeout, and syntax-error paths.

• Docker image, compose file, and lifecycle scripts for runs and tests.

Connection to Previous Lesson

Day 6 made latency and errors visible. Those signals baseline sandbox pressure—timeouts and blocks should appear as operational events, not silent hangs.

Enables Next Lesson

Safe local execution is a prerequisite for multi-agent orchestration, where agents exchange code without sharing one privileged process.

Architecture Context

• Where this component sits: between the agent/tool planner and the host OS, as an execution step or sidecar.

• Why it exists: model-generated code may probe files, spawn shells, or open sockets.

• Problem solved: untrusted snippets run under constraints while remaining measurable.

Integration

Integration Components

FastAPI validates payloads with Pydantic and calls SandboxService. The service runs SandboxExecutor in a child process, classifies the result, and records it in MetricsStore. The dashboard polls /metrics; /demo seeds every outcome class. Docker runs one uvicorn worker so in-memory metrics stay coherent.

Module Objectives Alignment

Contain blast radius, fail closed on dangerous imports and runaway loops, and expose counters for policy tuning.

Core Concepts

Subprocess isolation keeps faults out of the API process. A crash or busy loop in user code should not corrupt the gateway event loop. Temporary source is written, executed with a cleaned environment, then deleted.

Import policy uses a deny-list of dangerous roots. Strict allow-only lists break when CPython pulls helpers such as keyword for collections/json. Denying os, subprocess, and socket preserves useful stdlib while blocking common escapes.

Timeouts convert hangs into a deterministic TIMEOUT outcome. Metrics separate OK, blocked, timeout, and error so operators see whether policy or capacity is failing.

Production AI System Design Relevance

This is the execution guardrail of a production AI stack. It belongs next to tool runners and code-act agents, after planning and before durable side effects. Day 6 observability wraps it; Day 8 multi-agent workflows consume it as a shared capability.

Workflow

• Request flow: client posts code to /execute or triggers /demo; FastAPI validates fields and timeout bounds.

• Execution flow: executor wraps code with the import guard, spawns python, waits up to the limit, and classifies output.

• Data flow: results update counters and a bounded recent-events list; /metrics returns a snapshot for the UI.

• State changes: totals increment, rates recalculate, demo_runs advances, and event history rotates.

Production Integration

• Architecture fit: prefer a dedicated executor with CPU/memory cgroups over the public API pod.

• Enterprise patterns: per-tenant policies, signed job envelopes, artifact-only returns.

• Scalability: queue across executor workers; centralize metrics when replicas exceed one.

• Observability: export outcome counters, timeout rate, and duration histograms.

• Security: drop capabilities, read-only rootfs, default-deny network, never mount Docker.sock.

Step-by-Step Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering/tree/main/lesson7/aiam-day07

The executor owns isolation so the API stays thin. The service maps cases to executor calls and annotates import hits for the dashboard. Metrics stay behind a lock and return copied snapshots. Validation rejects empty or oversized payloads. Syntax failures record as ERROR, not timeouts. Production adds cgroups and network policy beyond this demo.

Coding Highlights

Deny dangerous roots while preserving normal stdlib loads:

if root in _blocked:
    raise ImportError("Blocked: " + name)

This fails closed on escape modules without breaking transitive imports.

Timeouts become structured outcomes:

except subprocess.TimeoutExpired:
    return SandboxResult(ok=False, output="TIMED OUT", timeout=True, ...)

Operators alert on timeout=True instead of guessing from missing responses.

Working Demo Link:

Production Considerations

• Scalability: queue jobs; bound concurrency per tenant; store histograms centrally.

• Security: pair import denial with OS isolation; a deny-list alone is not a jail.

• Monitoring / logging: log outcome, duration, and request id—not full untrusted source.

• Testing: unit-test OK/block/timeout/syntax; API-test that demo metrics are non-zero.

• Failure handling: always unlink temp files; surface executor crashes as errors.

• Edge cases: relative imports, empty output, nested blocked imports.

Validation

• Verification: health green; demo populates every counter; dashboard HTML serves.

• Testing strategy: executor unit tests plus FastAPI contract and metric-mutation tests.

• Success criteria: all tests pass; after demo, primary metrics are non-zero.

• Expected outputs: OK for arithmetic/math/json/collections; blocked OS/subprocess/socket; timeout on infinite loops; error on syntax.

• Benchmarks: blocked imports finish in tens of milliseconds; timeouts match configured limits.

• Production checklist: resource limits set, network default-deny, secrets absent from images, alerts on timeout rate.

Key Insights

Isolation without measurement invites silent failure. Import denial needs OS controls. Common mistakes: brittle allow-lists, sharing one interpreter with the API, and averaging away timeout tails.

Important Classes

• SandboxExecutor — child-process runner with import guard and timeout.

• MetricsStore — thread-safe counters and recent events.

• SandboxService — demo cases and API-facing execution.

Key Methods

• SandboxExecutor.run() — isolate, execute, classify.

• MetricsStore.record() / snapshot() — mutate and read dashboard state.

• SandboxService.run_demo() — deterministic coverage of all outcomes.

Real-World Examples

• A coding agent executes proposed PR snippets in ephemeral executors; blocked subprocess calls increment policy metrics and never reach CI hosts.

• An analytics copilot evaluates user formulas server-side; timeouts protect shared capacity when a customer submits an infinite loop, while other tenants keep being served.

This lesson adds observability to an AI service. Requests emit JSON and contribute latency measurements. RED metrics turn runtime behavior into operational signals. A dashboard presents percentiles, throughput, failures, routes, and events. Demo traffic proves telemetry changes when work executes.

Highlights

• JSON logs with route, status, latency, and request IDs.

• Thread-safe RED metrics with p50, p95, and p99 latency.

• FastAPI dashboard with route counters and recent events.

• Docker execution, health checks, demo traffic, and tests.

What We Build

• An observability package containing logging, metrics, and service modules.

• A latency histogram and synchronized in-memory metrics store.

• Health, metrics, simulation, demo, and dashboard endpoints.

• Docker lifecycle scripts and unit/API validation.

Connection to Previous Lesson

The previous lesson protected model output with scanning, redaction, and integrity checks. This lesson makes that service measurable and its reliability visible.

Enables Next Lesson

Error and latency signals establish the baseline for sandboxed code execution, making new resource pressure and failures visible.

Architecture Context

• Where this component sits: at the HTTP request boundary, surrounding gateway and AI workload execution.

• Why it exists: production failures include gradual latency growth and intermittent upstream errors, not only crashes.

• Problem solved: opaque request handling becomes searchable events and quantitative service-health signals.

Integration

Integration Components

FastAPI delegates workload generation to ObservabilityService, with Pydantic validating input. The service writes each outcome to stdout and MetricsStore. The dashboard polls /metrics; /demo produces healthy, slow, and failed samples. Docker packages one reproducible worker.

Module Objectives Alignment

The implementation measures behavior, latency distribution, and error rate. Tests verify JSON fields, percentile ordering, APIs, dashboard availability, and demo metrics.

Core Concepts

Structured logging produces one JSON document per event, allowing direct field indexing. Route, status, latency, and request ID support correlation. Payloads are excluded because they may contain private data.

Latency averages hide long-tail behavior. Nearest-rank calculation sorts observations and selects the requested percentage rank. Production systems use bounded histogram buckets to control memory.

RED means Rate, Errors, and Duration: demand, reliability, and user-visible performance. Increased throughput is unhealthy when error rate and p99 also rise.

The lock keeps counters, histograms, and events coherent. Snapshot copies prevent partial UI reads. Multiple workers require centralized aggregation.

Production AI System Design Relevance

Observability belongs beside gateway middleware, measuring AI stages consistently. Its signals support SLOs, capacity planning, incidents, and deployments.

Workflow

• Request flow: a caller opens the dashboard or invokes /simulate or /demo; FastAPI validates input and calls the service.

• Execution flow: the service selects a route profile, derives latency and status, assigns severity, and creates a request ID.

• Data flow: metadata goes to stdout and the metrics store; /metrics returns snapshots rendered by the browser.

• State changes: counters increment, latency enters the histogram, percentiles recalculate, and recent events rotate.

Production Integration

• Architecture fit: move timing and classification into gateway middleware.

• Enterprise deployment: export through OpenTelemetry or Prometheus and ship stdout logs centrally.

• Scalability: aggregate bounded histograms across replicas; never average computed percentiles.

• Observability: alert on SLO burn rate, sustained errors, and tail latency.

• Security: never log prompts, completions, authorization headers, or tenant secrets.

Step-by-Step Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering/tree/main/lesson6/aiam-day06

JSONFormatter converts records into stable fields. Logger setup removes prior handlers to prevent duplicates. Histogram owns percentile math; MetricsStore owns synchronization and history. ObservabilityService sends identical classifications to logs and metrics.

FastAPI exposes health, telemetry, simulation, demo, and dashboard routes. The browser consumes snapshots. Demo traffic includes known slow and failed samples, then healthy events. Pydantic rejects malformed input; tests cover histograms, formatting, APIs, and mutation.

Coding Highlights

Nearest-rank percentiles are deterministic:

rank = max(1, math.ceil((p / 100) * len(values)))
return sorted(values)[rank - 1]

Severity reflects both failure and slowness:

level = "ERROR" if status >= 400 else "WARNING" if latency >= 500 else "INFO"

This separates slow success from outright failure, preserving operational meaning.

Working Demo Link:

Production Considerations

• Scalability: bound histogram memory and label cardinality.

• Security: redact sensitive fields and protect dashboard access.

• Monitoring/logging: retain correlation IDs and alert on sustained SLO violations.

• Testing: verify boundaries, formatter output, APIs, and demo metrics.

• Failure handling: telemetry failure must not break user requests.

• Edge cases: handle empty samples, concurrency, malformed labels, and downstream timeouts.

Validation

• Verification: inspect JSON, call health and metrics endpoints, and compare pre/post-demo snapshots.

• Testing strategy: unit tests cover math and formatting; API tests cover contracts and state.

• Success criteria: all tests pass and dashboard values refresh after execution.

• Expected outputs: ordered percentiles, non-zero RED metrics, and per-route totals.

• Benchmarks: metric recording should remain sub-millisecond locally.

• Production checklist: health green, labels bounded, secrets absent, retention defined, SLO load test passed.

Key Insights

Averages conceal tail pain; percentiles expose it. Structured fields improve investigations but increase cardinality. Common mistakes include logging payloads, unbounded labels, averaging percentiles, and letting telemetry failures affect users.

Important Classes

• JSONFormatter renders machine-readable events.

• Histogram calculates latency percentiles.

• MetricsStore synchronizes RED metrics.

• ObservabilityService coordinates execution and telemetry.

Key Methods

• percentile() computes p50, p95, and p99.

• record_request() updates counters, latency, and events.

• run_demo() generates deterministic observable traffic.

Real-World Examples

• A model gateway compares provider p95 latency and error rate, shifting traffic away from degradation before exhausting its SLO.

• A retrieval pipeline correlates request IDs across search, reranking, and generation to isolate latency regressions.

This lesson adds an output-side security layer to the AI service. It scans generated text for leaked secrets and PII before responses leave the system, and signs payloads with HMAC so integrity can be verified downstream. Findings are redacted, counted, and exposed through a live dashboard. The result is a self-contained service that enforces rather than observes.

Highlights

• Regex-based detection of secrets (API keys, tokens) and PII (email, phone, card).

• Deterministic redaction with per-category counters.

• HMAC-SHA256 signing and constant-time verification.

• FastAPI dashboard with non-zero, demo-driven metrics.

What We Build

• A security domain package (hardening, metrics, service).

• An OutputGuard that scans and redacts text.

• A SignatureVerifier for HMAC signing and tamper detection.

• API surface (/scan, /sign, /verify, /metrics, /demo, /dashboard, /health).

• Unit and API tests plus Docker lifecycle scripts.

Connection to Previous Lesson

The previous lesson enforced throughput and cost limits before model execution. This lesson extends enforcement to the output boundary, guarding what the system emits rather than what it accepts.

Enables Next Lesson

Clean, signed, redacted output is a precondition for trustworthy telemetry, enabling the observability stack in the next lesson to reason over safe data.

Architecture Context

• Where it sits: at the response egress, between model output and the caller.

• Why it exists: models can echo secrets or personal data present in prompts, context, or tool results.

• Problem solved: prevents data exfiltration and provides integrity guarantees on payloads crossing trust boundaries.

Integration

Integration Components

FastAPI validates payloads with Pydantic and delegates to SecurityService. The service runs the guard and signer, records outcomes in a shared in-memory metrics store, and returns a decision. The dashboard polls /metrics and triggers /demo.

Module Objectives Alignment

The module targets three production objectives: prevent sensitive-data leakage, guarantee payload integrity, and make both measurable for operators.

Core Concepts

Output scanning is pattern-driven because secrets and PII share stable syntactic shapes. Compiling patterns once and applying them per request keeps latency low while covering multiple categories in a single pass. Detection and redaction are separated conceptually but computed together: detection drives metrics and block decisions, redaction produces a safe-to-emit string.

HMAC signing solves a different problem — integrity, not confidentiality. A shared secret produces a signature that a holder can verify but an attacker cannot forge without the key. Verification uses constant-time comparison to avoid timing side channels.

A deliberate design choice treats an expected rejection (a tampered payload correctly failing verification) as a success signal, while an unexpected accept is the true failure. This aligns metrics with security intent rather than raw booleans. State is kept in-process to keep scope small while preserving interfaces that can move to shared stores later.

Production AI System Design Relevance

This lesson is the egress guardrail of an AI platform. It belongs next to response serialization and gateway middleware, where every model output can be inspected, redacted, and attested before delivery.

Workflow

• Request flow: caller submits text to scan or a payload to sign/verify.

• Execution flow: the service runs guard or signer, then records an event.

• Data flow: validated input enters the service, mutates in-memory metrics, and returns a decision; the dashboard reads snapshots via /metrics.

• State changes: scan/block counters increment, secret/PII tallies grow, HMAC counters update, and a bounded recent-events list is refreshed.

Production Integration

• Architecture fit: embed as egress middleware in API workers.

• Enterprise patterns: centralize pattern sets and signing keys per tenant or environment.

• Scalability: stateless scanning scales horizontally; shared counters move to Redis when multi-worker.

• Observability: counters, rates, and an event timeline feed dashboards and alerts.

• Security: secrets come from environment variables; comparisons are constant-time.

Step-by-Step Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering/tree/main/lesson5/aiam-day05

The guard compiles patterns at construction. Each scan collects findings, computes category counts, and builds a redacted copy. The service wraps guard and signer, timing each call and writing an event to the metrics store. The API layer validates inputs, exposes endpoints, and serves the dashboard. Errors surface as validation failures at the edge. The demo exercises clean text, every detector, and both HMAC paths so no metric stays zero.

Coding Highlights

Constant-time verification prevents timing attacks:

def verify(self, payload: str, signature: str) -> bool:
    return hmac.compare_digest(self.sign(payload), signature)

Intent-aware status keeps metrics meaningful — a caught tamper is a win, not a failure:

status = "tamper_caught" if expect_reject else "hmac_fail"

This matters because naive boolean logging flags correct security behavior as an error.

Working Demo Link:

Production Considerations

• Scalability: externalize counters for multi-worker deployments.

• Security: never log raw findings; store only categories.

• Monitoring/logging: track block rate and HMAC failure rate.

• Testing: cover each detector and both verification paths.

• Failure handling: fail closed on scan errors.

• Edge cases: overlapping patterns and unicode payloads.

Validation

• Verification: unit tests assert detection, redaction, and HMAC correctness.

• Testing strategy: API tests confirm endpoints and non-zero demo metrics.

• Success criteria: all tests pass; dashboard values update on demo.

• Expected outputs: blocked secrets/PII, valid/tampered HMAC results.

• Benchmarks: sub-millisecond scans on short text.

• Checklist: secrets from env, redaction on, tests green, health OK.

Key Insights

Redaction and detection must ship together; detection without safe output is incomplete. A common mistake is scanning too late or logging the secrets being caught. The tradeoff is coverage versus false positives.

Important Classes

• OutputGuard — scanning and redaction.

• SignatureVerifier — HMAC signing and verification.

• SecurityService — orchestration and metrics.

Key Methods

• scan() — findings, counts, redacted text.

• sign() / verify() — integrity operations.

• run_demo() — deterministic metric population.

Real-World Examples

• Chatbot egress filter: a support assistant redacts customer emails and card numbers from replies before display.

• Webhook integrity: a deploy service signs payloads with HMAC so receivers reject forged requests.

This lesson adds production controls for AI request throughput and token spend.
You implement tier-aware throttling before expensive model execution happens.
You also track per-user cost in real time so usage aligns with budget policies.
A live dashboard exposes request, throttle, latency, and budget metrics.
The child project remains self-contained and runnable with local scripts.

Highlights

• Token-bucket throttling with standard, premium, and admin tiers

• Per-request input/output token accounting converted to USD spend

• Budget-alert counters and event timeline for operational visibility

• FastAPI endpoints for health, checks, metrics, dashboard, and demo

• Dockerized runtime with script-driven lifecycle management

What We Build

• Rate-control domain package (bucket, cost_tracker, metrics, service)

• API and dashboard surface (/check, /metrics, /demo, /dashboard, /health)

• Deterministic demo scenario for non-zero metric updates

• Unit and API tests covering behavior and contracts

• Operational scripts for start, stop, test, and cleanup

Connection to Previous Lesson

The previous lesson centered on memory behavior and metric observability.
This lesson shifts from observation to enforcement by adding policy controls.
Now the system does not just report usage; it actively regulates it.

Enables Next Lesson

These controls reduce blast radius and cost drift, enabling stronger policy and security hardening in the next lesson.

Architecture Context

• Where this component sits

• Why it exists

• Problem solved

This component sits between request intake and model execution, enforcing throughput and spend constraints before model work begins.

Integration

Integration Components

FastAPI validates request payloads and forwards policy work to the service layer. The service coordinates bucket checks, spend recording, and metric updates. The dashboard polls /metrics and triggers /demo for validation.

Module Objectives Alignment

The module aligns with production goals: stable throughput, controlled cost growth, and measurable system behavior.

Core Concepts

Token bucket is chosen because it allows controlled bursts while limiting sustained traffic. That fits real AI workloads better than fixed-window throttling, which can create boundary spikes.

Tier configuration maps product plans to infrastructure policy. Higher tiers get larger capacity and refill rates without requiring separate code paths.

Cost tracking is done per request to avoid delayed budget surprises. Real-time accounting enables early alerts and tighter operational response.

Metrics are modeled as counters, gauges, and event entries. This supports both dashboard rendering and automated smoke verification.

In-memory state keeps scope simple while preserving interfaces that can move to shared stores later.

Key Insights

Throughput control and cost control must ship together. Request count alone cannot represent AI cost risk, and spend-only controls cannot protect latency under burst load.

A common mistake is applying checks too late in the pipeline. Guardrails should run before model invocation.

Another mistake is demos that do not force state transitions. If metrics stay flat, defects can hide in integration paths.

Production AI System Design Relevance

This lesson acts as the governance layer of an AI system. It protects model-serving capacity, reduces tenant contention, and keeps economics visible to platform teams.

In the broader architecture, it belongs near API ingress, gateway middleware, or request orchestration boundaries.

Workflow

Explain:

• request flow

• execution flow

• data flow

• state changes

Request flow: client submits user, tier, and token metadata.
Execution flow: service checks tokens, records spend for allowed requests, updates metrics/events, returns policy decision.
Data flow: validated payload enters service, state mutates in-memory, dashboard fetches snapshots via /metrics.
State changes: bucket tokens deplete/refill, spend accumulates, remaining budget drops, alert counters increment.

Production Integration

Explain:

• production architecture fit

• enterprise deployment patterns

• scalability

• observability

• security considerations

Architecture fit: reusable middleware or embedded policy module in API workers.
Enterprise pattern: central tier/budget policy config per tenant or environment.
Scalability: externalize counters and budgets to Redis or durable stores for multi-replica consistency.
Observability: export metrics/events to Prometheus or OpenTelemetry and alert on burn-rate and throttle spikes.
Security: strict schema validation, bounded token values, and controlled admin policy updates.

Important Classes

• TokenBucket

• RateLimiter

• CostTracker

• MetricsStore

• RateLimitService

Key Methods

• TokenBucket.consume()

• RateLimiter.check()

• CostTracker.record()

• MetricsStore.record()

• RateLimitService.check_request()

• RateLimitService.run_demo()

Step-by-Step Implementation

GitHub Link:

https://github.com/sysdr/production-ai-engineering/tree/main/lesson4/aiam-day04

First, implement throttling primitives with deterministic behavior (TokenBucket, RateLimiter). This isolates rate math and makes tests precise.

Next, add CostTracker to convert token usage into spend at request time. This creates immediate economics visibility.

Then, centralize observability with MetricsStore. One write path prevents metric drift and simplifies debugging.

After that, compose all controls in RateLimitService, which executes the canonical transaction: throttle, account, emit metrics, return decision.

Expose FastAPI endpoints with explicit schema constraints for tier and token bounds. Validation prevents malformed policy inputs from corrupting state.

Add deterministic demo operations that guarantee non-zero metric movement. This is critical for dashboard checks and CI smoke tests.

Finally, wire scripts and Docker so runtime is fully independent and operational without external scaffolding.

Coding Highlights

Service-layer orchestration keeps endpoint handlers thin and testable. Policy maps avoid hard-coded branches, and event emission adds incident context beyond raw counters.

Working Demo Link:

Production Considerations

• Scalability: move shared state to distributed storage

• Security: lock down policy mutation and validate all inputs

• Monitoring: alert on throttle ratio, burn rate, and latency drift

• Logging: structured decision logs for audit and triage

• Testing: unit + API + script-level smoke coverage

• Failure handling: safe degradation when telemetry dependencies fail

• Edge cases: refill timing drift, burst storms, mis-tiered traffic

Validation

Verification uses burst-limit tests, refill timing assertions, and cost-math checks. Success means healthy endpoints, deterministic throttling, accurate spend, and non-zero metrics after demo. Expected outputs are rising counters, populated events, latency values, and budget alerts under stress. Production checklist: self-contained child scripts and reproducible startup/cleanup.

Real-World Examples

1. Enterprise support copilot: premium tenants get higher refill rates, while budget alerts prevent monthly overrun.

2. Internal developer assistant: build bots are rate-limited and token-accounted to protect shared model capacity.

This lesson adds production conversational memory: sliding-window short-term storage, a TTL semantic cache, and optional Redis. Agents that forget waste tokens; agents that remember forever raise cost and serve stale answers. You implement bounded session history, cache hit/miss paths, TTL eviction, and a FastAPI dashboard for retained, reused, and discarded state.

Highlights

• Sliding-window short-term memory with turn and token budgets.

• Exact-match semantic cache with LRU and TTL eviction.

• Redis persistence with in-memory fallback.

• Live dashboard for hits, misses, evictions, Redis ops, and latency.

• Docker Compose using official Python and Redis images.

What We Build

• ShortTermMemory and SemanticCache core stores.

• RedisStore and MemoryService coordinating persistence and metrics.

• FastAPI: /health, /metrics, /memory/*, /cache/*, /demo, /dashboard.

• CLI demo and pytest coverage for memory and API behavior.

• Scripts: start, demo, test, stop, cleanup.

Connection to Previous Lesson

Lesson 2 delivered a permission-gated tool boundary. Lesson 3 sits above it: memory decides what context survives and which answers can be reused without another model call.

Enables Next Lesson

Bounded memory enables rate limiting and cost control. Hit rates and token estimates feed Day 4 spend throttling.

Architecture Context

• Where this component sits — Between the agent turn loop and durable storage; after tool/model results, before next prompt assembly.

• Why it exists — Context and reuse must be explicit policies, not unbounded lists.

• Problem solved — Prevents context overflow, duplicate LLM calls, and lost session state across restarts when Redis is available.

Component Architecture

ShortTermMemory and SemanticCache own local state; RedisStore wraps persistence; MemoryService orchestrates both and feeds METRICS; app.py exposes HTTP and the UI.

Integration

Integration Components

• memory/short_term.py, semantic_cache.py, redis_store.py, service.py, metrics.py

• app.py API/dashboard; main.py CLI; Compose and lifecycle scripts

Module Objectives Alignment

• Context discipline, cache economics, persistence, observability, deployability.

Core Concepts

Short-term memory keeps a per-session deque capped by turns and estimated tokens, dropping oldest pairs when budgets break—the tradeoff between recall and prompt cost. Semantic cache keys normalized queries; hits avoid recomputation, but TTL keeps answers from outliving truth. Redis is the shared durable layer for multi-instance deployments; when unreachable, the service degrades to in-memory so demos and tests still run. Metrics treat store, lookup, eviction, and Redis ops as first-class events because memory bugs rarely crash—they quietly raise cost or serve stale data.

Key Insights

• Memory is a cost and correctness control, not a convenience buffer.

• Unbounded history becomes a production incident.

• Cache without TTL creates silent staleness; TTL without metrics hides thrash.

• In-process metrics suit single-worker lessons; replicas need shared stores.

• Fail open on Redis for availability; surface connection state on /health.

Production AI System Design Relevance

Production agents need a memory plane that bounds context, caches expensive answers, and survives restarts. This lesson is that plane for later rate limiters and multi-agent workflows.

Workflow

• request flow — API/CLI calls MemoryService for store, history, cache lookup/set, or demo.

• execution flow — Update local structures, optionally mirror to Redis, record latency and outcomes.

• data flow — Turns and cache values move through service methods; dashboard polls /metrics.

• state changes — Deques grow/trim; cache entries expire; Redis keys set with TTL; counters update.

Control Flow

Request → service method → local update → optional Redis → metrics → response.

Production Integration

• production architecture fit — Shared library beside the agent runtime; Redis as multi-pod fabric.

• enterprise deployment patterns — Compose/K8s, official images, env REDIS_URL, health probes.

• scalability — App replicas require Redis for shared sessions/cache.

• observability — Hit rate, evictions, Redis ops, latency, event timelines.

• security considerations — No secrets in repo; private Redis; sanitize session IDs and payloads.

Data Flow

Store/lookup APIs accept turns and cache values; metrics snapshots drive the dashboard; demo exercises window trim, hit/miss, and TTL eviction.

Important Classes

Turn, ShortTermMemory, SemanticCache, RedisStore, MemoryService, MetricsStore.

Key Methods

store(), get(), cache_lookup(), cache_store(), run_demo(), record(), snapshot().

Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering/tree/main/lesson3/aiam-day03

Bounded session memory → TTL cache → Redis with safe fallback → MemoryService + metrics → FastAPI/dashboard → Compose packaging. Validate with unit tests, /demo non-zero metrics, and Redis-connected health. Redis failures degrade without crashing the API.

Coding Highlights

while turns and self._token_estimate(turns) > self.max_tokens:
    turns = turns[2:]  # drop oldest pair

Token budget trimming prevents prompt blowups before the model is called.

if entry and (time.time() - entry["ts"]) < self.ttl:
    self.hits += 1
    return entry["value"]

TTL-checked hits are the economic win; expired keys become misses and evictions.

METRICS.record(op="cache_lookup", cache_hit=hit is not None, redis_ops=redis_ops)

Normalized metrics make memory behavior operable, not anecdotal.

Working Demo Link:

Production Considerations

• scalability: shared Redis across replicas.

• security: private Redis, no API keys, bounded payloads.

• monitoring: hit rate, evictions, Redis connectivity, latency.

• logging: structured op labels without secrets.

• testing: window limits, hit/miss, TTL, demo metric assertions.

• failure handling: Redis offline → local memory; health reports status.

• edge cases: empty sessions, case-normalized keys, LRU at max size.

Validation

• verification: pytest; Compose health; /demo smoke checks.

• testing strategy: unit tests without Redis; integration with Redis connected.

• success criteria: start.sh / demo.sh / run_tests.sh work from the child project alone; post-demo metrics non-zero.

• expected outputs: hits/misses, TTL evictions, Redis ops when connected, live /dashboard events.

• benchmarks: hit rate and latency_ms_avg after demo > 0.

• production checklist: official images, .gitignore, cleanup script, no parent-setup dependency.

Real-World Examples

Support bots keep the last N turns per ticket in Redis so pod handoffs stay coherent, while FAQ answers cache for minutes to cut duplicate LLM spend. Coding assistants cache identical build-error explanations with short TTL so outdated remediation is not served after dependency changes.

This lesson builds the Layer 3 tool execution boundary for a production AI agent: a schema-validated registry, permission-gated dispatch, and observable outcomes.

You implement a ToolOrchestrator that rejects unknown tools, missing parameters, unsafe input, and unauthorized writes before any side effect runs.

A FastAPI dashboard surfaces live tool calls, denials, blocks, and latency.

Highlights

• Layer 3 tool boundary with typed schemas, permission checks, and explicit error states.

• Single orchestrator dispatch instead of ad hoc Python functions.

• Live metrics for allowed calls, denials, unsafe input, failures, and latency.

• Self-contained Dockerized FastAPI service with dashboard, CLI demo, tests, and cleanup.

What We Build

• ToolOrchestrator validating tool names, parameters, and permissions.

• Registry tools: calculate, echo, write_log.

• FastAPI endpoints: /health, /metrics, /tools/call, /demo, /dashboard.

• Live dashboard for metrics and recent events.

• Docker and scripts for start, demo, test, stop, and cleanup.

Connection to Previous Lesson

Lesson 1 introduced the four-layer secure agent and least-privilege tool discovery. Lesson 2 turns that into execution—validating inputs, checking permissions, dispatching tools, and returning structured results.

Enables Next Lesson

Typed, permissioned, observable tool calls let later lessons add memory, persistence, and workflows without rebuilding safety checks.

Architecture Context

• Where this component sits — Between agent logic and external capabilities; Layer 3 gates existence, permission, validation, and reporting.

• Why it exists — Model actions must not execute directly; the boundary narrows exposure and creates audit points.

• Problem solved — Blocks unknown tools, bad parameters, unsafe expressions, and unauthorized writes before runtime side effects.

Component Architecture

registry.py owns dispatch; metrics.py owns counters; app.py serves HTTP; main.py runs CLI; scripts handle operations.

Integration

Integration Components

• tools/registry.py — schema, registry, orchestrator.

• tools/metrics.py — in-process dashboard metrics.

• app.py — HTTP API and dashboard.

• main.py — CLI demo.

• Dockerfile, docker-compose.yml, shell scripts — deployment workflows.

Module Objectives Alignment

• Tool safety, schema discipline, observability, deployability, and testability.

Core Concepts

The registry is a production allowlist declaring capabilities and requirements. Permission-gated dispatch enforces least privilege—read-only callers cannot write logs. Schema validation returns structured ToolResult errors for dashboards and audits. The calculate sandbox rejects code-execution patterns, showing each tool needs its own validation. Metrics classify ok, denied, blocked, and failed outcomes separately.

Key Insights

• Tool calls are authorization events, not mere function calls.

• Unknown tools fail closed; model output is untrusted.

• Expected denials prove the safety layer works.

• In-memory metrics suit lessons; production needs shared storage for replicas.

• Centralize validation; permission denied, unsafe input, and unknown tools differ operationally.

Production AI System Design Relevance

This is the execution gateway preventing an LLM from becoming an unbounded RPC caller—scaling to databases, tickets, email, payments, and infrastructure actions.

Workflow

• request flow — Tool name, parameters, permissions, and label reach the orchestrator.

• execution flow — Registry lookup, permission check, parameter validation, dispatch, ToolResult wrap.

• data flow — Input through ToolOrchestrator.call(); metrics receive normalized events.

• state changes — Static registry; METRICS counters and events update at runtime.

Control Flow

Request → registry → permission → validation → execute → metrics → response.

Production Integration

• production architecture fit — Behind auth, before side effects; emits traces to shared observability.

• enterprise deployment patterns — Containers, pinned images, env config, API gateway; externalize metrics for replicas.

• scalability — Horizontal dispatch when metrics and audit logs are externalized.

• observability — Structured logs, counters, latency histograms, durable audit events.

• security considerations — Server-side identity, rate limits, approval for high-impact tools.

Real-World Examples

Support agents summarize tickets for all users; only supervisors issue refunds via permission-gated billing tools. Cloud assistants read health freely; restart and migration tools require elevated permissions and audit logging.

Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering/tree/main/lesson2/aiam-day02

Data Flow

JSON/CLI input passes validation, returns ToolResult; metrics get normalized events.

Important Classes

ToolSchema, ToolResult, ToolOrchestrator, MetricsStore.

Key Methods

available(), call(), record(), run_demo().

Step-by-Step Implementation

Registry allowlist → result envelope → orchestrator → tool handlers → metrics → FastAPI → dashboard → Docker packaging.

Coding Highlights

if tool_name not in TOOL_REGISTRY:
    return ToolResult(tool=tool_name, success=False, error="Unknown tool")

if schema.required_permission not in permissions:
    return ToolResult(tool=tool_name, success=False, error="Permission denied")

METRICS.record(tool=tool_name, success=True, latency_ms=ms, output=output)

Working Demo Link:

Production Considerations

• scalability: externalize metrics for replicas.

• security: server-side identity, not request permissions.

• monitoring, logging, testing, failure handling, edge cases: timeouts, input caps, idempotent retries only.

Validation

• verification: orchestrator tests, API tests, Docker build, health checks.

• success criteria: start.sh runs dashboard, demo.sh runs CLI, run_tests.sh passes, cleanup.sh stops services.

• expected outputs: non-zero post-demo metrics, timestamped events, structured expected failures.

• production checklist: no secrets, official Python image, clear failure classification, health and metrics endpoints.

This lesson introduces the foundation of a production AI agent: a four-layer secure architecture where every request must pass through Security, Tools, Memory, and finally the LLM. You will build a self-contained Day 1 system with typed contracts, fail-closed injection defense, permissioned tools, and bounded session memory.

A FastAPI dashboard exposes live metrics—requests, blocks, tokens, cost, and layer latency—so observability is learned alongside the agent itself.

The project runs offline in stub mode or with a real OpenAI key, and ships with Docker scripts for start, demo, test, and cleanup. By the end, you have a reusable agent kernel that later lessons extend without collapsing security into ad-hoc prompts.

Highlights

• Build a production-shaped agent with four concentric layers: Security (L4) → Tools (L3) → Memory (L2) → LLM (L1).

• Enforce prompt-injection defense, RBAC, and rate limiting before any model call.

• Emit live metrics (requests, blocks, tokens, cost, layer latency) to a FastAPI dashboard.

• Run the full stack with an official python:3.12-slim Docker image via self-contained start.sh / demo.sh.

• Treat stub LLM mode as a first-class engineering path so CI and demos work without API keys.

What We Build

• A SecureAgent orchestrator that routes every request through L4 → L3 → L2 → L1 and filters responses on the way out.

• Layer modules: SecurityPerimeter, ToolOrchestrator, InMemoryStore, CoreLLM.

• A FastAPI service (app.py) exposing /health, /metrics, /chat, /demo, and a live /dashboard.

• Pytest coverage for injection blocking and metrics side effects, plus Dockerized start/demo/test scripts.

• Cost and latency accounting that stays non-zero in stub mode so the dashboard teaches observability habits.

Enables Next Lesson

Day 2 builds on the L3 tool boundary: real tool execution, schema validation, and safer tool-output handling. After Day 1 you already have permission-filtered tool discovery and output redaction — Day 2 turns that registry into an execution pipeline without rewriting security or memory.

Architecture Context

Where this component sits — At the edge of a production AI product: every user (or service) request enters through the agent perimeter before reaching a model provider. The dashboard and metrics sit beside the agent as the first observability surface.

Why it exists — Prompt-only “safety” is insufficient. Production systems need perimeter controls (length, sanitize, injection, authz, rate limit), capability gating for tools, session memory with bounds, and a narrowly-scoped model call with cost/latency telemetry.

Problem solved — Prevents untrusted input from reaching L1; prevents over-privileged tools from appearing; bounds memory growth; makes failures observable (blocked vs rejected vs ok) instead of silent prompt failures.

Integration

Integration Components

Module Objectives Alignment

• Security-first agent design — L4 is mandatory, not optional middleware.

• Separation of concerns — tools, memory, and LLM are distinct replaceable layers.

• Operability — metrics and dashboard are part of the build, not a later add-on.

• Testability — stub LLM + injection tests make Day 1 CI-complete without secrets.

Core Concepts

Concentric layers, not a pipeline of peers

Engineering reasoning: if security is “one step among equals,” a future refactor can bypass it. Placing security as the outermost gate makes SecureAgent.run structurally unable to call the LLM without passing L4.

Prompt injection as an authorization failure

Injection is not a “bad reply” quality issue; it is an attempt to elevate privilege over the system prompt and tools. Raising PermissionError maps it to the same class of failure as unauthorized access — metrics label it blocked, not ok.

Least-privilege tools

Tool registries without permission checks leak capabilities into the prompt surface. Filtering available tools by SecurityContext.permissions shrinks the attack path even when L1 is imperfect.

Bounded session memory

Unbounded chat history becomes a cost and privacy bomb. A sliding window (max_turns) is the minimal production discipline before durable stores appear in later lessons.

Stub LLM as a design choice

Production CI should not depend on vendor availability. A deterministic stub with estimated tokens/cost teaches the same observability contracts as a live model while keeping demos offline-capable.

In-process metrics with a single worker

For Day 1, a threaded MetricsStore plus uvicorn --workers 1 gives correct live dashboard updates without Redis. The tradeoff is intentional and documented — multi-worker needs a shared metrics backend later.

Key Insights

• Security before creativity — tokens spent on injected prompts are both wasted money and risk; block early.

• Status taxonomy matters — ok / blocked / rejected drives ops dashboards and alerting differently than HTTP 200 with a polite refusal string.

• Tradeoff: regex injection filters — fast and explainable, but incomplete against novel jailbreaks; combine with model-side policy and human review queues in production.

• Tradeoff: in-memory session store — great latency, poor durability and multi-replica stickiness; fine for Day 1, not for multi-AZ production alone.

• Common mistake — putting API keys in source or committing .env. Use .env.example with empty values only.

• Common mistake — multi-worker uvicorn without shared metrics, then wondering why the dashboard stays at zero.

• Common mistake — treating HTML escape as XSS-only concern; escaped user text also reduces markup-based prompt smuggling in logs and UIs.

Production AI System Design Relevance

Inside a production AI platform, Day 1 is the agent control plane kernel:

Clients → API Gateway → Secure Agent (L4–L1) → Model Provider / Tools
                         ↓
                   Metrics / Dashboard / Logs

Later modules hang off this kernel: durable memory, tool sandboxes, retrieval, evaluation, canary models, and policy engines all assume a clean security perimeter and typed request/response contracts exist.

Workflow

Request flow

1. Client sends prompt + identity context (user_id, session_id, permissions).

2. SecureAgent wraps input as AgentRequest.

3. L4 validates/sanitizes or raises.

4. L3 computes available tools for permissions.

5. L2 loads session history.

6. L1 produces AgentResponse.

7. L2 stores the turn; L3 filters output; metrics record outcome.

Execution flow

• Happy path: L4 pass → L3 list tools → L2 get → L1 call → L2 store → L3 filter → return.

• Injection/jailbreak: L4 raises PermissionError → metrics blocked → API returns blocked payload (no L1).

• Oversize input: L4 raises ValueError → metrics rejected.

Data flow

• Inbound: raw string → sanitized string → message list (history + user).

• Outbound: model/stub text → redacted text → JSON/API/dashboard event.

• Telemetry: tokens, cost, layer timings accumulate in MetricsStore.

State changes

• Session memory list grows by two messages per successful turn, then trims to window.

• Rate-limiter counters increment per user/minute bucket.

• Metrics counters (requests_*, tokens_total, cost_usd_total, layer sums) monotonically increase.

• Demo runs bump demo_runs after /demo.

Production Integration

Production architecture fit

Deploy the agent as a single-responsibility service behind an API gateway. Keep L1 provider credentials in a secret manager; never in the image. Use the dashboard only for low-traffic demos or internal ops — production metrics should also export to Prometheus/OpenTelemetry.

Enterprise deployment patterns

• Containerize with the official Python slim image (already in Dockerfile).

• One service replica for Day 1 parity with in-memory metrics; later: sticky sessions or external memory + shared metrics.

• Separate “demo” and “prod” configs via env (OPENAI_MODEL, rate-limit tiers).

Scalability

Vertical: increase CPU for regex + stub latency. Horizontal: requires externalizing memory and metrics. Tool and LLM calls dominate p95 — measure with layer_timings.

Observability

Structured logs per layer ([L4], [L3], [L2], [L1], [AGENT]), request IDs on SecurityContext, live /metrics + dashboard. Production should add distributed tracing spans around each layer.

Security considerations

Perimeter injection filters, RBAC permissions on tools, output redaction of secret-like patterns, rate limits, no committed secrets, HTML escaping of user input.

Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering/tree/main/lesson1/aiam-day01

This section is the canonical implementation guide for Day 1. No separate implementation guide file is used.

Component Architecture

aiam-day01/
├── agent/
│   ├── agent.py                 # SecureAgent orchestrator
│   ├── core/types.py            # Contracts
│   ├── core/metrics.py          # MetricsStore
│   └── layers/
│       ├── security.py          # L4
│       ├── tool_orchestrator.py # L3
│       ├── memory.py            # L2
│       └── llm.py               # L1
├── app.py                       # FastAPI + dashboard
├── main.py                      # CLI demo
├── tests/
├── Dockerfile / docker-compose.yml
├── start.sh / stop.sh / demo.sh / run_tests.sh / cleanup.sh
└── requirements.txt

Step-by-Step Implementation

1. Define contracts (types.py) — Pydantic models force consistent fields for metrics and APIs; prevents “dict soup” across layers.

2. Implement L4 — compile injection patterns once; sanitize; rate limit; RBAC log/assert. Exists to fail closed before cost is incurred.

3. Implement L3 — registry with permission tags; filter tools; redact secrets on egress.

4. Implement L2 — session-keyed lists with trim; exists for continuity without unbounded growth.

5. Implement L1 — branch on OPENAI_API_KEY; stub still estimates cost; small sleep ensures measurable latency for demos.

6. Orchestrate — SecureAgent owns ordering, timings, and metrics on success and exception paths.

7. Expose HTTP — FastAPI lifespan constructs one agent; /demo walks happy + attack prompts; dashboard polls /metrics.

8. Validate — unit tests for L4 and metrics; run_tests.sh for health/demo/non-zero metrics.

9. Package — Dockerfile installs requirements.txt, runs single-worker uvicorn; start.sh avoids duplicate containers.

Communication: layers do not call each other sideways; only SecureAgent composes them. Validation: Pydantic on the way in; regex/RBAC in L4; permission checks in L3. Error handling: PermissionError → blocked; ValueError → rejected; both recorded. Production considerations: env-based secrets, official base image, cleanup script for caches/images, .gitignore for .env and venv.

Coding Highlights

Layer order is policy-as-code — do not reorder casually:

request = self.security.process(request)          # L4 first
available = self.tools.get_available_tools(...)   # L3
history = self.memory.get_context(session_id)     # L2
response = self.llm.call(request, messages)       # L1 last
response = self.tools.filter_output(response)     # egress filter

Why it matters: every future tool or model swap still inherits the same security envelope.

Fail closed on injection:

if pattern.search(sanitised):
    raise PermissionError("Potential prompt injection detected")

Why it matters: returning a soft “I can’t help with that” from the LLM still spends tokens and can be inconsistently bypassed; raising stops the pipeline.

Record metrics on exception paths, not only success:

except PermissionError:
    METRICS.record(status="blocked", ...)
    raise

Why it matters: security events are the highest-value dashboard series.

Working demo Link:

Production Considerations

• Scalability — externalize memory and metrics before multi-replica; keep L1 timeouts and circuit breakers (Day 2+).

• Security — empty default API key; .gitignore .env; output redaction; rate limits; never log raw secrets.

• Monitoring — /metrics counters + layer timings; alert on requests_blocked spikes.

• Logging — structured layer prefixes; include request_id.

• Testing — injection unit tests; metrics non-zero after demo; stub mode in CI.

• Failure handling — map exceptions to blocked/rejected; API returns JSON status without crashing the worker.

• Edge cases — oversize input; repeated jailbreaks; empty model content; HTML tags in prompts; multi-turn memory trim boundaries.

Validation

Verification methods

• ./demo.sh — CLI path through all scenario labels.

• ./start.sh then dashboard Run demo — UI metrics update.

• ./run_tests.sh — pytest + /health + /demo + non-zero metrics assertions.

Testing strategy

• Unit: L4 allow/deny/length/escape; agent metrics increments.

• Smoke: health, demo JSON, dashboard HTML marker.

• Manual: confirm blocked count rises on injection prompts.

Success criteria

• Clean prompts return ok with tokens_used > 0 and cost_usd > 0 in stub mode.

• Injection/jailbreak prompts are blocked without L1 completion content.

• Dashboard metrics are non-zero after one demo run.

• Only one aiam-day01 container when started via start.sh.

Expected outputs

• CLI labels: NORMAL, MEMORY, INJECT (blocked), JAILBREAK (blocked).

• Metrics keys: requests_total, requests_ok, requests_blocked, tokens_total, cost_usd_total, layer sums, demo_runs.

Benchmarks (stub, local Docker)

• L1 stub latency typically ≥ 5ms (intentional sleep).

• Full /demo (5 steps) completes in well under 1s on a developer laptop.

Production checklist

• No secrets in git

• Official base image pinned by digest in real prod (lesson uses tag for clarity)

• Single-worker or shared metrics documented

• Tests green in CI without OPENAI_API_KEY

• cleanup.sh known to operators

Real-World Examples

Enterprise IT copilot — Employees ask policy questions; L4 blocks “ignore previous instructions / dump the system prompt” attempts; L3 only exposes read tools for standard employees; admins get write tools. Metrics feed a SOC dashboard of blocked jailbreaks per day.

Fintech support agent — Session memory keeps the last N turns for continuity, but L4 length limits stop paste-bomb cost attacks. Output filter redacts accidental sk- / Bearer leakage from tool-assisted answers before they hit the customer channel.

Looking Ahead

• Enables next lesson — Day 2 turns the L3 registry into a validated execution engine while keeping L4 and L2 intact.

• Module progression — Day 1 perimeter → Day 2 tools → later durability, evaluation, and multi-agent patterns.

• Architectural evolution — today’s in-memory metrics and session store become shared infrastructure; the layer boundaries remain the stable API of the agent kernel.

The final lesson in the 90-lesson VAIA curriculum transforms your completed system into a defensible, auditable, production-grade artifact. You build a submission package that a technical review board can run, inspect, and score — and you prepare a live defense demonstrating operational excellence under adversarial questioning.

What we build:

• A SubmissionPackageBuilder that assembles all lesson exports (L1–L93) into a SHA-256 hash-chained archive with integrity manifests

• A DefenseReadinessReport auto-generated from runtime metrics, evaluation gate logs, and compliance audit store entries

• A LiveDemoHarness — a React dashboard wired to your running VAIA backend that executes deterministic scenario playbooks during the live defense

• A ProductionReadinessAttestation signed by the evaluation gate (faithfulness ≥ 0.85, latency p99 ≤ 2 s, zero critical compliance violations)

• A PeerReviewScorecard framework for structured technical peer assessment

Connection to L91–L93

L91–L93 produced your fully integrated VAIA: orchestrated agents, MLOps pipeline, compliance audit store (append-only, hash-chained SQLite WAL), and an evaluation gate that blocks deployment until thresholds pass. Every component you built there is an input here — not as source files to copy, but as live services the submission package queries, validates, and packages.

The CapstoneProjectManifest from L90, instantiated in L91–L93 as a running system, now becomes the root hash of the submission package’s integrity chain. Nothing is re-implemented; everything is verified and documented.

Architecture Context

This lesson sits at the apex of the dependency graph spanning all 90 lessons. The submission package is not a zip of source files — it is a reproducible artifact: a Docker Compose stack that, when started on a clean machine, boots the full VAIA, runs the integration test suite, regenerates the DefenseReadinessReport, and serves the LiveDemoHarness on port 3000.

The defense itself follows a three-phase FSM: PREPARATION → LIVE_DEMO → ADVERSARIAL_QA. State transitions are gated by the peer review panel’s explicit approval signals, logged to the audit store, and included in the final submission archive.

What’s Next

The Vertical AI Agents course is complete — 95 lessons, 6 modules, from API fundamentals to enterprise multi-modal deployment.

The Hands On AI Agent Mastery 30-day course starts Tuesday, August 4.

Days 1-7 are free. Day 1 drops August 4.

Curriculum preview →https://aiamastery.substack.com/p/production-ai-engineering-building

Subscribe to get Day 1 → https://aiamastery.substack.com/subscribe

Read more

Three focused sprint lessons converting the L90 CapstoneProjectManifest into a running production system. By the end of L93 you will have shipped:

• VAIAOrchestrator: a finite-state-machine agent backed by CrewAI, wired to a Gemini 2.0 Flash reasoning core

• RAG Pipeline: ChromaDB vector store with Gemini embedding ingestion, retrieval, and faithfulness gating

• Compliance Audit Store: append-only SQLite WAL with SHA-256 hash-chaining (the L79 pattern applied end-to-end)

• Evaluation Gate: Ragas + Evidently metrics gating every promotion; system self-blocks if faithfulness drops below 0.85

• Live Dashboard: React 18 / Vite frontend consuming WebSocket streams from FastAPI, showing FSM state, metrics, and audit events in real time

Connection to L90

L90 produced a CapstoneProjectManifest — a hash-chained JSON document containing weighted feasibility scores, a chosen topology template, and a RequirementsTraceabilityMatrix linking 89 lessons’ exported components. L91-L93 treats that manifest as a bootstrap contract: setup.sh reads the manifest SHA-256 hash on startup and refuses to proceed if the chain is broken. Every architectural decision made here traces back to a manifest entry.

Enables L94-L95

L94-L95 requires a live, defensible system. Everything built here is instrumented to generate the submission artifact set: audit trail export, metric report, ADR corpus, and a LiveDemoEndpoint the panel can hit during the defense.

Architecture Context

L91-L93 sits at the integration apex of the 90-lesson path. Earlier modules supplied individual components:

The topology chosen from L90’s five templates drives the wiring in this lesson. The reference implementation uses Template 3: RAG-Augmented Reasoning Agent — single orchestrator, retrieval sub-agent, compliance middleware, evaluation gate, observability sidecar.

Read more

What We Build

• An interactive Capstone Scoping Workbench — a React dashboard backed by FastAPI that guides you through structured domain selection, requirements elicitation, architecture pattern matching, and risk qualification, producing a machine-readable CapstoneProjectManifest JSON consumed directly by the L91–L93 development sprint.

• A CapstoneFeasibilityValidator that calls Gemini 2.0 Flash to score your proposed vertical against eight production-readiness criteria (data availability, regulatory burden, latency tolerance, LLM dependency surface, and four more) and outputs a numeric feasibility matrix.

• A CapstoneRequirementsMatrix — a structured JSON artifact that maps every functional requirement to the specific VAIA lesson (L1–L89) whose implementation pattern satisfies it, creating an explicit curriculum-to-capstone traceability index.

• A CapstoneArchitectureBlueprint generator that uses your requirements matrix to recommend one of five validated architecture topologies (single-agent RAG, multi-agent orchestration, hybrid FSM+RAG, compliance-hardened, or streaming pipeline) with component-level justifications.

• A complete setup.sh that consumes the LegalReportRenderer output format from L86 to generate your first scope document draft, closing the curriculum loop.

Connection to L89 — Presenting & Defending Agent Architectures L89 built the ArchitecturePresentationEngine and StakeholderAdaptationLayer — the communication scaffolding for defending technical choices to mixed audiences. L90 consumes those patterns in reverse: the Scoping Workbench uses the same stakeholder taxonomy to produce scope documentation that will survive stakeholder review without revision. The SlideNarrativeGenerator output format from L89 becomes the prose template for the CapstoneScopeDocument‘s executive summary section.

Enables L91–L93 — Capstone Development Every artifact produced in this lesson is a direct build input for the development sprint. The CapstoneProjectManifest carries dependency declarations, component scaffolding targets, and acceptance criteria in a format the L91 scaffolding script reads at boot time to generate the correct directory tree, Docker Compose topology, and CI pipeline.

Architecture Context

Place in the 90-Lesson VAIA Path L90 sits at the curriculum’s inflection point — the last lesson whose primary output is a plan rather than a system. Lessons L1–L89 built individual VAIA components: RAG pipelines (L18–L25), multi-agent orchestration (L46–L60), MLOps (L61–L75), compliance hardening (L79–L82), and vertical case studies in finance, healthcare, and legal (L84–L86). L90’s job is to assemble the minimum viable selection from that catalog into a coherent, scoped, feasible project brief that fits a 3-lesson (L91–L93) sprint.

Integration With Prior Components The scoping workbench explicitly references five prior implementation patterns by import path, not by documentation link. This is intentional: it forces the scope document to stay grounded in what was actually built.

Module Objectives Alignment Module 10’s goal is not to build a new component — it is to demonstrate architectural judgment: the ability to scope a greenfield VAIA project from first principles, apply the curriculum’s patterns appropriately, and produce a plan that a senior engineer could execute without ambiguity.

Core Concepts

1. Scope as Architecture Constraint

The most expensive architectural decisions in a VAIA project are made implicitly during scoping — before a line of code exists. Choosing a vertical (legal, healthcare, financial, logistics) determines your regulatory surface. Choosing your primary user persona determines latency tolerance. Choosing your data sources determines whether you need a RAG pipeline, a fine-tuned adapter, or a hybrid. Each choice eliminates roughly 60% of the architecture space. Scoping is not documentation — it is architecture.

A production VAIA scope document must answer five questions deterministically:

1. Vertical boundary: What domain knowledge is in-scope vs. out-of-scope for the agent?

2. Regulatory surface: Which compliance frameworks apply, and what do they mandate technically?

3. Data topology: What does the agent retrieve, and from where?

4. LLM dependency surface: Which decisions are LLM-generated vs. deterministic?

5. Failure modes: What is the highest-consequence failure, and how does the architecture prevent it?

2. The Feasibility Matrix

Not every VAIA idea is buildable in a 3-lesson sprint. The CapstoneFeasibilityValidator applies a weighted scoring model across eight dimensions:

A weighted score ≥ 0.72 is required to proceed. Gemini 2.0 Flash generates the rationale for each dimension score; a deterministic scorer produces the numeric output. This is the hybrid determinism pattern from L84–L86 applied to meta-level project management.

3. Architecture Topology Selection

Five topology templates emerged across the curriculum:

Template A — Single-Agent RAG: One agent, one ChromaDB collection, one FastAPI backend. Suitable for narrow-domain Q&A with deterministic retrieval. Highest feasibility for a sprint. Reference: L25.

Template B — Multi-Agent Orchestration: Supervisor + specialist agents (2–4), shared message bus, SQLite WAL audit. Suitable for workflows with distinct sub-tasks. Reference: L52–L55.

Template C — Hybrid FSM + RAG: Finite state machine drives agent lifecycle; RAG handles knowledge retrieval at each state. Suitable for regulated workflows with mandatory state sequencing. Reference: L82, L86.

Template D — Compliance-Hardened: Template C + privilege sanitization layer + append-only hash-chained audit store + Gemini-as-judge evaluation loop. Suitable for healthcare, legal, finance verticals. Reference: L79–L86.

Template E — Streaming Pipeline: Async event-driven, Redis pub/sub, WebSocket frontend, low-latency user-facing. Suitable for real-time monitoring or alerting verticals. Reference: L63, L74.

The CapstoneArchitectureBlueprint generator maps your requirements matrix to one of these templates and outputs a component list with explicit lesson references for each component.

4. Traceability as a Design Invariant

The CapstoneRequirementsMatrix is not a checklist — it is a dependency graph. Every functional requirement points to:

• The VAIA lesson that implements it

• The named export from that lesson

• The acceptance criterion that proves it works

• The failure mode if it is omitted

This structure makes the capstone defensible in exactly the way L89 trained: you can answer “why did you choose this component?” with a curriculum reference, a lesson number, and a working code pointer.

Integration

Production Architecture Fit In production VAIA deployments, scoping is a formal gate — typically a written Architecture Decision Record (ADR) reviewed by a principal engineer before any infrastructure is provisioned. The CapstoneScopeDocument follows ADR conventions: context, decision, consequences, and alternatives considered. This is not pedagogical overhead; it is the artifact that prevents expensive mid-sprint pivots.

Enterprise Deployment Patterns Enterprise teams use three scoping anti-patterns that this lesson explicitly trains you to avoid:

1. Capability-first scoping: Starting with “I want to use multi-agent orchestration” rather than “I need to automate this workflow.” The CapstoneFeasibilityValidator blocks this by requiring a use-case description before topology selection is enabled.

2. Optimistic data assumptions: Assuming a clean, labeled corpus exists. The feasibility matrix’s “Data availability” dimension requires you to specify your corpus construction plan at scoping time.

3. Compliance deferral: Treating regulatory requirements as post-MVP. The scope document’s regulatory surface section is mandatory — it cannot be left blank.

Implementation

GitHub Link

https://github.com/sysdr/vertical-ai-agent-p/tree/main/lesson90/l90-capstone-scoping

Component Architecture

┌─────────────────────────────────────────────────────────┐
│                  Capstone Scoping Workbench              │
│                                                         │
│  React Dashboard (Vite + Recharts)                      │
│  ├── DomainSelector                                     │
│  ├── RequirementsElicitor                               │
│  ├── FeasibilityScorecard                               │
│  ├── TopologyRecommender                                │
│  └── ManifestExporter                                   │
│                                                         │
│  FastAPI Backend                                        │
│  ├── CapstoneFeasibilityValidator  ← Gemini 2.0 Flash   │
│  ├── CapstoneRequirementsMatrix                         │
│  ├── CapstoneArchitectureBlueprint                      │
│  ├── CapstoneScopeDocument                              │
│  └── CapstoneProjectManifest (JSON export)              │
│                                                         │
│  Persistence: SQLite WAL (scope versions)               │
│  Integration: L86 LegalReportRenderer format            │
└─────────────────────────────────────────────────────────┘

Control & Data Flow

1. User selects vertical domain and describes the use case in natural language.

2. CapstoneFeasibilityValidator calls Gemini 2.0 Flash with the use-case description + eight scoring rubrics. Gemini returns rationale; a deterministic scorer calculates weighted score.

3. If score ≥ 0.72, CapstoneRequirementsMatrix is generated by mapping the use case to VAIA lesson exports.

4. CapstoneArchitectureBlueprint reads the requirements matrix and selects an architecture topology.

5. CapstoneScopeDocument assembles the executive summary (using L89 prose template), technical plan, and risk register.

6. CapstoneProjectManifest serializes all artifacts to a single JSON file consumed by L91 scaffolding.

State Machine

The scoping workbench is explicitly stateful — users cannot jump to step 4 without completing steps 1–3. This enforces the dependency invariant: architecture selection is downstream of feasibility validation.

IDLE → DOMAIN_DEFINED → USE_CASE_DESCRIBED → FEASIBILITY_SCORED →
REQUIREMENTS_MAPPED → BLUEPRINT_GENERATED → SCOPE_DRAFTED → MANIFEST_EXPORTED

Coding Highlights

CapstoneFeasibilityValidator — Hybrid Determinism Pattern

# backend/capstone_feasibility_validator.py
import google.generativeai as genai
from dataclasses import dataclass
from typing import Dict

WEIGHTS = {
    "data_availability": 0.20,
    "regulatory_clarity": 0.15,
    "latency_tolerance": 0.10,
    "llm_dependency": 0.15,
    "integration_complexity": 0.15,
    "testing_surface": 0.10,
    "curriculum_coverage": 0.10,
    "novelty_risk": 0.05,
}

THRESHOLD = 0.72

@dataclass
class FeasibilityResult:
    scores: Dict[str, float]        # deterministic, 0.0–1.0 per dimension
    rationales: Dict[str, str]      # LLM-generated prose per dimension
    weighted_score: float           # deterministic aggregate
    proceed: bool                   # deterministic gate
    blocking_dimensions: list       # dimensions scoring < 0.5

async def validate_feasibility(use_case: str, domain: str) -> FeasibilityResult:
    model = genai.GenerativeModel("gemini-2.0-flash")

    prompt = f"""You are evaluating a VAIA capstone project proposal.
Domain: {domain}
Use Case: {use_case}

Score each dimension from 0.0 to 1.0 and provide a one-sentence rationale.
Return ONLY valid JSON with keys matching these dimensions:
{list(WEIGHTS.keys())}
Format: {{"dimension": {{"score": 0.0, "rationale": "..."}}}}"""

    response = await model.generate_content_async(prompt)
    raw = response.text.strip().removeprefix("```json").removesuffix("```")
    parsed = json.loads(raw)

    scores = {k: float(parsed[k]["score"]) for k in WEIGHTS}
    rationales = {k: parsed[k]["rationale"] for k in WEIGHTS}
    weighted = sum(scores[k] * WEIGHTS[k] for k in WEIGHTS)

    return FeasibilityResult(
        scores=scores,
        rationales=rationales,
        weighted_score=round(weighted, 4),
        proceed=weighted >= THRESHOLD,
        blocking_dimensions=[k for k, v in scores.items() if v < 0.5],
    )

Key pattern: Gemini generates rationale text; the numeric score is extracted from a structured JSON field and the weighted aggregate is computed deterministically in Python. No LLM involvement in the pass/fail decision. This is the same hybrid determinism applied in L84’s red-teaming scorer and L86’s contract clause extractor.

CapstoneRequirementsMatrix — Curriculum Traceability

# backend/capstone_requirements_matrix.py
LESSON_EXPORT_CATALOG = {
    "RAG pipeline": {"lesson": 25, "export": "AgenticRAGPipeline"},
    "Multi-agent orchestration": {"lesson": 52, "export": "AgentOrchestrator"},
    "Compliance audit store": {"lesson": 79, "export": "GovernanceComplianceOrchestrator"},
    "Bias mitigation": {"lesson": 81, "export": "BiasDetectionMitigationAgent"},
    "LLM evaluation": {"lesson": 60, "export": "MultiAgentSystemEvaluator"},
    "Fine-tuning adapter": {"lesson": 77, "export": "LoRAAdapterTrainer"},
    "Reward alignment": {"lesson": 82, "export": "RewardAlignmentEngine"},
    "Streaming frontend": {"lesson": 74, "export": "StreamingMetricsDashboard"},
}

def build_requirements_matrix(requirements: list[str]) -> dict:
    """Maps each requirement to a lesson export and acceptance criterion."""
    matrix = {}
    for req in requirements:
        for pattern, ref in LESSON_EXPORT_CATALOG.items():
            if pattern.lower() in req.lower():
                matrix[req] = {
                    **ref,
                    "acceptance_criterion": f"Import {ref['export']} from L{ref['lesson']} passes integration test",
                    "failure_mode": f"Missing {ref['export']} causes silent capability degradation"
                }
    return matrix

CapstoneProjectManifest — Machine-Readable Export

# backend/capstone_project_manifest.py
import json, hashlib, datetime

def export_manifest(
    scope: dict,
    requirements_matrix: dict,
    blueprint: dict,
    feasibility: dict
) -> dict:
    payload = {
        "schema_version": "1.0",
        "generated_at": datetime.datetime.utcnow().isoformat(),
        "curriculum_lesson": 90,
        "capstone_vertical": scope["domain"],
        "use_case": scope["use_case"],
        "feasibility": feasibility,
        "requirements": requirements_matrix,
        "architecture_blueprint": blueprint,
        "component_scaffold_targets": [
            v["export"] for v in requirements_matrix.values()
        ],
        "l91_bootstrap_instructions": {
            "topology": blueprint["selected_template"],
            "entry_lesson_imports": list(requirements_matrix.values()),
            "acceptance_criteria": [v["acceptance_criterion"] for v in requirements_matrix.values()],
        }
    }
    payload["manifest_hash"] = hashlib.sha256(
        json.dumps(payload, sort_keys=True).encode()
    ).hexdigest()
    return payload

The manifest_hash field enables the L91 scaffolding script to verify manifest integrity before generating the project directory. Same hash-chain pattern used in L79’s audit store.

Validation

Feasibility Gate: Run pytest tests/test_feasibility.py — asserts that a well-specified use case scores ≥ 0.72 and a deliberately vague use case scores < 0.72.

Requirements Coverage: Run pytest tests/test_requirements_matrix.py — asserts that every requirement maps to a lesson number in range [1, 89] with a non-null export name.

Manifest Integrity: Run pytest tests/test_manifest.py — asserts that manifest_hash matches the re-computed hash of the manifest body (tamper-evidence check from L79 pattern).

Frontend Progression: Cypress E2E test confirms that the “Generate Blueprint” button is disabled until feasibility score is displayed — verifying the state machine’s gate invariant.

Success Criteria:

• Feasibility validator produces a score in [0.0, 1.0] for any valid input

• Requirements matrix covers ≥ 80% of stated functional requirements

• Blueprint selects exactly one topology template with ≥ 3 component justifications

• Manifest passes hash integrity check

• Scope document is ≥ 600 words and includes all five mandatory sections

Assignment

Extend the Feasibility Validator with Domain-Specific Weight Profiles

The default feasibility weight vector is domain-agnostic. For regulated verticals (healthcare, legal, finance), the “regulatory clarity” dimension should carry more weight; for real-time monitoring verticals, “latency tolerance” is more critical.

Implement a DomainWeightProfile that:

1. Takes the selected domain as input

2. Returns a domain-adjusted weight vector (must still sum to 1.0)

3. Passes a parametric test that confirms weight sums to 1.0 for all supported domains

4. Stores the weight profile used in the CapstoneProjectManifest so the L91 scaffolding knows which feasibility model was applied

This directly builds toward L91: the development sprint will reference the manifest’s weight profile to determine which compliance components to scaffold by default.

Solution Hints

• The weight vector must sum to 1.0 after domain adjustment. Use a normalization step: divide each adjusted weight by the sum of all adjusted weights.

• Test with at least three domains: a regulated vertical (e.g., “healthcare”), a real-time vertical (e.g., “infrastructure monitoring”), and a general-purpose vertical (e.g., “customer support”).

• The manifest_hash must be recomputed after weight profile injection, or your integrity check will fail.

• Gemini’s score output is a float in the model’s JSON — validate it against [0.0, 1.0] with a Pydantic validator before passing it to the deterministic scorer to prevent injection of out-of-range values.

Looking Ahead

How This Enables L91–L93 The CapstoneProjectManifest is the single artifact that drives all three development lessons. L91 reads l91_bootstrap_instructions.topology to generate the project directory structure. L92 reads component_scaffold_targets to know which lesson exports to wire together. L93 reads acceptance_criteria to parameterize the final test suite.

Without a validated manifest from L90, the L91 scaffolding script cannot run. This is not a soft dependency — it is a hard boot-time check. This mirrors how production ML systems treat experiment configs: no config, no run.

Module Progress With L90 complete, Module 10 has delivered its first milestone: a validated, machine-readable project plan that bridges 89 lessons of component-building into a coherent final system. L91–L93 will be the proof: every component reference in the manifest is a link back to something you built, and the capstone is the sum of that work.

You have now scoped, designed, and defended a production VAIA system. Build it.

Youtube link

You will engineer a system that blocks prompt injection in under 5ms, compresses context by 60% before the LLM call, detects its own quality regressions and improves automatically, scales on Kubernetes from zero to ten replicas without dropping a request, passes a SOC 2 audit, and handles a regional cloud outage in under 30 minutes — without human intervention.

The five agents you ship in the capstone are portfolio-ready. They demonstrate the layer of AI engineering that enterprise teams actually pay for — and that no LangChain tutorial has ever covered.

30 Lessons. 4 Modules. 5 Production Agents. Starts August 4, 2026.

Explore Lessons 1–7: Secure Agent Foundations. Continue your journey through the complete Production AI Engineering.

What You Build

Throughout this course, you'll build five production-ready AI agents that gradually evolve into a complete enterprise AI platform. You'll engineer secure agent architectures, multi-agent collaboration, real-time streaming, self-improving LLMOps pipelines, and Kubernetes-native deployments with enterprise security, compliance, disaster recovery, and FinOps.

Every project solves a real production challenge using the same architectural patterns found in enterprise AI systems. By the end, you'll have a portfolio that demonstrates not just how to use AI—but how to engineer AI systems that organizations can confidently deploy at scale.

Agent 1 — Secure Customer Service Agent

Every Week 1 control in one request: injection blocked at Layer 4 in <5ms, RBAC enforced at Layer 3, semantic cache at Layer 2 reducing token cost by 40%, output scanned for API keys and PII before delivery.

Agent 2 — Multi-Agent Research System

SupervisorAgent fans out to three parallel WorkerAgent instances via asyncio.gather(). One worker failure does not cancel the others. Total latency = slowest single worker, not the sum.

Agent 3 — Streaming Analytics Agent

StreamingLLM → SSE endpoint → EventSource client. First token under 500ms. The X-Accel-Buffering: no header that makes nginx stop buffering your stream.

Agent 4 — Self-Healing LLMOps Agent

EvalHarness blocks deploys on regression. ABRouter routes 10% of live traffic to a new prompt variant. CircuitBreaker trips after 3 failures and rejects requests in microseconds. PromptOptimiser runs weekly and commits winners to version control.

Agent 5 — Enterprise Kubernetes Deployment

KEDA scales on agent_request_queue_depth, not CPU — before latency degrades, not after. SOC 2 audit trail with chain-hash tamper detection. HIPAA PHI detector blocks protected data at the perimeter. Route 53 failover completes in under 30 minutes. FinOps allocator bills teams for exactly what they consumed.

Who This Course Is For

Backend engineers who have built agents that work in demos and want to understand why they fail at scale.

Platform engineers who need to deploy LLM workloads on Kubernetes with the same operational rigour as any other production service.

Tech leads who own the “we need SOC 2 compliance for our AI system” conversation and need to know what to build.

Founding engineers at AI startups where the next customer due diligence will ask about security architecture, compliance controls, and disaster recovery.

What Makes This Different

No frameworks until you understand the layer below. You implement the security perimeter before touching any orchestration library. You implement the token bucket before reading about rate-limiting middleware.

Real numbers, not analogies. 60% token reduction from context compression. 15 percentage point accuracy gain from multi-agent debate. 67% infrastructure cost reduction from KEDA scale-to-zero. Every number comes from the lesson that demonstrates it.

Failure first. Every module ends with a “what breaks without this” section. You learn the failure mode the lesson prevents before you learn how it prevents it.

Production checklist on every lesson. Not “nice to have” suggestions — the specific checks that get caught in SOC 2 audits, load tests, and security reviews.

Learning Objectives

By Phase 1: You can articulate the four-layer agent security model, explain why tool calls require a five-gate validation pipeline, and deploy a secure agent base with 8-signal observability.

By Phase 2: You can implement parallel multi-agent dispatch, achieve first-token-under-500ms streaming, build layered error recovery with a dead-letter queue, and reduce token costs by 60% through context compression.

By Phase 3: You can build a CI quality gate that blocks deployments on regression, run statistically-valid A/B experiments on live traffic, implement a self-healing circuit breaker, and close the LLMOps feedback loop automatically.

By Phase 4: You can deploy an AI agent on Kubernetes with zero-downtime rolling updates, queue-depth auto-scaling, SOC 2 audit controls, HIPAA compliance safeguards, multi-region disaster recovery, and FinOps cost allocation.

Key Topics

Security Architecture: 4-layer enforcement model, prompt injection detection (regex + structural analysis), RBAC with role inheritance, output scanning for secrets and PHI, sandboxed subprocess execution with import allowlist.

Memory & Cost: Conversation buffer, semantic similarity cache (55% hit rate → 40% cost reduction), vector store for cross-session recall, four-strategy context compressor (60% token reduction), real-time Prometheus cost dashboard.

Reliability Engineering: Token bucket rate limiting, four-class error taxonomy, exponential backoff with jitter, model fallback chain, dead-letter queue, three-state circuit breaker, SLOs with error budget tracking and automated runbooks.

LLMOps: Evaluation harness with CI gate, A/B testing with deterministic routing and statistical significance, prompt optimisation with full audit trail, multi-agent debate (+15pp accuracy), knowledge graph for compliance-queryable relationships, full continuous improvement pipeline.

Enterprise Infrastructure: Kubernetes Deployment + HPA + KEDA (queue-depth scaling, scale-to-zero), SOC 2 immutable audit trail with chain-hash tamper detection, HIPAA PHI detection and 15-minute auto-logoff, multi-region disaster recovery (RTO < 30 min, RPO < 5 min), FinOps cost allocation with team showback and 80%-threshold budget alerts.

Prerequisites

• Python 3.11+

• Comfortable reading and writing async Python (asyncio, await)

• Basic Docker knowledge (build an image, run a container)

• No prior AI/agent experience required — but familiarity with API calls helps

• OpenAI API key optional — every lesson runs in stub mode without one

Course Structure

Phase 1 — Secure Agent Foundations

Lessons 1–7

The architecture that every production agent runs on. By Lesson 7 you have a complete, hardened agent base — security, tools, memory, rate limiting, observability, and sandboxed execution. This is the foundation Phases 2–4 build on without modification.

Lesson 1 — The 4-Layer Secure Agent Architecture

Every production agent enforces the same request path: Security Perimeter → Tool Orchestrator → Memory Manager → Core LLM. A request that fails Layer 4 never reaches Layer 1. You implement all four layers — including stubs for the ones Lessons 2–7 will fill — so the architecture is correct from Day 1.

Core types: SecurityContext, AgentRequest, AgentResponse

Lesson 2 — Tool Execution & Validation

Five gates, in order: registry lookup, Pydantic schema validation, RBAC permission check, sandboxed subprocess execution, output filtering. Miss gate 3 and an unpermissioned caller reaches your database tool. Miss gate 5 and your agent returns API keys in its responses.

Core classes: ToolDefinition, ToolOrchestrator

Lesson 3 — Memory Systems — Short-Term, Semantic Cache & Redis

Three layers with different jobs: ConversationBuffer keeps the last N turns in memory. SemanticCache returns cached responses to near-identical queries using cosine similarity — 55% hit rate in production means 40% cost reduction without touching a single prompt. RedisBackedMemory persists state across restarts and enables cross-session recall.

Core classes: ConversationBuffer, SemanticCache, RedisBackedMemory

Lesson 4 — Rate Limiting & Cost Control

The token bucket algorithm: capacity sets the burst ceiling, refill_rate sets the sustained limit. A request with a projected cost above PER_REQUEST_CAP is rejected before the LLM call. A user who crosses DAILY_BUDGET_USD is blocked until midnight. This is what prevents the $40,000 API bill.

Core classes: Bucket, RateLimiter, CostTracker

Lesson 5 — Security Hardening — Secrets Scanning, HMAC Signing & Output Guard

SecretsScanner detects API keys, tokens, and credentials in both inputs and outputs using pattern matching and entropy analysis. HMACSigner signs every tool call so tampered requests are rejected before execution. OutputGuard scans every response for SSNs, card numbers, and filesystem paths before the response leaves the agent. Core classes: SecretsScanner, HMACSigner, OutputGuard

Lesson 6 — Observability Stack — Metrics, Structured Logs & Distributed Traces

Eight production signals: four golden (traffic, latency, errors, saturation) and four AI-specific (cost per request, cache hit rate, security rejection rate, tool execution latency). Every layer boundary emits a structured JSON log line with request_id. Four Grafana alert rules. One Telemetry class that wires all three pillars.

Core class: Telemetry

Lesson 7 — Sandboxed Code Execution — Isolated, Resource-Capped & Audited

SandboxExecutor spawns a subprocess, enforces RLIMIT_AS memory cap, kills on timeout, applies an import allowlist, and rejects any non-JSON output. SandboxAuditLog writes a tamper-evident record of every execution — args are hashed, never stored raw. An open import list is a filesystem deletion waiting to happen.

Core classes: SandboxPolicy, SandboxExecutor, SandboxAuditLog

Phase 2 — Production Integration

Lessons 8–14

Patterns that make the Phase 1 foundation scale: parallel agents, streaming, layered error recovery, token compression, and production monitoring. By Lesson 14 the agent handles real concurrent load, streams output to the browser, recovers from failures without human intervention, and has a real-time dashboard showing spend per user per minute.

Lesson 8 — Multi-Agent Orchestration — The Supervisor Pattern

SupervisorAgent dispatches to N WorkerAgent instances via asyncio.gather(). Total wall-clock time = slowest worker, not the sum. A failed worker returns a WorkerResult with success=False — the supervisor logs it, excludes it from the response, and returns the rest. One worker going down does not take the request down.

Core classes: WorkerTask, WorkerResult, WorkerAgent, SupervisorAgent

Lesson 9 — ReAct Loop — Reasoning, Acting & Observing

ReActAgent alternates between Thought, Action, and Observation steps until the task is complete or the step budget is exhausted. Each Action calls a real tool; each Observation feeds the result back into the next Thought. The step budget is not optional — an uncapped ReAct loop is an infinite billing event.

Core classes: ReActStep, ReActAgent, StepBudget

Lesson 10 — Prompt Engineering — Templates, Versioning & Injection-Safe Construction

PromptTemplate separates static structure from dynamic inputs, enforces variable validation before assembly, and stores a version hash so every LLM call is traceable to the exact prompt that produced it. Prompt construction is where injection enters — building prompts by string concatenation is the root cause.

Core classes: PromptTemplate, PromptRegistry, PromptVersion

Lesson 11 — Structured Output — Schema Enforcement & Retry-on-Failure

StructuredOutputParser validates every LLM response against a Pydantic schema before it reaches application code. On a parse failure, OutputRetryHandler regenerates with the error message injected into the prompt — up to max_retries attempts. An unvalidated LLM output reaching a database write is a data corruption event, not an edge case.

Core classes: StructuredOutputParser, OutputRetryHandler

Lesson 12 — Agent State Machine — Lifecycle, Transitions & Persistence

AgentStateMachine enforces valid state transitions: IDLE → RUNNING → WAITING_FOR_TOOL → RUNNING → COMPLETE | FAILED. An invalid transition raises immediately — silent state corruption is harder to debug than a loud exception. State snapshots are persisted to Redis so a pod restart does not lose in-progress agent work. Core classes: AgentState, AgentStateMachine, StateSnapshot

Lesson 13 — Async Agent Pipeline — Concurrent Execution & Backpressure

Three sequential tool calls at 1,500ms become three parallel calls at 500ms — a 3× latency reduction with no change to the output. asyncio.gather(return_exceptions=True) ensures one failed tool never cancels the others. BackpressureQueue rejects new requests when the in-flight count exceeds MAX_CONCURRENT — protecting the LLM API from overload.

Core classes: AsyncToolOrchestrator, BackpressureQueue, ToolPlan

Lesson 14 — Context Window Management — Compression, Prioritisation & 60% Token Reduction

Four strategies, applied cheapest-first: strip tool call metadata → remove near-duplicates → truncate system prompt to 800 chars → LLM-summarise the oldest half. Most requests exit after strategy 1 or 2. Strategy 4 (the LLM call) triggers only when the others are insufficient. 60% token reduction at 10k requests/day is a meaningful monthly cost difference.

Core class: ContextCompressor

Phase 3 — LLMOps & Advanced Orchestration

Lessons 15–21

The feedback loop that makes your agent continuously improve without human intervention. By Lesson 21 the agent blocks quality regressions in CI, runs live A/B experiments, heals from failures in milliseconds, improves its own prompt accuracy on a weekly schedule, and uses structured relationship graphs for compliance-queryable audit trails.

Lesson 15 — Evaluation Framework — Automated Quality Gates & CI Integration

EvalCase defines input, must_contain assertions, must_not_contain assertions, and a max_cost_usd cap. EvalHarness runs a suite against a live agent, produces a PASS/FAIL report, and exits with code 1 — blocking the CI merge. A 5 percentage point quality regression is caught in minutes, not days.

Core classes: EvalCase, EvalResult, EvalHarness

Lesson 16 — Vector Store & RAG — Retrieval-Augmented Generation Pipeline

VectorStore indexes documents as dense embeddings and retrieves the top-K most semantically relevant chunks per query. RAGPipeline injects retrieved context into the prompt before the LLM call — grounding responses in real documents rather than parametric memory. Chunk size and overlap are tuned per corpus; wrong values silently destroy retrieval quality.

Core classes: VectorStore, RAGPipeline, ChunkConfig

Lesson 17 — Fine-Tuning Pipeline — LoRA Adapters & Dataset Curation

FineTuningPipeline curates a training dataset from production traces, applies LoRA adapters to reduce GPU memory by 70% vs full fine-tuning, and validates the fine-tuned model against EvalHarness before promotion. A fine-tuned model that regresses on the eval suite never reaches production.

Core classes: TrainingDataset, LoRAConfig, FineTuningPipeline

Lesson 18 — Agent Guardrails — Input Validation, Output Filtering & Policy Enforcement

GuardrailPipeline runs three sequential checks: input policy (topic restrictions, length limits, PII detection), tool call policy (allowed tools per role, parameter bounds), and output policy (factual grounding score, toxicity filter, PII redaction). A request that fails any check is rejected with a structured error — not silently degraded.

Core classes: InputGuardrail, ToolGuardrail, OutputGuardrail, GuardrailPipeline

Lesson 19 — Human-in-the-Loop — Approval Gates, Escalation & Audit Trail

HITLGate pauses execution at configurable decision points and routes the pending action to a human reviewer via webhook. EscalationPolicy defines which tool calls require approval (irreversible writes, high-cost operations, regulated data access). Every approval and rejection is written to the audit log with reviewer identity and timestamp.

Core classes: HITLGate, EscalationPolicy, ReviewRecord

Lesson 20 — Agent Persistence — State Recovery, Checkpointing & Resume

CheckpointManager snapshots full agent state — conversation history, tool results, current step — to durable storage at every state transition. On restart, AgentRecovery replays from the last valid checkpoint rather than from scratch. A checkpoint that is never tested is an assumption, not a guarantee.

Core classes: Checkpoint, CheckpointManager, AgentRecovery

Lesson 21 — Cost Optimisation — Model Routing, Caching & Budget Enforcement

CostRouter classifies each query by complexity and routes it to the cheapest model that can satisfy the quality threshold — local SLM for simple queries, frontier model for complex ones. BudgetEnforcer tracks spend per user, per team, and per agent type in real time and hard-blocks requests that would breach the daily cap.

Core classes: CostRouter, ComplexityClassifier, BudgetEnforcer

Phase 4 — Enterprise Deployment & Capstone

Lessons 22–30

Kubernetes, compliance, and five production agents. By Lesson 30 you have a deployment that a security team can audit, a compliance team can certify, and an SRE team can operate — and a portfolio of five agents that demonstrate every technique from Phases 1–3 working together.

Lesson 22 — Streaming Responses — SSE Token Pipeline & Backpressure

StreamingLLM calls the LLM with stream=True and yields tokens as they arrive. The FastAPI route flushes every 5 tokens as an SSE data: frame — single-token flushes are noisy, 20-token flushes feel laggy. X-Accel-Buffering: no prevents nginx from holding the stream. First token under 500ms.

Core class: StreamingLLM

Lesson 23 — Multi-Modal Agent — Vision, Audio & Document Inputs

MultiModalRouter detects input type (text, image, audio, PDF) and dispatches to the correct preprocessing pipeline before the LLM call. VisionTool extracts structured data from images. DocumentParser chunks PDFs with layout awareness — splitting mid-table destroys context. One unified AgentRequest schema regardless of modality.

Core classes: MultiModalRouter, VisionTool, DocumentParser

Lesson 24 — Agent Testing Patterns — Unit, Integration & Load Tests

AgentTestHarness provides deterministic stub LLM responses so unit tests run without an API key and in under 1 second. IntegrationTestSuite spins up the full stack in Docker Compose and validates end-to-end request paths. LoadTestRunner (Locust) measures p95 latency and error rate under 500 concurrent users — the numbers that catch resource leaks before production.

Core classes: AgentTestHarness, IntegrationTestSuite, LoadTestRunner

Lesson 25 — Production Deployment — Docker, Kubernetes & Zero-Downtime Rollouts

maxUnavailable: 0 in the rolling update strategy means zero dropped requests during deploys. runAsNonRoot: true eliminates an entire class of container escape impact. The readiness probe means no traffic arrives until the pod signals ready. The preStop: sleep 5 closes the race condition between SIGTERM and the load balancer’s routing table update. Artifacts: k8s/namespace.yaml, k8s/deployment.yaml, k8s/hpa.yaml, k8s/configmap.yaml

Lesson 26 — Prompt Injection Defense — Detection, Structural Analysis & Runtime Blocking

PromptInjectionDetector runs three layers in under 5ms: regex signatures for known attack patterns, structural analysis for instruction-override attempts, and anomaly scoring against the expected prompt shape. A detected injection is logged with full context and blocked before reaching the LLM — not filtered after.

Core classes: InjectionSignature, StructuralAnalyser, PromptInjectionDetector

Lesson 27 — Agent Versioning — Artifact Tracking, Rollback & Promotion Gates

AgentVersionRegistry stores every deployed agent as an immutable artifact: prompt version, model ID, tool manifest, eval suite hash, and deployment timestamp. PromotionGate runs the eval suite against the candidate version and blocks promotion if pass rate falls below threshold. Rolling back is a single function call — not a manual config edit.

Core classes: AgentVersion, AgentVersionRegistry, PromotionGate

Lesson 28 — Compliance & Audit Log — Immutable Records, Chain-Hash & Evidence Pack

Each ImmutableAuditWriter entry includes a chain_hash — the SHA-256 of the previous entry’s JSON. Modify any entry and the chain breaks. generate_evidence_pack() produces structured output pointing to the actual evidence: S3 bucket ARN, IAM policy document, CloudWatch alert ARNs. Auditors need pointers to real evidence, not prose descriptions.

Core classes: ImmutableAuditWriter · Key function: generate_evidence_pack()

Lesson 29 — Capacity Planning — Load Modelling, Scaling Thresholds & Cost Forecasting

CapacityModel projects required replicas, token budget, and monthly API cost from three inputs: expected RPS, average tokens per request, and target p95 latency. ScalingSimulator replays historical traffic patterns against the model to validate thresholds before they go live. A capacity plan built on assumed traffic numbers fails the first time real traffic arrives.

Core classes: CapacityModel, ScalingSimulator, CostForecast

Lesson 30 — Production Capstone — Five Agents, One Enterprise Platform

All five agents deployed as a single production system. Agent 1 (Secure Customer Service): injection blocked in under 5ms, RBAC enforced, semantic cache active. Agent 2 (Multi-Agent Research): three parallel workers, one failure does not cascade. Agent 3 (Streaming Analytics): first token under 500ms, nginx bypass active. Agent 4 (Self-Healing LLMOps): CircuitBreaker, EvalHarness, and PromptOptimiser running as a closed loop. Agent 5 (Enterprise Deployment): SOC 2 audit trail, HIPAA PHI detection, KEDA queue-depth scaling, Route 53 failover tested and documented. This is the system you put in your portfolio.

Beyond the Curriculum

Most AI tutorials stop once the model generates a response. Production AI Engineering begins where those tutorials end. Throughout this course, you'll learn how to build AI systems that remain secure under attack, scale under heavy traffic, recover from failures automatically, satisfy enterprise compliance requirements, and continuously improve through automated LLMOps pipelines.

Every lesson focuses on solving a real production engineering challenge using proven architectural patterns—not shortcuts or abstractions. By the final capstone, you won't just have five portfolio-ready AI agents—you'll understand how modern enterprise AI platforms are designed, operated, and trusted in production.

Ready to Build Production AI Systems?

If you’ve made it this far, you’ve already seen that this isn’t another “build an AI chatbot” course. It’s a complete roadmap to engineering secure, scalable, and enterprise-ready AI systems from the ground up.

Subscribe to unlock every lesson that demonstrates real Production AI Engineering skills.

Subscribe Now