By the end of this lesson you will have:

• A Runbook Engine — structured, versioned, step-by-step procedures for every repeatable ops task

• An Incident Response System — severity classification, escalation tree, on-call paging, and post-mortem generation

• A Maintenance Scheduler — calendar-driven maintenance windows with approval workflows

• A Backup Procedure Manager — policy engine defining what backs up, when, how, and how restore is validated

• A Monitoring Guide Builder — living documentation for every alert and SLO your system owns

All five live in a single React + FastAPI dashboard, backed by PostgreSQL, with a shared search layer.

Where This Fits in Your System

Read more

• A structured user documentation portal with searchable manuals

• Interactive tutorial system with step-by-step guided flows

• Embedded video tutorial placeholders with transcript support

• Dynamic FAQ engine with categorization and search

• Contextual in-app help system with tooltip overlays and help drawer

This is Week 23 of our series. Last lesson we built the technical backbone — architecture docs, API references, deployment guides. Today we build what the users actually touch: the help experience. Next lesson, Day 157, we cover operational runbooks — the bridge between user-facing docs and internal ops.

Where This Fits in the Overall System

User documentation is the surface layer of your entire platform’s knowledge graph. When users hit friction — they either abandon the product, or they find an answer. The difference between those two outcomes is your help system.

In production infrastructure platforms (think Datadog, Vercel, Render), the help system is not bolted on as an afterthought. It’s deeply integrated: context-aware tooltips know which page you’re on; FAQ searches pull from the same content API as the manuals; video tutorials are tagged to specific UI components.

Read more

By the end of this lesson, you will have a complete technical documentation system for your Infrastructure Management platform. Here’s the agenda:

• Architecture documentation — machine-readable and human-readable system maps

• API reference — auto-generated from code, always in sync

• Deployment guides — reproducible runbook scripts

• Troubleshooting guides — decision-tree format runbooks

• Maintenance procedures — SLA-driven operational playbooks

This week’s target: wrap production-readiness with documentation that acts as both a knowledge base and an operational contract.

Why Technical Documentation Is an Engineering Artifact, Not a Writing Task

Documentation written after the fact is archaeology. The kind that actually gets used is generated from the system — it derives its accuracy from code, config, and telemetry, not from someone’s recollection.

Think about how Stripe, GitHub, and AWS write docs. Their API references are never manually authored. They’re emitted by annotation processors and schema validators running in CI. When the code changes, the docs change. This lesson builds that exact pipeline for your system.

The Architecture: Documentation as a Pipeline

Your documentation system has three layers:

Layer 1 — Source Layer: Python decorators, OpenAPI annotations, and inline docstrings embedded directly in your FastAPI/Flask backend.

Layer 2 — Generation Layer: A docs-build service (MkDocs + mkdocstrings + Redoc) that pulls from source, renders HTML, and packages a versioned artifact.

Layer 3 — Serving Layer: A React dashboard that presents architecture docs, API reference, deployment runbooks, and troubleshooting trees as navigable, searchable content.

The key insight: documentation is a build artifact. It has a version, a build step, and tests that verify it isn’t stale.

Component 1: Architecture Documentation

Architecture docs must answer four questions: What runs where? How do components talk? What are the failure boundaries? What does the data flow look like?

You generate these from two sources — your docker-compose.yml (which declares services and networks) and annotated Python modules (which describe business intent). A parser walks both, builds a JSON graph, and renders it as an interactive SVG + Markdown.

# Pseudo-code: Architecture extractor
def extract_components(compose_path, source_root):
    services = parse_compose(compose_path)       # ports, volumes, depends_on
    annotations = scan_modules(source_root)       # @component, @depends decorators
    return merge(services, annotations)           # unified graph

The output is a architecture.json that feeds both the dashboard visualization and the exported Markdown page.

Component 2: API Reference Documentation

FastAPI emits an OpenAPI 3.1 spec at /openapi.json automatically. The gap is richness — field descriptions, error codes, example payloads, and authentication notes are only present if engineers annotate their route handlers.

Your build pipeline:

1. Hits /openapi.json on the running dev server

2. Validates the spec (no undocumented 200 responses without schemas)

3. Feeds it to Redoc to render a three-panel reference UI

4. Publishes as a versioned HTML page with each release

# Pseudo-code: API doc validator
def validate_openapi(spec):
    for path, methods in spec['paths'].items():
        for method, op in methods.items():
            assert 'description' in op, f"Missing description: {method} {path}"
            assert '200' in op['responses'], f"Missing 200 response: {method} {path}"

Engineers who skip descriptions fail the build. That’s how you keep the spec useful.

Component 3: Deployment Guides

A deployment guide that isn’t executable is a fairy tale. Yours is a structured YAML runbook with embedded shell steps, pre-condition checks, and rollback instructions.

# deployment_runbook.yml structure
runbook:
  name: "Production Deploy v2"
  pre_checks:
    - cmd: "kubectl get nodes"
      expect: "Ready"
  steps:
    - name: "Pull images"
      cmd: "docker-compose pull"
    - name: "Run migrations"
      cmd: "python manage.py migrate --check"
  rollback:
    - cmd: "docker-compose down && git checkout HEAD~1"

The Python runbook runner executes each step, captures stdout/stderr, logs pass/fail, and presents a live execution trace in the React dashboard.

Component 4: Troubleshooting Guides

Troubleshooting docs written as prose are useless at 3 AM. Yours are decision trees stored as JSON — each node is a symptom, each edge is an observation, each leaf is an action.

{
  "symptom": "API returns 503",
  "check": "curl /health",
  "branches": {
    "timeout": { "action": "restart backend container" },
    "connection_refused": { "action": "check postgres is up" },
    "200": { "escalate": "check upstream load balancer" }
  }
}

The React UI renders this as a clickable flowchart. The on-call engineer follows branches interactively — no reading walls of text. Each leaf records which action was taken, building an incident knowledge base over time.

Component 5: Maintenance Procedures

Maintenance procedures encode when and how — not just what. They’re SLA-linked: a procedure for database vacuum runs weekly; certificate rotation runs 30 days before expiry.

Your maintenance scheduler reads maintenance.yml, registers jobs in APScheduler, and logs execution to a PostgreSQL maintenance_log table. The React dashboard shows the next scheduled run, last execution result, and a manual trigger button.

This closes the loop: documentation isn’t just reference material. It’s an active participant in system operations.

How It All Connects in the Infrastructure Management System

[Backend API] → [OpenAPI Spec] → [Redoc] → [API Reference Page]
[docker-compose] → [Architecture Extractor] → [Arch Diagram Page]
[maintenance.yml] → [APScheduler] → [Maintenance Log] → [Ops Dashboard]
[troubleshoot.json] → [React Tree] → [Interactive Runbook]
[deployment.yml] → [Runbook Runner] → [Execution Trace Dashboard]

Every component feeds the React documentation dashboard — a single pane of glass for all technical knowledge about the system.

Key Insight: Documentation Debt Is Infrastructure Debt

When AWS had its famous 2017 S3 outage, one contributing factor was runbooks that didn’t match the actual system topology. Operators followed documented procedures that no longer reflected reality. Generated, validated documentation eliminates an entire class of operational failure.

Implementation & Build Guide

Prerequisites

Tool Minimum Version Check Command Python 3.11+ python3 --version Node.js 20+ node --version npm 10+ npm --version Docker (optional) 24+ docker --version curl any curl --version

Github Link:

https://github.com/sysdr/infrawatch-fullstack-p/tree/main/day155/techdocs

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│              TechDocs Documentation System               │
│                                                         │
│  React Dashboard (3155)                                 │
│  ├── Overview (Health Score Gauge)                      │
│  ├── Architecture (Service Map)                         │
│  ├── API Reference (OpenAPI Browser)                    │
│  ├── Deployment (Runbook Executor)                      │
│  ├── Troubleshoot (Decision Tree)                       │
│  └── Maintenance (Procedure Runner)                     │
│                           │                             │
│  FastAPI Backend (8155)   │                             │
│  ├── /api/docs/architecture   ← generate_architecture() │
│  ├── /api/docs/api-reference  ← validate_openapi()     │
│  ├── /api/docs/runbooks       ← execute_runbook()      │
│  ├── /api/docs/troubleshoot   ← navigate_tree()        │
│  ├── /api/docs/maintenance    ← execute_procedure()    │
│  └── /api/docs/health-score   ← aggregate all          │
│                           │                             │
│  SQLite (techdocs.db)     │                             │
│  ├── maintenance_log                                    │
│  ├── runbook_execution                                  │
│  ├── troubleshoot_session                               │
│  └── architecture_snapshot                             │
└─────────────────────────────────────────────────────────┘

Option A: Run Without Docker (Recommended for Learning)

Step 1 — Clone / Set Up Project

# Make the build script executable
chmod +x start.sh stop.sh

# Run the full build (creates all files, installs deps, runs tests, starts services)
./start.sh

Expected output:

======================================
 Creating Project Structure 
======================================
[OK] ✓ /path/to/techdocs/backend/app/api
[OK] ✓ /path/to/techdocs/backend/app/docs_engine
...
======================================
 Running Tests 
======================================
tests/test_docs_system.py::TestArchitectureEngine::test_generate_architecture_doc_has_services PASSED
tests/test_docs_system.py::TestApiDocsEngine::test_validate_spec_returns_coverage PASSED
...
20 passed in 1.23s

Step 2 — Verify Backend is Running

curl http://localhost:8155/health

Expected output:

{
  "status": "healthy",
  "service": "techdocs",
  "timestamp": "2025-01-15T14:30:00.000Z",
  "version": "1.0.0"
}

Step 3 — Verify Architecture Endpoint

curl http://localhost:8155/api/docs/architecture | python3 -m json.tool | head -30

Expected output:

{
  "architecture": {
    "version": "1.0.0",
    "services_count": 6,
    "connections_count": 6,
    "services": [
      {
        "name": "api-gateway",
        "type": "gateway",
        "port": 8155,
        ...
      }
    ]
  }
}

Step 4 — Verify API Reference Endpoint

curl http://localhost:8155/api/docs/api-reference | python3 -c "
import json, sys
d = json.load(sys.stdin)
print(f'Endpoints: {d[\"reference\"][\"total_endpoints\"]}')
print(f'Coverage: {d[\"validation\"][\"coverage_percentage\"]}%')
"

Expected output:

Endpoints: 10
Coverage: 100.0%

Step 5 — Test Deployment Runbook Execution

curl -s -X POST http://localhost:8155/api/docs/runbooks/production_deploy/execute | python3 -m json.tool

Expected output:

{
  "success": true,
  "runbook": "production_deploy",
  "steps": [
    {"phase": "pre_check", "name": "health_check", "status": "pass", "output": "Health: All services nominal"},
    {"phase": "pre_check", "name": "db_connectivity", "status": "pass", "output": "..."},
    {"phase": "deploy", "name": "create_backup", "status": "success", "output": "..."},
    ...
  ]
}

Step 6 — Test Troubleshooting Tree Navigation

# Step 1: List available trees
curl http://localhost:8155/api/docs/troubleshoot

# Step 2: Get a specific tree
curl http://localhost:8155/api/docs/troubleshoot/api_503

# Step 3: Navigate the decision tree
curl -s -X POST http://localhost:8155/api/docs/troubleshoot/api_503/navigate \
  -H "Content-Type: application/json" \
  -d '["timeout", "container_stopped"]'

Expected final output:

{
  "type": "resolution",
  "action": "Run: docker-compose restart backend",
  "resolution": "Backend container was stopped — restarted successfully",
  "path": ["timeout", "container_stopped"]
}

Step 7 — Test Maintenance Procedure Execution

# List procedures
curl http://localhost:8155/api/docs/maintenance

# Execute log_rotation
curl -s -X POST http://localhost:8155/api/docs/maintenance/log_rotation/execute | python3 -m json.tool

Expected output:

{
  "success": true,
  "procedure": "log_rotation",
  "duration_seconds": 0.05,
  "steps": [
    "[OK] find_old_logs: Found 3 log files for rotation",
    "[OK] compress_logs: Compressed: app.log.1 app.log.2 app.log.3",
    "[OK] verify_retention: Retention policy verified: 30 days"
  ],
  "log_id": 1
}

Step 8 — Check Documentation Health Score

curl http://localhost:8155/api/docs/health-score | python3 -c "
import json, sys
d = json.load(sys.stdin)
print(f'Score: {d[\"total_score\"]}/100  Grade: {d[\"grade\"]}')
for k, v in d['components'].items():
    print(f'  {k}: {v[\"score\"]}/{v[\"max\"]}')
"

Expected output:

Score: 80/100  Grade: B
  architecture_freshness: 35/60
  api_coverage: 40/40
  maintenance_sla: 5/40

Option B: Run With Docker

Step 1 — Build and Start

USE_DOCKER=true ./start.sh

OR manually:

cd techdocs
docker-compose build
docker-compose up -d

Step 2 — Check Container Health

docker-compose ps

Expected output:

NAME                STATUS          PORTS
techdocs-backend-1  Up (healthy)    0.0.0.0:8155->8155/tcp
techdocs-frontend-1 Up              0.0.0.0:3155->3155/tcp

Step 3 — Follow Logs

# Backend logs
docker-compose logs -f backend

# All services
docker-compose logs -f

Step 4 — Run Tests Inside Container

docker-compose exec backend python -m pytest tests/ -v

Step 5 — Stop Docker Services

./stop.sh
# OR
cd techdocs && docker-compose down

Running Tests Manually

Unit Tests Only

cd techdocs/backend
source venv/bin/activate
python -m pytest tests/test_docs_system.py::TestArchitectureEngine -v
python -m pytest tests/test_docs_system.py::TestApiDocsEngine -v
python -m pytest tests/test_docs_system.py::TestTroubleshootingEngine -v
python -m pytest tests/test_docs_system.py::TestMaintenanceProcedures -v

Integration Tests Only

cd techdocs/backend
source venv/bin/activate
python -m pytest tests/test_docs_system.py::TestAPIEndpoints -v

All Tests with Coverage

cd techdocs/backend
source venv/bin/activate
pip install pytest-cov --quiet
python -m pytest tests/ -v --cov=app --cov-report=term-missing

Expected test summary:

================================ test session starts ==================================
collected 20 items

tests/test_docs_system.py::TestArchitectureEngine::test_generate_architecture_doc_has_services PASSED
tests/test_docs_system.py::TestArchitectureEngine::test_architecture_doc_has_data_flows PASSED
tests/test_docs_system.py::TestArchitectureEngine::test_architecture_doc_has_version PASSED
tests/test_docs_system.py::TestArchitectureEngine::test_health_score_returns_numeric PASSED
tests/test_docs_system.py::TestArchitectureEngine::test_services_have_required_fields PASSED
tests/test_docs_system.py::TestApiDocsEngine::test_validate_spec_returns_coverage PASSED
tests/test_docs_system.py::TestApiDocsEngine::test_undocumented_endpoint_reduces_coverage PASSED
tests/test_docs_system.py::TestApiDocsEngine::test_generate_api_reference_extracts_endpoints PASSED
tests/test_docs_system.py::TestTroubleshootingEngine::test_get_all_trees_returns_list PASSED
tests/test_docs_system.py::TestTroubleshootingEngine::test_each_tree_has_required_fields PASSED
tests/test_docs_system.py::TestTroubleshootingEngine::test_get_tree_by_id PASSED
tests/test_docs_system.py::TestTroubleshootingEngine::test_navigate_to_resolution PASSED
tests/test_docs_system.py::TestTroubleshootingEngine::test_navigate_to_decision_node PASSED
tests/test_docs_system.py::TestMaintenanceProcedures::test_get_all_procedures_returns_list PASSED
tests/test_docs_system.py::TestMaintenanceProcedures::test_procedures_have_required_fields PASSED
tests/test_docs_system.py::TestAPIEndpoints::test_health_endpoint PASSED
tests/test_docs_system.py::TestAPIEndpoints::test_architecture_endpoint PASSED
tests/test_docs_system.py::TestAPIEndpoints::test_api_reference_endpoint PASSED
tests/test_docs_system.py::TestAPIEndpoints::test_list_runbooks_endpoint PASSED
tests/test_docs_system.py::TestAPIEndpoints::test_execute_runbook_endpoint PASSED
...
================================ 20 passed in 2.14s ==================================

UI Dashboard Verification

Open your browser and navigate to:

URL What You Should See

http://localhost:3155

TechDocs Dashboard — Overview with Health Score gauge

http://localhost:3155

→ Architecture Service map with 6 services and data flows table

http://localhost:3155

→ API Reference All 10 endpoints with method badges, expandable detail

http://localhost:3155

→ Deployment Two runbooks with Execute button; live step trace

http://localhost:3155

→ Troubleshoot Three symptom cards; interactive decision tree on click

http://localhost:3155

→ Maintenance 5 procedure cards with Run Now button and execution log http://localhost:8155/api/docs Swagger UI with all endpoints http://localhost:8155/api/redoc Redoc rendered API reference

Troubleshooting Common Issues

Backend fails to start — port already in use:

lsof -ti :8155 | xargs kill -9

Frontend blank page:

# Check Vite is running
curl http://localhost:3155
# Check proxy is hitting backend
curl http://localhost:3155/api/docs/architecture

Tests fail with import errors:

cd techdocs/backend
source venv/bin/activate
pip install -r requirements.txt
python -c "from app.main import app; print('Import OK')"

Docker build fails:

docker-compose build --no-cache
docker-compose logs backend

Stopping All Services

./stop.sh

This kills local processes on ports 8155 and 3155, stops Docker containers if running, and cleans up PID/log files.

File Structure After Build

techdocs/
├── backend/
│   ├── app/
│   │   ├── main.py                    ← FastAPI app + all routes
│   │   ├── models/database.py         ← SQLAlchemy models
│   │   ├── docs_engine/
│   │   │   ├── architecture.py        ← System topology generator
│   │   │   ├── api_docs.py            ← OpenAPI validator + reference builder
│   │   │   └── deployment.py          ← Runbook executor
│   │   ├── troubleshoot/
│   │   │   └── decision_tree.py       ← Decision tree engine
│   │   └── maintenance/
│   │       └── procedures.py          ← Procedure scheduler + executor
│   ├── tests/
│   │   └── test_docs_system.py        ← 20 unit + integration tests
│   ├── requirements.txt
│   ├── Dockerfile
│   └── venv/                          ← Python virtual environment
├── frontend/
│   ├── src/
│   │   ├── App.jsx                    ← Root component + routing
│   │   ├── components/
│   │   │   ├── Sidebar.jsx            ← Navigation
│   │   │   ├── PageHeader.jsx         ← Reusable header
│   │   │   └── Card.jsx               ← Reusable card
│   │   └── pages/
│   │       ├── OverviewPage.jsx       ← Health score gauge
│   │       ├── ArchitecturePage.jsx   ← Service map
│   │       ├── ApiReferencePage.jsx   ← Endpoint browser
│   │       ├── RunbooksPage.jsx       ← Runbook executor
│   │       ├── TroubleshootPage.jsx   ← Decision tree UI
│   │       └── MaintenancePage.jsx    ← Procedure runner
│   ├── package.json
│   ├── vite.config.js
│   └── Dockerfile
├── docker-compose.yml
├── start.sh
└── stop.sh

Assignment Solution Hints

The assignment asks you to build a documentation health score gauge. Here’s the approach:

Step 1 — API endpoint already exists: GET /api/docs/health-score returns all three component scores. The backend is already built.

Step 2 — Add a gauge to Overview page:

// In OverviewPage.jsx, use recharts RadialBarChart
import { RadialBarChart, RadialBar, ResponsiveContainer } from 'recharts'

const gaugeData = [{ value: healthData.total_score, fill: gradeColor(healthData.grade) }]
// RadialBarChart cx/cy 50%, startAngle=90, endAngle computed from score

Step 3 — Grade colour mapping:

const gradeColor = g => ({ A: '#22c55e', B: '#eab308', C: '#f97316', D: '#ef4444' }[g] || '#8a9bbf')

Step 4 — The gauge is already implemented in OverviewPage.jsx as part of this lesson. Study it and extend it to show a tooltip explaining each component’s contribution on hover.

This Week’s Goal

By end of this lesson your system has:

• ✅ Auto-generated API reference that fails CI if undocumented

• ✅ Architecture diagram generated from live compose config

• ✅ Executable deployment runbooks with rollback steps

• ✅ Interactive troubleshooting decision trees

• ✅ Scheduled maintenance procedures with execution logs

Assignment

Task: Add a documentation health score to the dashboard.

The score should measure:

1. Percentage of API endpoints with descriptions (from OpenAPI spec)

2. Percentage of maintenance procedures executed on-schedule in the last 30 days

3. Age of the last architecture doc rebuild (penalise if >7 days)

Combine into a 0–100 score displayed as a gauge on the dashboard.

Solution Hints:

1. Parse /openapi.json — count total operations, count those with description field. Ratio × 40 = component 1 score.

2. Query maintenance_log for last 30 days, compare against scheduled frequency in maintenance.yml. Ratio × 40 = component 2 score.

3. Read architecture.json mtime, compute age in days. If <7: 20 points; 7–14: 10 points; >14: 0 points.

4. Sum components, display as a Radial gauge using recharts RadialBarChart.

Takeaway

Documentation built as a first-class engineering artifact — generated, validated, versioned, and executed — is what separates systems that are maintainable at scale from those that collapse under their own operational weight. Stripe doesn’t write API docs. They write code that emits API docs. That’s the standard you’re now building to.

Working Code Demo :

By end of this lesson you will have:

• A Production Integration Test Suite that validates all running services end-to-end

• Automated security control checks (JWT, CORS, rate limiting, TLS headers)

• Alerting pipeline smoke tests — confirming Prometheus → Alertmanager → notification chain

• Disaster Recovery drills — scripted failover that proves your recovery targets are real

• A React Integration Dashboard displaying live pass/fail status across all validation domains

Why This Lesson Exists

Every engineering team has a go-live checklist. Most are wrong or incomplete. The gap between “monitoring is configured” (Day 153) and “monitoring actually fires correctly under real failure” is exactly what kills production systems quietly. Netflix calls this chaos injection at deploy time — they validate failure paths before declaring a deploy stable. We’re doing the same, at our scale.

Component Architecture

The integration test harness wraps around your entire stack. Think of it as a thin validation layer that sits outside your application boundary and probes it the same way a real user or adversary would.

┌─────────────────────────────────────────────────────┐
│              Integration Test Orchestrator           │
│   (pytest + asyncio + httpx)                        │
├──────────┬──────────┬─────────────┬─────────────────┤
│ Service  │ Security │  Alerting   │   DR Validation │
│ Health   │ Controls │  Pipeline   │   & Failover    │
│ Probes   │ Checks   │  Smoke Test │   Drills        │
└────┬─────┴────┬─────┴──────┬──────┴──────┬──────────┘
     │          │             │             │
     ▼          ▼             ▼             ▼
  FastAPI    JWT/CORS     Prometheus    PostgreSQL
  Redis      Rate Limit   Alertmanager  Redis Failover
  Postgres   TLS Headers  Webhook       Backup Restore

The orchestrator runs sequentially through four validation domains. Each domain emits a structured result object. The React dashboard polls a /api/integration/results endpoint (WebSocket for live updates) and renders the state tree.

Read more

• A Prometheus + Grafana observability stack collecting real application metrics

• Alertmanager with routed alert rules (warning vs. critical severity)

• A FastAPI backend exposing /metrics, /health, and business logic endpoints

• An incident response workflow with on-call rotation logic

• Runbook automation — structured response playbooks tied directly to alerts

• A React dashboard (Datadog/PagerDuty-inspired UI) showing live system health, alert feeds, and on-call status

The Architecture at a Glance

Read more

Yesterday you locked the doors with security hardening. Today you optimize the engine. Here’s the agenda:

• Redis caching layer — cut database round-trips by 80%

• Resource monitoring dashboard — CPU, memory, disk, and request metrics in real time

• Auto-scaling logic — spin workers up/down based on load signals

• Capacity planning module — forecast when you’ll need more infrastructure

• Performance benchmark suite — prove the gains with numbers

By end of day you’ll have a live dashboard that looks and behaves like what the Datadog or New Relic SRE team stares at during a product launch.

Where This Fits in the System

Your stack now has: API Gateway → App Servers → PostgreSQL + Redis → Background Workers. The performance layer wraps all of these — it doesn’t replace anything, it sits alongside and observes, caches, and reacts. Think of it as the nervous system.

Read more

Yesterday (Day 150), we deployed our production application. Today we harden it. Here’s the agenda:

• Security Control Layer — rate limiting, IP allowlisting, request validation

• Firewall Rules — UFW-based network-level access restrictions

• Intrusion Detection — Fail2ban watching live logs and banning attackers

• Security Monitoring Dashboard — real-time React UI showing threat events, ban logs, and system posture

By end of lesson, you’ll have a live dashboard that detects, blocks, and visualizes attacks happening in real time.

Where This Fits in the 180-Day System

Week 21: Deploy → Week 22: Harden → Week 23: Tune → Week 24: Scale
          Day 150          Day 151         Day 152
       [Application]   [Security]      [Performance]
       [Deployment ]   [Hardening]     [Tuning     ]

Security hardening sits between deployment and performance tuning intentionally. You don’t tune what isn’t secured — attackers will exploit any performance shortcuts.

Component Architecture

Read more

• Deploy a Python/FastAPI backend and React frontend to production Docker containers

• Wire environment variables through a secrets-safe .env pipeline

• Set up Prometheus metrics collection with a custom /metrics endpoint

• Implement layered health checks: liveness, readiness, and dependency probes

• Launch a Grafana performance dashboard — live, real data, zero mocks

The Mental Model: Deployment Is Not “Just Running Code”

Every engineer at some point treats deployment as “copy code, run it.” That mental model collapses at scale. Production deployment is an operational contract — a guarantee that your app starts correctly, fails predictably, exposes its own health, and integrates into the monitoring fabric of your infrastructure.

Netflix, Shopify, and Stripe all treat deployments as observable events, not silent background tasks. Their systems emit telemetry from the first millisecond of startup.

Component Architecture

┌─────────────────────────────────────────────────────┐
│                   PRODUCTION STACK                   │
│                                                     │
│  React Frontend ──► Nginx Proxy ──► FastAPI Backend │
│                                          │           │
│                                     PostgreSQL DB    │
│                                          │           │
│                              Prometheus ◄─────────── │
│                                   │                  │
│                               Grafana Dashboard      │
└─────────────────────────────────────────────────────┘

The backend exposes three categories of endpoints:

1. Business endpoints — actual application logic

2. Health probes — /health/live, /health/ready, /health/dependencies

3. Metrics endpoint — /metrics scraped by Prometheus every 15 seconds

The frontend is a React SPA served via Nginx — same pattern used by Vercel’s internal deployments. Nginx also reverse-proxies API calls to the backend container, eliminating CORS complexity.

Environment Variables: The Right Way

Most junior engineers hardcode secrets. Mid-level engineers move secrets to .env files. Senior engineers design an environment promotion pipeline where the same app binary runs in dev, staging, and prod — only the environment context changes.

.env.development  →  dev database, debug logging ON
.env.staging      →  staging database, debug logging OFF  
.env.production   →  prod database, structured JSON logging, rate limiting ON

Your backend reads config using pydantic-settings — it validates types, provides defaults, and fails loudly at startup if required vars are missing. No silent misconfiguration.

Health Checks: Three Levels, Three Purposes

Kubernetes and Docker orchestrators need to know: Is this container alive? Is it ready to serve traffic? Are its dependencies healthy?

Liveness probe — answers “Is the process running?” A 200 means the container shouldn’t be killed. Returns 503 only on catastrophic internal failure.

Readiness probe — answers “Can this instance take traffic right now?” If the DB connection pool is exhausted, return 503 here. The load balancer routes around you.

Dependency check — answers “Are all my downstream services healthy?” Checks PostgreSQL connectivity, Redis ping, and external API reachability. Used by deployment automation to gate rollouts.

# Pseudo-code pattern
@app.get("/health/ready")
async def readiness():
    db_ok = await check_db_connection()
    cache_ok = await check_redis_ping()
    if db_ok and cache_ok:
        return {"status": "ready"}
    raise HTTPException(503, detail="dependencies not ready")

This pattern is identical to what Google’s production infrastructure uses internally — fail fast at startup, fail clear in operation.

Prometheus Metrics: What to Instrument

Don’t instrument everything. Instrument the four golden signals (coined by Google SRE):

Signal Metric Name What It Tells You Latency http_request_duration_seconds Are requests slow? Traffic http_requests_total How much load? Errors http_errors_total Are things failing? Saturation process_open_fds Am I running out of resources?

Your FastAPI app will use prometheus-fastapi-instrumentator to auto-instrument all routes with zero boilerplate, plus custom counters for business-specific metrics (e.g., deployments triggered, health check failures).

Grafana Dashboard: What Good Looks Like

A production Grafana dashboard has three rows:

1. Availability row — uptime, health check status, error rate

2. Performance row — P50/P95/P99 latency, requests/sec, active connections

3. Resource row — CPU, memory, open file descriptors, GC pressure

This is structurally identical to Datadog’s APM default view. You’re building the open-source version.

Step-by-Step Implementation Guide

Github Link:

https://github.com/sysdr/infrawatch-fullstack-p/tree/main/day150/day150_app

Architecture Overview

React SPA (port 3000)
    ↓ (HTTP)
Nginx Reverse Proxy (port 80)
    ↓ (/api/*)
FastAPI Backend (port 8000)
    ├── /health/live      → Liveness probe
    ├── /health/ready     → Readiness probe
    ├── /health/deps      → Dependency check
    ├── /api/v1/*         → Business logic
    └── /metrics          → Prometheus scrape target
         ↑
    Prometheus (port 9090) → scrapes every 15s
         ↓
    Grafana (port 3001) → PromQL queries → dashboards

Prerequisites Check

# Verify tools
python3 --version        # Expected: Python 3.11+
node --version           # Expected: v20+
docker --version         # Expected: Docker 24+
docker compose version   # Expected: v2.20+
curl --version

Option A: Local Mode (No Docker)

1. Run the Build Script

chmod +x start.sh stop.sh
./start.sh

Expected output:

══ STEP 1: Creating Project Structure ══
[BUILD] Directory tree created
[BUILD] All directories verified ✓
...
══ STEP 7: Setting Up Python Virtual Environment ══
[BUILD] Python dependencies installed
══ STEP 8: Running Unit Tests (Local) ══
PASSED tests/test_health.py::test_liveness_returns_200
PASSED tests/test_health.py::test_readiness_returns_200_when_deps_healthy
PASSED tests/test_health.py::test_readiness_returns_503_when_db_down
PASSED tests/test_health.py::test_dependency_check_endpoint
PASSED tests/test_health.py::test_root_endpoint
PASSED tests/test_health.py::test_metrics_endpoint_returns_prometheus_format
PASSED tests/test_deployments.py::test_trigger_deployment
PASSED tests/test_deployments.py::test_list_deployments
PASSED tests/test_deployments.py::test_feature_flag_evaluation
PASSED tests/test_deployments.py::test_metrics_summary
10 passed in ~2.5s

2. Functional Verification

# Liveness
curl http://localhost:8000/health/live
# Expected: {"status":"alive","uptime_seconds":4.2,"timestamp":...}

# Root info
curl http://localhost:8000/
# Expected: {"name":"Day150App","version":"1.0.0","environment":"development",...}

# Readiness (may be degraded if no DB/Redis — this is expected behavior)
curl http://localhost:8000/health/ready
# Expected: 200 {"status":"ready",...} or 503 if DB not running

# Prometheus metrics
curl http://localhost:8000/metrics | head -20
# Expected: # HELP http_requests_total ... + metric lines

# Trigger a deployment
curl -X POST http://localhost:8000/api/v1/deployments \
  -H "Content-Type: application/json" \
  -d '{"environment":"staging","version":"v1.2.3","service_name":"api-service"}'
# Expected: {"id":"deploy-...","status":"success",...} (90% of the time)

# Feature flag evaluation
curl http://localhost:8000/api/v1/feature-flags/dark-mode
# Expected: {"flag_name":"dark-mode","result":"enabled",...}

# Metrics summary
curl http://localhost:8000/api/v1/metrics/summary
# Expected: {"total_deployments":1,"successful":1,"failed":0,"success_rate":100.0,...}

3. API Interactive Docs

http://localhost:8000/docs

All endpoints are explorable and testable here.

Option B: Docker Compose Mode

1. Run with Docker Flag

./start.sh --docker

Expected output (abridged):

══ STEP 7: Building & Starting Docker Compose Stack ══
[+] Building backend ... done
[+] Building frontend ... done
[+] Running 6/6
 ✔ Container day150_postgres   Healthy
 ✔ Container day150_redis      Healthy
 ✔ Container day150_backend    Healthy
 ✔ Container day150_frontend   Started
 ✔ Container day150_prometheus Started
 ✔ Container day150_grafana    Started

2. Verify Docker Endpoints

# Backend health
curl http://localhost:8000/health/live
# Expected: {"status":"alive",...}

curl http://localhost:8000/health/ready
# Expected: {"status":"ready","dependencies":{"postgresql":"healthy","redis":"healthy"}}

# Dependency detail
curl http://localhost:8000/health/dependencies
# Expected: {"overall":"healthy","services":{"postgresql":{"healthy":true,"latency_ms":1.2},...}}

# Prometheus
curl http://localhost:9090/-/healthy
# Expected: Prometheus Server is Healthy.

# Prometheus metrics available
curl "http://localhost:9090/api/v1/query?query=up"
# Expected: JSON with metric values

# Frontend
curl -s http://localhost:3000 | grep -o '<title>.*</title>'
# Expected: <title>Day 150 — Deployment Dashboard</title>

3. Grafana Dashboard

URL:      http://localhost:3001
Username: admin
Password: day150admin

Navigate to: Dashboards → Day150 → Day 150 — Application Deployment

You should see:

• App Uptime counter (top left)

• Request Rate stat

• Error Rate stat

• Health Status indicator (1 = READY)

• P95/P50 Latency time series (updates after a few requests)

• Dependency Health time series

• HTTP Requests by Status (updates in real-time)

Generate traffic to populate charts:

# Stress the API with 50 requests
for i in $(seq 1 50); do
  curl -s http://localhost:8000/api/v1/deployments \
    -X POST -H "Content-Type: application/json" \
    -d "{\"environment\":\"prod\",\"version\":\"v$i.0\",\"service_name\":\"svc\"}" > /dev/null
  curl -s "http://localhost:8000/api/v1/feature-flags/flag-$((i % 4))" > /dev/null
done
echo "Done — check Grafana in 15s"

Understanding the Code

Why pydantic-settings?

# This fails LOUDLY at startup if DATABASE_URL is missing or wrong type
class Settings(BaseSettings):
    DATABASE_URL: str          # required — no default
    DEBUG: bool = False        # optional with default
    HEALTH_CHECK_INTERVAL: int = 15

If you forget DATABASE_URL in your .env, your app exits immediately with a validation error. You catch misconfigurations before the first request, not during.

Three-Level Health Checks

/health/live  → Docker HEALTHCHECK → "restart if dead"
/health/ready → Load balancer → "route traffic here?"  
/health/deps  → Deployment gate → "safe to deploy?"

These map directly to Kubernetes livenessProbe, readinessProbe, and pre-deployment checks.

Prometheus Auto-Instrumentation

Instrumentator().instrument(app).expose(app)

This one line adds http_requests_total, http_request_duration_seconds, and http_requests_in_progress to every route automatically. You still add custom business metrics on top.

Common Issues & Fixes

Symptom Cause Fix /health/ready returns 503 DB not running Start PostgreSQL or use local mode metrics endpoint missing Instrumentator not installed pip install prometheus-fastapi-instrumentator Grafana shows “No data” Backend not scraped yet Wait 30s for first scrape; check Prometheus targets Frontend blank CORS origin mismatch Check ALLOWED_ORIGINS in .env

Running Tests Independently

cd day150_app/backend
source venv/bin/activate

# All tests
pytest tests/ -v

# Single test file
pytest tests/test_health.py -v

# With coverage report
pytest tests/ -v --cov=app --cov-report=term-missing

# Specific test
pytest tests/test_health.py::test_liveness_returns_200 -v

Stopping Everything

./stop.sh

For Docker specifically:

cd day150_app
docker compose down -v  # -v removes volumes (DB data)

What You’ve Built

By completing this lesson, you’ve deployed:

1. A FastAPI production backend with structured JSON logging, Pydantic config validation, and three-tier health checks

2. A React dashboard that polls live endpoints and visualizes deployment operations in real-time

3. A Prometheus + Grafana observability stack with the four golden signals already wired up

4. A complete test suite covering health probes, deployment endpoints, and feature flag logic

This is architecturally identical to how Stripe, Shopify, and Cloudflare deploy and monitor their services — you have the same operational primitives.

Success Criteria

By end of this lesson, you should be able to:

• [ ] Hit GET /health/live and get {"status": "alive"} within 100ms

• [ ] Hit GET /health/ready and see all dependencies marked healthy

• [ ] See your app’s request metrics appear in Prometheus at localhost:9090

• [ ] Open Grafana at localhost:3001 and see live P95 latency updating every 15 seconds

• [ ] Trigger a simulated failure, watch the health check turn red, watch it recover

Key Insight to Remember

The app that deploys cleanly, exposes its health, and emits structured telemetry from minute one is worth ten times the app that “works on my machine.” Deployment is product quality.

Homework Assignment

Task: Extend the monitoring stack with a custom business metric.

Add a Prometheus counter called feature_flag_evaluations_total with labels {flag_name, result}. Instrument it in a new /api/feature-flags/{flag_name} endpoint that randomly returns enabled or disabled (simulate a feature flag service). Create a Grafana panel showing evaluation rate by flag name.

Solution Hints:

1. Use prometheus_client.Counter with labelnames=["flag_name", "result"]

2. Call .labels(flag_name=name, result=result).inc() on each request

3. In Grafana: query rate(feature_flag_evaluations_total[1m]), group by result label

4. Add a legend with {{flag_name}} - {{result}} format string

5. Set panel type to “Time series” with stacked display

Working Code Demo:

Today you ship the production database tier for the infrastructure stack you’ve been assembling. This is one of the highest-stakes sessions in the series because every architectural decision here has a direct, measurable cost when something goes wrong at 2 AM.

Today’s agenda:

• Deploy a primary-replica database cluster with real replication mechanics

• Build a backup pipeline that actually verifies its own restores

• Stand up a monitoring layer that measures what it claims to measure

• Wire in a performance tuning advisor that reads real query plans

• Connect everything to a production-grade control plane dashboard

Where This Fits

Last session (Day 148) you configured load balancing, SSL, and CDN at the edge. Today you go one tier deeper — the database. Tomorrow (Day 150) you deploy the application on top of what you build here.

Think of this week as building a three-tier sandwich: Day 148 = the crust (edge/CDN), Day 149 = the filling (database), Day 150 = the top layer (application). The crust protects, the filling is what users are actually paying for, and the top layer is what they touch. If today’s tier goes down, everything above it goes with it.

Component Architecture

Read more

Yesterday you proved your DevOps pipeline could deploy, roll back, and recover. Today we build the thing that pipeline actually deploys to — a real production environment. By the end of this lesson you’ll have a load balancer routing live traffic across three application servers, SSL certificates encrypting every request, a CDN simulator caching content at the edge, and a monitoring system watching it all breathe. Four systems, one dashboard, zero mock data.

Here’s the high-level shape of what you’re assembling:

• A load balancer that distributes requests using three different algorithms and automatically routes around failures

• An SSL/TLS certificate authority that issues, inspects, and renews real X.509 certificates

• A CDN edge cache simulator with independent regional caches, TTL expiry, and hit/miss accounting

• A production monitoring collector that samples real system metrics and fires threshold-based alerts

Read more

By the end of today, you’ll have a fully wired DevOps integration suite that proves every stage of your pipeline works together — not just in isolation. Specifically:

• End-to-end pipeline test harness — fires a code push and watches it travel from commit → build → test → deploy

• Deployment procedure validator — verifies rollout sequencing, canary health gates, and rollback triggers

• Disaster recovery (DR) test runner — simulates failure scenarios and measures RTO/RPO against targets

• Monitoring validation layer — confirms alerts fire, dashboards refresh, and on-call routing works

• DevOps runbook generator — auto-documents every procedure as machine-readable + human-readable output

Where this fits: You’ve spent the last two weeks securing your pipeline (Day 146). Today is “Integration Day” — the day you prove that security gates, deployment logic, monitoring hooks, and disaster recovery mechanisms all cooperate correctly before production goes live (Day 148).

Component Architecture

Read more

Yesterday (Day 145) you built a disaster recovery layer that can survive data loss. Today we close the attack surface around everything that pipeline touches. Here’s the agenda:

• Security scanning baked into the CI/CD pipeline (SAST, dependency audit, container scanning)

• Secret management — Vault-backed dynamic credentials, zero plaintext secrets in code or env files

• Compliance-as-code — automated policy checks that gate deployments

• Vulnerability management — CVE tracking dashboard with severity routing

• Security monitoring — real-time alerting on anomalous events

By end of day you’ll have a full SecurityOps dashboard — think Snyk meets Vault meets a SIEM — running locally and verifiable end-to-end.

Where This Fits — Week 21 Target

Week 21: Production Resilience
  Day 143 — Performance Optimization  ✓
  Day 144 — Scaling Strategies        ✓
  Day 145 — Disaster Recovery         ✓
  Day 146 — DevOps Security           ← YOU ARE HERE
  Day 147 — DevOps Integration        (tomorrow)

Security isn’t a post-launch concern. At companies like Netflix and Shopify, security gates run inside the deployment pipeline — a commit that introduces a critical CVE never reaches production, full stop.

Core Concept: Shift-Left Security

Read more

Here’s the agenda for this session:

• Backup Engine — automated PostgreSQL WAL archiving + snapshot scheduling to MinIO

• Health Monitor + Failover Controller — heartbeat detection with auto-promote of a replica

• Recovery Tester — a chaos runner that validates your RTO and RPO on a schedule

• Business Continuity Plan (BCP) Module — runbooks stored and surfaced through the API

• DR Dashboard — a React UI that looks and feels like PagerDuty’s incident console

Where This Fits in the Week

Your infrastructure stack now has observability eyes (Day 144). Disaster recovery is the immune system — it reacts when those eyes see something bad. The DR components sit between your primary site and your standby site, continuously syncing state and ready to reroute traffic in under 60 seconds.

Core Concept: RTO vs RPO — The Two Numbers That Define Your DR Contract

Every business conversation about disaster recovery eventually reduces to two numbers:

RTO (Recovery Time Objective) — How long can your service be down before the business bleeds? Netflix targets seconds. A hospital EHR might tolerate 4 hours. Your architecture must be designed to meet this number, not just aspire to it.

RPO (Recovery Point Objective) — How much data loss is acceptable? An e-commerce checkout system might say “zero seconds” — every transaction must survive. A logging pipeline might say “15 minutes” is fine.

These aren’t soft guidelines. They’re the engineering spec. If your WAL archiving runs every 5 minutes, your RPO is at least 5 minutes. If your DNS TTL is 300 seconds and your replica takes 40 seconds to promote, your RTO is at least 340 seconds. The math is merciless.

Component Architecture

The system has three logical zones:

Primary Site runs your FastAPI backend, PostgreSQL primary, and Redis cache. The Backup Engine continuously ships WAL segments to MinIO every 60 seconds and runs full snapshots nightly. The Health Monitor pings a /health endpoint every 10 seconds with a 3-failure threshold before triggering the Failover Controller.

DR Site runs a PostgreSQL streaming replica (WAL receiver) and a warm-standby FastAPI instance. It receives WAL in near real-time (~2-5 seconds lag under normal conditions). On failover, the Failover Controller issues pg_promote() to the replica, flushes DNS, and the standby app begins serving traffic.

Observability Layer captures RTO measurements (timestamp of failure detected → timestamp service confirmed healthy at DR site), surfaces them on the dashboard, and archives every test run result.

The Five Components — How They Work

1. Backup Engine (backup_engine.py)

This module has two jobs running as background threads:

WAL Archiving — PostgreSQL’s archive_command is configured to call a Python script that uploads each completed WAL segment to MinIO. Each segment is ~16MB and represents roughly 60 seconds of write activity. You get continuous backup with sub-minute RPO.

Snapshot Scheduler — Every hour (configurable), the engine calls pg_basebackup to take a full base backup, compresses it with pigz, and uploads to MinIO under a dated prefix. These are your recovery anchors.

# Pseudocode: WAL upload on archive_command trigger
def archive_wal_segment(wal_path: str) -> bool:
    compressed = gzip_compress(wal_path)
    key = f"wal/{datetime.utcnow().isoformat()}/{os.path.basename(wal_path)}.gz"
    return minio_client.put_object(bucket, key, compressed)

2. Health Monitor + Failover Controller (health_monitor.py)

The monitor runs a tight loop: HTTP GET to the primary’s health endpoint every 10 seconds. Three consecutive failures flip the internal state machine from OPERATIONAL → DEGRADED → FAILOVER_INITIATED.

The Failover Controller then:

1. Issues SELECT pg_promote() against the replica

2. Waits for pg_is_in_recovery() to return false (confirms promotion)

3. Updates the service’s DNS record (or routing config) to point at the DR site

4. Logs the failover timestamp for RTO calculation

5. Fires a notification to the alert channel

# Pseudocode: State transition logic
if consecutive_failures >= THRESHOLD:
    state = "FAILOVER_INITIATED"
    promote_replica(dr_db_conn)
    update_dns(DR_ENDPOINT)
    record_failover_event(triggered_at=now())

3. Recovery Tester (recovery_tester.py)

This is the most underbuilt component in most real systems — and the most critical. A DR plan you’ve never tested is a hypothesis, not a guarantee.

The tester operates in two modes:

Chaos Mode — Deliberately kills the primary process, starts a timer, waits for the Health Monitor to detect failure, and measures the full time to service restoration at the DR site. This is your measured RTO.

Data Integrity Mode — Writes a known set of records to the primary just before simulating failure, then queries the DR site after promotion and verifies every record is present. The gap between “last write” and “earliest recoverable write at DR” is your measured RPO.

Both modes log results to a dr_test_results table and surface them on the dashboard.

4. Business Continuity Plan Module (bcp_manager.py)

Runbooks are only useful if engineers can find them at 3am. The BCP module stores runbooks as structured JSON documents in PostgreSQL, versioned and searchable. The API exposes /api/runbooks and /api/runbooks/{incident_type} so the dashboard can surface the right playbook the moment an alert fires.

Each runbook has: incident_type, severity, steps[], estimated_rto_minutes, owner, and last_tested_at. The last field matters — it’s how you prove the runbook isn’t stale.

5. DR Dashboard (React)

The UI mirrors what you’d see in a real-time incident tool like PagerDuty or Grafana’s alerting view:

• Status banner — green/amber/red based on current system state

• RTO/RPO gauges — dials showing current measurements vs targets

• Backup timeline — shows last N backup events with sizes and durations

• Test results table — history of every recovery drill with pass/fail and measured RTO

• Runbook panel — context-sensitive playbooks that appear when state is not OPERATIONAL

• Failover trigger button — manual override with confirmation dialog

Data Flow: Normal Operations → Disaster → Recovery

[Primary DB] --WAL--> [MinIO Bucket]
                           |
                    [DR Replica] <-- streaming replication (live)
                    
[Health Monitor] --10s poll--> [Primary /health]
                                     |
                              3x FAIL detected
                                     |
                        [Failover Controller]
                         1. pg_promote(replica)
                         2. DNS update
                         3. Log RTO start→end
                         4. Alert fired
                                     |
                        [DR Site now PRIMARY]
                        [Dashboard → RED → resolving → GREEN]

Implementation, Build, Test & Demo Guide

Github Link:

https://github.com/sysdr/infrawatch-fullstack-p/tree/main/day145/day145-dr

Prerequisites

• Day 144 observability stack running (or standalone PostgreSQL + Docker available)

• Docker + Docker Compose installed

• Python 3.11+, Node 20+

Project Structure

day145-dr/
├── backend/
│   ├── app/
│   │   ├── main.py               # FastAPI entry point
│   │   ├── backup_engine.py      # WAL archiving + snapshot scheduler
│   │   ├── health_monitor.py     # Heartbeat + failover controller
│   │   ├── recovery_tester.py    # Chaos runner + RTO/RPO validator
│   │   ├── bcp_manager.py        # Runbook CRUD API
│   │   └── models.py             # SQLAlchemy models
│   ├── tests/
│   │   ├── test_backup.py
│   │   ├── test_failover.py
│   │   └── test_recovery.py
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── App.jsx
│   │   ├── components/
│   │   │   ├── StatusBanner.jsx
│   │   │   ├── RtoRpoGauges.jsx
│   │   │   ├── BackupTimeline.jsx
│   │   │   ├── TestResultsTable.jsx
│   │   │   └── RunbookPanel.jsx
│   │   └── index.css
│   └── package.json
├── docker-compose.yml
├── docker-compose.dr.yml          # DR site containers
├── start.sh
└── stop.sh

Step 1 — Project & File Structure Creation

The start.sh script handles everything. Here is what it does and what you should see at each step.

chmod +x start.sh && ./start.sh

Expected output (first 30 seconds):

[DR-BUILD] Creating project structure...
[DR-BUILD] All directories verified ✓
[DR-BUILD] Installing Python dependencies in venv...
[DR-BUILD] Installing Node dependencies...
[DR-BUILD] Starting infrastructure (MinIO, PostgreSQL Primary, PostgreSQL Replica)...
[DR-BUILD] Waiting for PostgreSQL primary to be ready...
[DR-BUILD] PostgreSQL primary ready ✓
[DR-BUILD] Configuring streaming replication...
[DR-BUILD] Replication slot created: dr_slot ✓
[DR-BUILD] Replica connected and streaming ✓

Step 2 — Backend Architecture Details

2a. Backup Engine

The engine uses two threads:

WAL Thread — PostgreSQL’s archive_command is set to python3 /app/wal_archive.py %p %f. Every time Postgres finishes writing a WAL segment, it calls this command. The script uploads the file to MinIO bucket wal-archive.

Snapshot Thread — Every 3600 seconds (configurable via SNAPSHOT_INTERVAL_SECONDS env var), calls pg_basebackup -Ft -z -P and streams the tarball to MinIO bucket snapshots.

To verify WAL archiving is working:

# Inside the primary container
psql -U druser drdb -c "SELECT * FROM pg_stat_archiver;"

Expected output:

 archived_count | last_archived_wal          | last_archived_time
 12             | 000000010000000000000004   | 2025-05-20 14:23:01

2b. Health Monitor States

The monitor maintains an in-memory state machine with these transitions:

• OPERATIONAL → checks pass every 10s

• DEGRADED → 1–2 consecutive failures; alert fired but no failover

• FAILOVER_INITIATED → 3rd consecutive failure; automatic promotion begins

• DR_ACTIVE → promotion confirmed; metrics logged

• RESTORED → primary rebuilt and resync complete; failback done

State is persisted to the system_state table so the dashboard survives restarts.

2c. Recovery Tester — What “Real” Means

The tester does NOT use mocks. It:

1. Inserts 100 test rows into recovery_test_data table on the primary

2. Records the timestamp of the last insert (write_watermark)

3. Calls docker stop primary_postgres (or sends SIGTERM to the process)

4. Starts a timer

5. Polls the DR endpoint until it returns HTTP 200

6. Queries recovery_test_data at DR site and counts matching rows

7. Calculates: rto = time_to_200 - failure_detected_at, rpo = write_watermark - latest_row_at_dr

Step 3 — Build Without Docker

# 1. Create virtual environment
python3 -m venv venv && source venv/bin/activate

# 2. Install dependencies
pip install -r backend/requirements.txt

# 3. Start local Postgres (must have pg installed)
pg_ctl -D /usr/local/var/postgresql start

# 4. Initialize schema
psql -U postgres -f backend/app/schema.sql

# 5. Start backend
uvicorn backend.app.main:app --port 8000 --reload

# 6. Start frontend
cd frontend && npm install && npm run dev

Expected: Frontend on http://localhost:5173, API on http://localhost:8000

Step 4 — Build With Docker

# Start full stack (primary + replica + MinIO + backend + frontend)
docker compose -f docker-compose.yml -f docker-compose.dr.yml up -d --build

# Verify all containers running
docker compose ps

Expected output:

NAME                        STATUS          PORTS
day145_postgres_primary     running         5432/tcp
day145_postgres_replica     running         5433/tcp
day145_minio                running         9000-9001/tcp
day145_backend              running         0.0.0.0:8000->8000/tcp
day145_backend_dr           running         0.0.0.0:8001->8000/tcp
day145_frontend             running         0.0.0.0:3000->3000/tcp

Step 5 — Unit Tests

# With venv active
cd backend && python -m pytest tests/ -v

Expected:

tests/test_backup.py::test_wal_upload_to_minio          PASSED
tests/test_backup.py::test_snapshot_created             PASSED
tests/test_failover.py::test_health_check_healthy       PASSED
tests/test_failover.py::test_failover_triggered         PASSED
tests/test_recovery.py::test_rto_within_slo             PASSED
tests/test_recovery.py::test_rpo_within_slo             PASSED
tests/test_recovery.py::test_runbook_crud               PASSED

7 passed in 4.31s

Step 6 — Integration Test (End-to-End Failover)

# Run the automated chaos test
curl -X POST http://localhost:8000/api/recovery/run-test \
  -H "Content-Type: application/json" \
  -d '{"mode": "chaos", "data_rows": 100}'

Watch real-time progress:

# In a second terminal
watch -n 2 'curl -s http://localhost:8000/api/system/state | python3 -m json.tool'

Expected state progression:

{"state": "OPERATIONAL", "since": "2025-05-20T14:00:00Z"}
  ↓ (test kills primary)
{"state": "FAILOVER_INITIATED", "consecutive_failures": 3}
  ↓ (~30-45 seconds later)
{"state": "DR_ACTIVE", "rto_seconds": 38, "rpo_seconds": 4}

Step 7 — Demo Verification

Dashboard Walkthrough

Open

http://localhost:3000

:

1. Status Banner — Should show green “OPERATIONAL” with uptime counter

2. RTO/RPO Gauges — Circular dials; RTO target 60s, RPO target 30s. After test run, actual values appear.

3. Backup Timeline — Table showing last 10 WAL archives and snapshots with sizes (should see rows within 60 seconds of startup)

4. Run DR Test button — Click “Run Chaos Test”. Watch the banner turn amber, then red during failover, then green with “DR ACTIVE” label.

5. Test Results — After test completes, a row appears in the history table with measured RTO, RPO, and pass/fail against your SLOs.

6. Runbook Panel — When system is not OPERATIONAL, the relevant runbook auto-loads in the side panel.

Verify Backup in MinIO

# MinIO console: http://localhost:9001
# Credentials: minioadmin / minioadmin
# Navigate to: wal-archive bucket → verify WAL segments present
# Navigate to: snapshots bucket → verify base backup present

Verify Replication Lag

curl http://localhost:8000/api/replication/status

Expected:

{
  "primary_lsn": "0/4002D88",
  "replica_lsn": "0/4002D88",
  "lag_bytes": 0,
  "lag_seconds": 1.2
}

Step 8 — Stop Everything

./stop.sh

Troubleshooting

Symptom Cause Fix Replica not streaming pg_hba.conf missing replication entry start.sh patches this automatically MinIO 403 on upload Bucket policy Script creates buckets with mc mb on startup Failover not triggering Health endpoint returning cached 200 Check HEALTH_CHECK_CACHE_TTL=0 env var RTO > 60s DNS TTL too high Script sets TTL to 5 seconds via nginx upstream reload

Key API Endpoints

Endpoint Method Description /api/system/state GET Current DR state machine state /api/backups GET List of backup events /api/replication/status GET Real-time replication lag /api/recovery/run-test POST Trigger chaos or integrity test /api/recovery/results GET History of all test runs /api/runbooks GET All BCP runbooks /api/runbooks/{type} GET Runbook for specific incident type /api/failover/trigger POST Manual failover with auth token

Working Code Demo:

• Distributed tracing with OpenTelemetry + Jaeger — follow a single request across 5 microservices

• Metrics collection with Prometheus — scrape, store, and query time-series data

• Observability dashboards with Grafana — production-grade panels that mirror Datadog/New Relic layouts

• SLI/SLO monitoring — define Service Level Indicators and track error budgets in real time

• Alerting rules with Alertmanager — route incidents to the right people before users notice

Read more

• Enhanced Kubernetes deployment with production-grade manifests and rolling update strategies

• Istio service mesh integration for traffic management, mTLS, and observability sidecars

• Horizontal Pod Autoscaler (HPA) + Vertical Pod Autoscaler (VPA) for intelligent scaling

• Liveness, readiness, and startup probes wired to a real health monitoring dashboard

• Resource quotas, LimitRanges, and PodDisruptionBudgets for cluster-wide governance

Component Architecture

Read more

Yesterday you automated deployment pipelines. Today you take one level deeper — you’re going to define your entire infrastructure in code, version it like application code, and provision environments repeatably. Here’s what we tackle:

• Terraform scripts for core infrastructure resources

• Infrastructure versioning with remote state management

• Environment provisioning (dev / staging / prod workspaces)

• Infrastructure testing with pytest + checkov

• Change management via plan/apply workflows with audit trails

By end of day: a working Python + React dashboard that shows your infrastructure state, drift detection results, and change history — the same way HashiCorp Cloud Platform or Spacelift displays it.

Read more

By the end of this lesson, you will have a production-grade CI/CD system that does four things most hobby projects never do:

• Multi-environment promotion — code flows automatically from dev → staging → production, with gates between each

• Canary deployments — ship new code to 5% of users first, watch metrics, then expand or kill it

• Automated rollback — if error rates spike, the system reverts itself without human intervention

• Deployment monitoring dashboard — a React UI that shows live pipeline status, deployment health, and canary traffic splits

This builds directly on Day 140’s quality pipeline. Those test gates now become the entry condition for every deployment.

Core Concept: The Three Problems Advanced CI/CD Solves

Problem 1 — One environment is a lie. If you only test in production, every deploy is a gamble. Multi-environment promotion creates a chain of increasingly production-like environments. Dev is fast and forgiving. Staging mirrors production exactly. Production gets only what staging already validated.

Problem 2 — Binary deploys are risky. Old-school deploys are 0% or 100%. Either nobody has the new code or everybody does. Canary deployments give you a dial: route 5% of real traffic to v2 while v1 handles 95%. Watch error rates, latency p99, and business metrics. If anything looks wrong, flip back instantly. Netflix, Facebook, and Google all ship this way — they call it “progressive delivery.”

Problem 3 — Humans are slow incident responders. At 2 AM, nobody wants to be woken up to manually roll back. Automated rollback watches SLOs (Service Level Objectives) continuously. If the canary’s error rate exceeds your threshold, the system reverts the traffic split within seconds, not minutes.

Read more

• A QA Integration Service (Python/FastAPI) that orchestrates test execution across unit, integration, and end-to-end layers

• Quality Gate Engine — automated pass/fail decisions based on coverage thresholds, error budgets, and performance baselines

• Test Reporting Dashboard (React) with real-time pipeline status, metrics charts, and drill-down failure analysis

• Procedure documentation generator — auto-exports quality runbooks from live pipeline data

Where This Fits

You’ve spent the last two weeks writing tests — unit, integration, UAT. Today is the day those tests graduate from “files on disk” to a living quality system. Think of it like going from having individual smoke detectors to installing a full building fire-safety network: alarm panels, suppression systems, evacuation logs.

In production, companies like Spotify, Stripe, and Atlassian don’t just run tests — they route test results through quality gates that decide whether a build proceeds. That gate is what you’re building today.

Core Concept: Quality Gates Are Decision Engines

A quality gate is not a test. It’s a policy enforcer. It consumes test results and makes a binary decision: does this build meet the bar we set?

The bar has three dimensions in real systems:

Dimension Example Threshold Why It Matters Coverage ≥ 80% line coverage Prevents untested code reaching prod Failure Rate 0 failing tests One broken test = one production risk Performance p95 < 500ms Regressions caught before merge

When Netflix deploys 1000x/day, each deploy goes through a gate. If anything drops below threshold, the gate blocks the deploy — automatically, no human needed.

Architecture: How the Pipeline Flows

Component Breakdown

┌─────────────────────────────────────────────────────┐
│                  QA Integration Service              │
│                                                     │
│  [Test Runner] → [Result Aggregator] → [Gate Engine]│
│       ↓                ↓                    ↓       │
│  pytest/jest      JSON Reports        Pass/Fail     │
│                        ↓                    ↓       │
│               [Report Storage]      [Webhook Notify]│
│                        ↓                            │
│               [React Dashboard]                     │
└─────────────────────────────────────────────────────┘

Test Runner — executes pytest (backend) and jest (frontend) in isolated subprocesses. Captures stdout, exit codes, and coverage XML.

Result Aggregator — parses JUnit XML + coverage.xml into a unified schema. Every test run becomes a structured event with timestamp, suite name, pass/fail counts, and per-file coverage.

Gate Engine — compares aggregated results against configurable thresholds stored in quality_config.json. Produces a gate verdict with a detailed breakdown.

Report Storage — SQLite (dev) / PostgreSQL (prod) stores every pipeline run. Enables trend analysis: “has coverage been dropping over the last 10 commits?”

React Dashboard — polls the QA service every 5 seconds. Shows pipeline status, quality gate verdict, coverage trends, and test failure details.

Control Flow: A Pipeline Run Step-by-Step

1. Trigger — POST /api/pipeline/run with suite name (e.g., "backend", "frontend", "e2e")

2. Dispatch — Gate Engine spawns subprocess, runs appropriate test command

3. Parse — On completion, JUnit XML parsed into structured run object

4. Evaluate — Each gate rule checked: coverage ≥ threshold? failures == 0? duration < max?

5. Persist — Run stored with verdict (PASS/FAIL/WARNING) and all metrics

6. Broadcast — WebSocket pushes status update to connected dashboard clients

7. Expose — GET /api/pipeline/runs serves history; GET /api/pipeline/latest for current verdict

The key architectural insight: the pipeline is stateless per run but stateful across runs. Each execution is atomic; trends emerge from the historical record.

Quality Gate Configuration

This is where most teams get it wrong — they hardcode thresholds. Production systems make thresholds configurable per environment:

{
  “gates”: {
    “coverage”: { “warning”: 75, “failure”: 60 },
    “test_failure_rate”: { “failure”: 0 },
    “performance_p95_ms”: { “warning”: 800, “failure”: 2000 }
  },
  “environments”: {
    “staging”: { “coverage”: { “failure”: 50 } },
    “production”: { “coverage”: { “failure”: 80 } }
  }
}

Staging has looser gates because it’s an experimentation zone. Production gates are strict because failures there are customer-facing.

Test Automation Validation

“Validate test automation” doesn’t mean re-running tests. It means verifying the test infrastructure itself is healthy:

• Are test suites discoverable? (pytest --collect-only exit code 0)

• Do test fixtures set up and tear down cleanly? (no leftover temp files, no leaked DB state)

• Is coverage instrumentation active? (coverage.xml exists after run)

• Are tests deterministic? (flaky test detector — run same suite 3x, flag any test that doesn’t produce consistent results)

This is called test hygiene validation — a practice large eng orgs run as a separate CI step before the actual test run.

Reporting System Architecture

Real reporting isn’t a pretty PDF — it’s a queryable data store with a visualization layer on top.

Data Schema per Run:

pipeline_run {
  id, triggered_at, suite, status,
  total_tests, passed, failed, skipped,
  coverage_pct, duration_ms, gate_verdict,
  failures: [{test_name, file, line, error_msg}]
}

The React dashboard visualizes:

• Trend line — coverage % over last 20 runs (recharts LineChart)

• Status board — current gate verdict with color coding (green/yellow/red)

• Failure table — expandable rows with error details and file:line references

• Suite breakdown — unit vs integration vs e2e pass rates

Quality Procedures Documentation

The automation generates a living QUALITY_PROCEDURES.md from actual pipeline data — not hand-written docs that go stale. It captures:

• Current gate thresholds (from quality_config.json)

• Last 5 pipeline verdicts with timestamps

• Failure patterns (which files fail most often)

• Coverage hotspots (files below threshold)

This is how mature teams ensure documentation stays current: generate it from the source of truth, not from memory.

Implementation, Build, Test & Demo Guide

Github Link:

https://github.com/sysdr/infrawatch-fullstack-p/tree/main/day140/qa_integration

Prerequisites

• Python 3.11+

• Node.js 20+

• Docker + Docker Compose (optional)

• curl and jq installed

Architecture Summary

qa_integration/
├── backend/
│   ├── main.py                          # FastAPI entry point
│   ├── requirements.txt
│   ├── pytest.ini
│   ├── app/
│   │   ├── api/pipeline.py              # REST + WebSocket endpoints
│   │   ├── core/{config,database}.py    # Settings + SQLAlchemy async
│   │   ├── models/pipeline.py           # PipelineRun ORM model
│   │   └── services/
│   │       ├── test_runner.py           # Subprocess test executor
│   │       ├── result_parser.py         # JUnit XML + coverage parser
│   │       ├── gate_engine.py           # Quality gate evaluator
│   │       └── report_generator.py      # Markdown report generator
│   ├── sample_tests/                    # Real tests for gate demo
│   └── tests/unit/ + integration/
├── frontend/
│   └── src/
│       ├── App.js                       # Dashboard shell
│       ├── api/pipeline.js              # Axios API client
│       ├── hooks/usePipeline.js         # Data fetching + WebSocket
│       └── components/                  # GateVerdict, Charts, Tables
├── reports/                             # junit XML, coverage.xml, report.md
├── quality_config.json                  # Gate thresholds
├── start.sh
└── stop.sh

Part A: Without Docker

Step 1 — Build and Start Everything

chmod +x start.sh stop.sh
./start.sh

Expected output (last lines):

[OK] Backend running at http://localhost:8001
[OK] Pipeline run complete → Verdict: PASS | Coverage: 85.2% | Tests: 10
[OK] Hygiene check → healthy: True
Day 140 QA Integration — Build Complete!
  Backend API:  http://localhost:8001
  Frontend:     http://localhost:3001

Step 2 — Verify Backend Health

curl http://localhost:8001/api/pipeline/health

Expected:

{”status”: “ok”, “service”: “qa-integration”}

Step 3 — Trigger a Backend Pipeline Run

curl -s -X POST http://localhost:8001/api/pipeline/run \
  -H “Content-Type: application/json” \
  -d ‘{”suite”:”backend”,”environment”:”development”}’ | python3 -m json.tool

Expected (excerpt):

{
  “id”: 1,
  “suite”: “backend”,
  “status”: “PASS”,
  “gate_verdict”: “PASS”,
  “total_tests”: 10,
  “passed”: 10,
  “failed”: 0,
  “coverage_pct”: 85.2,
  “gate_details”: {
    “coverage”: {”actual”: 85.2, “status”: “PASS”},
    “test_failure_rate”: {”actual”: 0, “status”: “PASS”},
    ...
  }
}

Step 4 — Trigger a Frontend Pipeline Run

curl -s -X POST http://localhost:8001/api/pipeline/run \
  -H “Content-Type: application/json” \
  -d ‘{”suite”:”frontend”,”environment”:”staging”}’

Step 5 — View Run History

curl -s http://localhost:8001/api/pipeline/runs | python3 -m json.tool | head -60

Step 6 — Run Hygiene Validation

curl -s -X POST http://localhost:8001/api/pipeline/validate-hygiene | python3 -m json.tool

Expected:

{
  “healthy”: true,
  “checks”: [
    {”name”: “Test Discovery”, “status”: “PASS”, “detail”: “All tests discoverable”},
    {”name”: “Reports Directory”, “status”: “PASS”, “detail”: “Writable”},
    {”name”: “Quality Config”, “status”: “PASS”, “detail”: “4 gates configured”}
  ],
  “issues”: []
}

Step 7 — Generate Quality Procedures Report

curl -s http://localhost:8001/api/pipeline/report | python3 -c “import sys,json; print(json.load(sys.stdin)[’report’])”

Expected: a markdown document with gate thresholds, run history table, and failure hotspots.

Step 8 — Open Dashboard

Navigate to

http://localhost:3001

in your browser.

What you should see:

• Green header with QA Integration branding

• 4 stat cards: Total Runs, Pass Rate, Avg Coverage, Last Verdict

• Trigger panel — select suite and environment, click ▶ Run Pipeline

• After triggering: Gate Evaluation cards (one per gate with PASS/WARNING/FAIL)

• Coverage Trend line chart with PROD MIN and WARNING reference lines

• Pipeline Run History table — click any row to expand failures

• Hygiene tab — run infrastructure health checks

• Report tab — generate live quality procedures doc

Step 9 — Run Tests Manually

cd qa_integration
source .venv/bin/activate
cd backend

# Sample tests (gate engine behavior)
python -m pytest sample_tests/ -v

# Unit tests
python -m pytest tests/unit/ -v

# Integration tests
python -m pytest tests/integration/ -v

# Full run with coverage
python -m pytest sample_tests/ tests/ -v \
  --cov=app --cov-report=term-missing

Step 10 — Stop Services

./stop.sh

Part B: With Docker

Step 1 — Build and Start

cd qa_integration/start.sh --docker

Step 2 — Verify

curl http://localhost:8001/api/pipeline/health
curl http://localhost:3001

Step 3 — Trigger Pipeline Run via Docker

curl -X POST http://localhost:8001/api/pipeline/run \
  -H “Content-Type: application/json” \
  -d ‘{”suite”:”backend”,”environment”:”production”}’

Step 4 — Stop Docker Stack

cd qa_integration/ ./stop.sh

Verification Checklist

Check Command Expected Backend health curl :8001/api/pipeline/health {"status":"ok"} Pipeline run POST /api/pipeline/run JSON with gate_verdict Run history GET /api/pipeline/runs Array of runs Hygiene check POST /api/pipeline/validate-hygiene {"healthy":true} Report GET /api/pipeline/report Markdown report Dashboard

http://localhost:3001

Green dashboard renders Coverage trend chart Dashboard → scroll down Line chart visible Failure drill-down Click a run row in history Failure details expand Hygiene tab Dashboard → Hygiene tab Infrastructure checks Report tab Dashboard → Report → Generate Markdown doc rendered

API Reference

Method Path Description GET /api/pipeline/health Service health POST /api/pipeline/run Trigger pipeline run GET /api/pipeline/runs All run history GET /api/pipeline/latest Latest run verdict POST /api/pipeline/validate-hygiene Check test infrastructure GET /api/pipeline/report Generate quality procedures WS /api/pipeline/ws Real-time status updates

Interactive API docs: http://localhost:8001/docs

Assignment Steps

Task: Add a “slow test detection” alert card to the dashboard.

Step 1 — Ensure execution_time_budget_ms is in quality_config.json (it already is, threshold: 5000ms).

Step 2 — In gate_engine.py, verify slow_test_warnings is populated when duration_ms > budget_warning.

Step 3 — In PipelineRun model, slow_test_warnings column stores the list.

Step 4 — In App.js, after GateDetails, add:

{latest?.slow_test_warnings?.length > 0 && (
  <div style={{ background: ‘#fffbeb’, border: ‘1.5px solid #d97706’, borderRadius: 12, padding: 16 }}>
    <b>⚠ Slow Tests Detected</b>
    {latest.slow_test_warnings.map((t, i) => (
      <div key={i}>{t.test_name} — {t.duration_ms.toFixed(0)}ms</div>
    ))}
  </div>
)}

Step 5 — In sample_tests/test_gate_engine.py, add a test that creates a slow test entry and verifies the warning card data is in the gate result.

Hint: To simulate a slow test without actually waiting, add "slow_tests": [{"test_name": "test_db_load", "duration_ms": 7000}] to your make_result() call in a new test case.

Success Criteria

By end of today, you should have:

• [ ] QA Integration Service running at localhost:8001

• [ ] Quality gates evaluating coverage, failures, and performance

• [ ] React dashboard showing real-time pipeline status

• [ ] At least one pipeline run stored with full metrics

• [ ] Auto-generated QUALITY_PROCEDURES.md reflecting live data

• [ ] Docker Compose bringing up the entire stack in one command

Assignment

Extend the gate engine to support a fourth gate: test execution time budget. If any single test takes longer than 10 seconds, it should trigger a WARNING verdict (not failure, since slowness isn’t broken — but it needs investigation).

Steps:

1. Add execution_time_budget_ms to quality_config.json

2. In the Gate Engine, after parsing results, identify any test with duration_ms > threshold

3. Collect slow tests into a slow_test_warnings list on the verdict object

4. Surface them in the dashboard as a yellow warning card, separate from failures

Hint: JUnit XML has a time attribute on each <testcase> element. Parse it during aggregation and store per-test durations in your run schema.

Key Takeaway

A QA pipeline without gates is a report generator. Gates are what transform quality data into deployment decisions. The moment you wire a gate verdict to a pipeline stage condition (which you’ll do tomorrow), quality enforcement becomes automatic — no human needs to check if coverage dropped. The system decides.

That’s the mental model shift: from “we run tests” to “tests govern deployments.”

Working Code Demo:

• A full UAT orchestration backend (Python/FastAPI) that manages test plans, scenarios, sessions, and feedback

• A React UAT Dashboard that mirrors the look and feel of modern QA platforms like TestRail and Maze

• Real usability scoring engine that computes SUS (System Usability Scale) scores and heatmaps

• A feedback collection pipeline with sentiment tagging

• End-to-end user experience validation workflows with pass/fail gating

Why UAT Is Not Just “Let Users Click Around”

Most teams treat UAT as a rubber stamp — hand the app to a stakeholder, watch them smile, ship it. That’s not UAT. That’s wishful thinking.

Real UAT is a structured protocol. Think of how NASA runs pre-launch checklists — UAT is your pre-production checklist written from the user’s cognitive model, not the engineer’s. Amazon runs thousands of UAT sessions per quarter through internal beta programs before any significant feature goes live on their retail platform. Netflix calls their equivalent “dog-fooding cohorts” — real employees using real features under real conditions before 200M subscribers see it.

The goal: prove the system behaves the way real humans expect it to, not just the way engineers designed it.

Architecture Overview

Read more

What We’re Building Today

By the end of this lesson you will have a working, production-grade performance testing platform — not a toy, not a demo with mock numbers. Here is what gets built:

• A Locust-powered test suite with five distinct test profiles, each answering a different question about your system

• A FastAPI backend that orchestrates test runs, polls live metrics every two seconds, computes latency percentiles, evaluates SLO thresholds, and streams everything over WebSocket

• A React dashboard — styled after tools like Grafana and k6 Cloud — that shows live RPS charts, P50/P95/P99 latency curves, error rates, and a real-time SLO pass/fail verdict

• A SQLite store for run history so you can compare test results across deployments

The full project runs with a single command, with or without Docker.

Read more