Yesterday you put a cache in front of your query layer and watched p99s drop. Good. But caching hides a storage problem, it doesn’t fix one. Today you fix the one underneath: your log events have been sitting in whatever format was convenient to write, and nobody asked whether that format was any good to read.

Here’s what you’re building:

• A dual-tier storage engine — an append-only row store for fresh writes, and a columnar Parquet store for anything older than 15 minutes

• A background compaction job that migrates segments from row to columnar format once their write activity cools off

• A storage catalog in Postgres that tracks, per partition, which format it’s in and routes queries accordingly

• A query router in the API gateway that picks hot or cold path based on the shape of the query, not just its time range

Why This Matters

A write-heavy row format and a read-heavy columnar format are optimized for opposite access patterns, and most systems pick one and suffer for the other half of their traffic. LinkedIn’s Pinot and Uber’s original log pipeline both hit this: ingest was fast when everything was row-oriented and append-only, but a single “average latency by service, last 24 hours” query meant scanning full rows to read one field, over billions of rows. Netflix solved a version of this with their move to Iceberg — recent data lands in a write-optimized format, gets compacted into columnar Parquet on a schedule, and query engines never know the difference because a metadata layer tells them where to look. The pattern isn’t exotic. It’s just tiering by access pattern instead of by age for its own sake, though age happens to be a decent proxy for “nobody’s writing to this anymore.”

System Design Deep Dive

Row-oriented vs. columnar, and why you need both. A row store puts everything for one record next to everything else for that record — good when you’re appending one record at a time and reading records mostly whole. A columnar store puts every service_name next to every other service_name, every latency_ms next to every other latency_ms. Terrible for single-record writes (you’re touching N different column files per insert), fantastic for “give me the p95 latency for service X” because you read one column, skip the other nineteen, and get compression ratios that make a bulk scan cheap. Trying to serve both patterns from one format means one of your workloads is always paying a tax it didn’t ask for.

Compaction is the tax you pay instead. The trade you’re making is: writes stay cheap because they only touch the hot row store, and reads on old data stay cheap because a background process eventually reorganizes it — but that reorganization costs CPU, I/O, and a window where a segment is neither cleanly hot nor cleanly cold. Get the compaction trigger wrong (too aggressive) and you’re burning cycles converting segments that are still being written to. Too lazy, and your “cold” store balloons with row-format data that nobody optimized, and your columnar reads slow down because half of what they’re scanning isn’t actually columnar yet.

Segment sizing determines your floor and ceiling. Too-small segments mean compaction runs constantly and Parquet’s columnar benefits never amortize — you pay per-file overhead (footer metadata, dictionary encoding setup) more often than you should. Too-large segments mean a segment stays “hot” and unqueryable-efficiently for longer, and a single slow compaction job blocks a bigger unit of data. We land on 15-minute, size-capped segments (128MB target) — small enough to compact fast, big enough that Parquet’s per-file overhead isn’t the dominant cost.

Catalog-driven routing beats format-sniffing. You could have the query layer open a segment and figure out from its file extension or header what format it’s in. Don’t. That couples your query path to your storage internals and means every reader needs to understand both formats’ quirks. Instead, a lightweight catalog table (segment_id, partition_key, format, min_ts, max_ts, row_count) tells the router where to look before it opens anything. This is the same idea behind Iceberg’s manifest files and Hudi’s timeline — metadata is cheap to query and lets you change storage internals without touching every consumer.

Point lookups and aggregations need different routing logic entirely. A query for “show me log event abc-123” is a point lookup — go straight to hot store if it’s recent, or use a partition-pruned columnar scan if it’s old, either way touching one record’s worth of data. A query for “count errors by service over the last 6 hours” is an aggregation spanning both tiers — you query hot and cold separately and merge, because forcing one query path to handle both shapes means over-fetching for one of them. This is the mistake teams make most often here: they build one query interface and let it silently do a full scan for point lookups because the router only knows how to do range scans.

Read more

Elasticsearch exposes logs through REST and GraphQL. Datadog ships SDKs so apps ingest without hand-crafting HTTP. Stripe rate-limits API keys by tier. Slack receives webhooks when events fire. Today you wire all seven access patterns into one platform.

By the end you’ll have:

• Versioned REST API with search, stream, and stats

• GraphQL for flexible log queries and mutations

• Tiered rate limiting and daily quotas

• Python and JavaScript SDKs plus a CLI

• Webhook delivery on ERROR/WARN events

• Batch ingest/query/delete for high throughput

Why this matters

Your storage and analytics layers are useless if nothing can reach them safely. The API layer is the contract every client trusts — mobile apps, batch jobs, on-call engineers, and partner systems all enter here.

Core concepts

Read more

• A device state machine that reconstructs current device status from raw logs

• A state store that persists device state across restarts without a heavy database

• A REST and WebSocket API for querying live device states

• A browser dashboard showing real-time device health across a fleet

This fits into Week 26’s IoT arc: Day 177 handled disconnected edge collection, Day 178 optimized bandwidth-constrained transport, and today we build the intelligence layer — knowing what state every device is in at any moment.

Why Device State Matters

Imagine managing 50,000 temperature sensors in cold-chain logistics. Each sensor streams logs every 30 seconds. A single log line says {"device":"fridge-42","temp":8.2,"battery":12}. But the question that matters to operations is not “what did fridge-42 just report?” — it is “what is the current state of fridge-42, and has anything changed since last check?”

That distinction — from raw event to current state — is what separates a log sink from an operational system.

Cloudflare’s IoT gateway team faces exactly this challenge managing millions of edge nodes. Their internal state service reconstructs device health from streamed telemetry, never relying on devices to explicitly announce state changes. State is inferred from the log stream itself.

Core Concept: State Machines from Log Streams

A device does not exist in a vacuum. At any moment it is in exactly one state: online, degraded, offline, alert, or unknown. Transitions between these states are driven by log events.

The core insight is that logs are evidence of state, not the state itself. Your system observes evidence and maintains a derived state model. This is called event-sourced state reconstruction.

Log Event  →  State Transition Rules  →  Current State

State transitions work as follows:

• A telemetry log within the expected interval keeps a device online

• A telemetry log with a battery value below threshold moves it to degraded

• No telemetry for more than 2x the expected interval moves it to offline

• A log with an error field or threshold breach moves it to alert

This mirrors how Apache Flink processes IoT streams at Alibaba — the stream processor holds a rolling state view for every device key, updated on each incoming record.

Architecture

The system has four layers:

Log Ingestion Layer — An HTTP endpoint accepts batches of device logs. Each log carries a device ID, timestamp, and payload. The ingestion layer validates and forwards to the state processor.

State Processor — The central brain. For each incoming log, it applies transition rules, updates the state store, and emits a change event if the state actually changed. It also runs a background timer that checks for devices that have gone silent.

State Store — A lightweight SQLite-backed store with an in-memory cache. Stores the current state, last seen timestamp, last log payload, and a transition history ring buffer (last 20 transitions per device).

Query API + WebSocket Hub — A FastAPI server exposing /devices, /devices/{id}, and a WebSocket endpoint /ws/state-changes that pushes diffs to connected dashboards.

Read more

• Multi-tier cache architecture with L1 (in-process Caffeine) + L2 (Redis) for log query results

• Cache-aside pattern integrated into the log query pipeline with TTL and eviction strategies

• Cache warming on startup to pre-populate high-frequency query results

• Cache invalidation triggered by new log ingestion events via Kafka consumers

Why This Matters

In a distributed log processing system, the query layer is typically your hottest path. Logs are written once but queried many times — dashboards polling every 30 seconds, alerts re-evaluating the same time windows, and engineers repeatedly querying the same service/severity combinations during incidents.

Without caching, every query hits PostgreSQL. At modest scale (10K logs/sec), even a well-indexed DB begins buckling under the read amplification from dashboards and alerting pipelines. Netflix’s operational metrics infrastructure processes billions of log events daily — their query layer survives through aggressive caching at multiple levels, with cache hit rates above 95% for dashboard queries.

The architectural insight: read-heavy workloads don’t scale by optimizing the database — they scale by avoiding the database entirely for repeated queries.

System Design Deep Dive

1. Multi-Tier Cache Architecture

The standard pattern in high-throughput systems is a two-level cache:

QueryService → L1 Cache (Caffeine, in-process) → L2 Cache (Redis, distributed) → PostgreSQL

L1 (Local/Heap Cache):

• Sub-millisecond access, no network hop

• Small capacity (512MB–1GB heap budget)

• Invalidated on pod restart — not shared across instances

• Best for: ultra-hot queries (top-10 dashboard panels)

L2 (Distributed Cache — Redis):

• ~1–2ms access over loopback/LAN

• Shared across all API instances

• Survives pod restarts; TTL-controlled

• Best for: warm queries, session-scoped aggregations

Trade-off: Two-tier adds complexity. You now have two caches to invalidate and two failure modes. Accept this complexity only when L2 latency (~2ms) is still too high for your SLA. For most log platforms, L2 alone suffices until you’re serving 50K+ QPS.

Anti-pattern: Never cache at both layers with different TTLs without a coherence strategy. If L1 TTL (30s) > L2 TTL (10s), you’ll serve stale L1 data after L2 has been refreshed. L1 TTL must always be ≤ L2 TTL.

2. Cache-Aside (Lazy Loading) Pattern

Cache-aside is the dominant pattern for log query caching:

1. Check L1 → hit: return immediately
2. Check L2 → hit: populate L1, return
3. Miss: query DB → populate L2 → populate L1 → return

Why not write-through? Log ingestion is high-volume and write-through would cache every possible query permutation on write — combinatorially impossible. Cache-aside caches only what’s actually queried.

Key insight: Cache key design is your most important architectural decision here. A poor key strategy leads to cache pollution (too many unique keys, low hit rate) or stale data (keys too coarse, wrong data returned).

// Bad: too granular — each timestamp variation = cache miss
String key = "logs:" + serviceId + ":" + timestamp.toEpochMilli();

// Bad: too coarse — wrong results returned for different services
String key = "logs:recent";

// Good: query-shape based key with time bucketing
String key = String.format("logs:svc=%s:lvl=%s:bucket=%d",
    serviceId, level, timestamp.truncatedTo(ChronoUnit.MINUTES).toEpochSecond());

Time-bucketing (rounding timestamps to the nearest minute/5-minute window) dramatically improves hit rates for time-series log queries while keeping staleness bounded.

3. Cache Invalidation via Kafka

The hardest problem in caching is knowing when to invalidate. In a log processing system, cache invalidation triggers when:

1. New logs arrive for a cached service/time window

2. A log is retroactively tagged or reprocessed

3. TTL expiry (baseline safety net)

The architectural pattern: use Kafka as the invalidation bus. Your log-consumer service already consumes log events — add a side-effect that publishes cache invalidation events to a cache-invalidation topic. The api-gateway subscribes and evicts matching keys.

Kafka (log-events) → LogConsumer → [process log] + [publish to cache-invalidation topic]
                                                          ↓
                                              API Gateway (cache-invalidation consumer)
                                                          ↓
                                              Evict L1 + L2 keys for affected service/window

Why not direct Redis DEL from the consumer? Because your consumer shouldn’t know about the cache topology of your query layer. Decoupling through Kafka means you can add a new caching tier without touching the consumer.

Failure mode: If the cache-invalidation consumer lags, caches serve stale data. Mitigate with a short TTL (60–120s) as the safety net, so even if invalidation events are delayed, staleness is bounded.

Read more

When log volume jumps from thousands to millions of events per second, a single database write path collapses. This platform treats Kafka as the durable backbone for multi-source ingestion, parallel consumption, duplicate-safe processing, compacted entity state, and live analytics—so operators see lag and throughput while the pipeline is still moving.

What We Built

• A Spring Boot service that produces application logs, infrastructure metrics, and transactional events into dedicated Kafka topics, with an outbox path for reliable transaction publishes

• Batch and partitioned consumers with enrichment, aggregation, sticky partition assignment, retries, and a dead-letter topic for poison messages

• Transactional dual-topic produces plus cache-and-database idempotency so redeliveries do not create duplicate processed rows

• Compacted entity-state topics with tombstones, materialization into JPA stores, and a Kafka Streams topology that feeds a WebSocket dashboard

Why This Matters

At Netflix- or Datadog-scale observability, logs are not “files”—they are continuous streams that must fan out to enrichment, alerting, and analytics without blocking the write path. Uber-style ride telemetry and Amazon order pipelines hit the same constraint: producers must stay fast, consumers must scale horizontally by partition, and “processed once” must survive restarts. The anti-pattern is a shared consumer group with a single partition and at-most-once commits—throughput looks fine until a crash silently drops events or a rebalance doubles them. Partition keys, consumer groups, and explicit idempotency are how high-scale platforms keep both latency and correctness under load.

System Design Deep Dive

Four patterns carry the architecture.

Multi-source producers with different durability knobs. Application-log producers can favor throughput (acks=1, lz4). Transactional and dual-topic paths use acks=all with producer idempotence. That split matches reality: not every shipper needs the same durability budget, but money-adjacent and audit events do.

Partitioning and sticky assignment. Events keyed by source land on a multi-partition topic; consumers use sticky assignment and concurrency so parallel workers stay on warm partitions across rebalances. Lag metrics expose whether you need more consumers or better keys—not more threads on one hot partition.

Exactly-once at the application boundary. Kafka transactions publish to raw-logs and audit-logs atomically. Consumers read with isolation.level=read_committed. An idempotency lock (processing → processed) plus a unique eventId in the database stops duplicates when the same record is redelivered. Streams itself uses at_least_once—a deliberate trade-off: windowed analytics tolerate recomputation better than inventory-style side effects do.

Compaction and streams for state and visibility. Entity updates go to a compacted topic; deletes are null-value tombstones. Materializers upsert or clear JPA and cache. Kafka Streams builds one- and five-minute window stores for endpoint, status, and region counts, queried interactively and pushed over STOMP to the dashboard.

Data Flow & State

Data path

HTTP ingest and scheduled generators publish into topics such as application-logs, log-events-partitioned, raw-logs, entity-state-compacted, and log-events. Batch consumers enrich and persist; failures after retries land on logs-dlq. Streams reads log-events and writes window aggregates to sink topics. Redis (or an in-memory fallback) holds aggregation counters and idempotency keys; H2 or PostgreSQL holds durable rows. Prometheus scrapes Actuator; the UI polls or subscribes for live KPIs.

Lifecycle states

Outbox rows move PENDING → SENT on successful Kafka send, or LOCAL when Kafka is disabled. Idempotency keys transition acquire → processing → processed, with release on failure so retries can proceed. Compacted entities cycle update → materialize; tombstones remove DB and cache entries. Cluster health reports HEALTHY/DEGRADED; Streams reports running or rebalancing.

Production Insight

The architectural lesson is that “exactly once” is a composition, not a broker checkbox. Transactions, read_committed, idempotent keys, and unique constraints together close the duplicate window; Streams at_least_once is acceptable for dashboards. Teams that flip one flag and ignore application locks still double-count under redelivery.

Github Link:

https://github.com/sysdr/sdc-java-p/tree/main/week_6_integrated_project/week_6_integrated_project

Implementation Walkthrough

Prerequisites

• Java 17+

• Maven 3.8+

• Docker (for Kafka, Zookeeper, Redis; optional PostgreSQL and Prometheus)

Build

From the project root:

mvn clean package
mvn test

Run

Local mode (H2, no brokers—REST still works):

mvn spring-boot:run

With Kafka:

docker compose up -d zookeeper kafka redis
mvn spring-boot:run -Dspring-boot.run.profiles=local,kafka

With Kafka Streams and the live dashboard path:

docker compose up -d zookeeper kafka redis
mvn spring-boot:run -Dspring-boot.run.profiles=local,kafka,streams

Full stack profile (PostgreSQL + Redis + Streams):

docker compose up -d
mvn spring-boot:run -Dspring-boot.run.profiles=docker

Verify

• App:

http://localhost:8080/

• API index: http://localhost:8080/api

• Health: http://localhost:8080/actuator/health

• Trigger the in-app load test via the cluster admin load-test endpoint documented in README.md

• Run the dashboard stream pipeline action on the UI (POST /api/dashboard/flow/run)

Success looks like: Actuator UP, non-zero producer/consumer counters after load, and dashboard KPIs updating over WebSocket when the streams profile is active.

Working Code Demo:

Observe

• Prometheus:

http://localhost:9090

• (scrape config in monitoring/prometheus.yml → /actuator/prometheus on port 8080)

• Dashboard WebSocket: SockJS /ws, topic /topic/metrics

• Key signals: producer success/error timers, consumer success/failure, DLQ count, idempotency hit/miss, state update/delete, Streams store health

Closing

You leave with a runnable map of how production log platforms combine Kafka durability, partition parallelism, transactional publishes, compaction-backed state, and stream analytics—and how to verify each layer under load rather than trusting a single configuration flag.

• A log transport layer that compresses, batches, and prioritises log payloads before sending

• Delta encoding so repeated log fields transmit only their changes

• An adaptive scheduler that reads available bandwidth and adjusts batch size and interval in real time

• A receiver endpoint that decompresses, reconstructs, and stores logs with delivery acknowledgement

• A dashboard showing transport efficiency metrics: bytes saved, compression ratio, queue depth

The Constrained-Network Reality

Yesterday you built a collector that buffers logs when the network disappears. Today the network is back, but it is slow, metered, or shared with operational traffic. Sending raw JSON logs over a 50 kbps GPRS link is not just inefficient; it actively competes with the control messages your devices depend on.

Stripe’s global payment terminals face exactly this. Many operate over cellular data with carrier-imposed byte caps. Their log shipping layer reduces payload size by 70 to 80 percent using field-level delta encoding combined with per-batch zlib compression. The insight is that consecutive logs from the same device share most of their structure: the same service name, the same host, the same log level prefix. Only timestamps, counters, and occasional message text change between entries.

Core Concepts

Delta Encoding for Structured Logs

Rather than transmitting each log as an independent JSON object, you record a reference frame and transmit only the fields that differ from it. A hundred consecutive INFO logs from payment-service on device-42 may share seven of nine fields. You send those seven fields once, then only two values per subsequent log.

Adaptive Batching

Fixed batch intervals waste bandwidth when logs are sparse and cause latency spikes when logs are dense. An adaptive scheduler measures the rolling average log rate and adjusts the flush interval so each batch targets a configurable payload budget, say 4 KB compressed. When the network is fast, smaller intervals keep latency low. When the link degrades, intervals widen automatically.

Priority Lanes

Not all logs justify bandwidth. ERROR and CRITICAL logs ship immediately on a high-priority lane with no batching. INFO and DEBUG logs enter the batching pipeline and may be held for seconds or shed entirely when queue depth exceeds a threshold. This mirrors how Cloudflare’s edge nodes handle telemetry: security events bypass the batch queue entirely and stream to the ingestion cluster as they occur.

Compression Strategy

zlib level 6 is the practical sweet spot for IoT: it achieves 60 to 75 percent compression on JSON log streams at roughly 1 MB/s throughput on a Raspberry Pi-class CPU. LZ4 is faster but compresses 15 to 20 percent less. For payloads under 200 bytes, skip compression entirely since the zlib header overhead cancels the gain.

Read more

• A self-tuning batch controller that reads live system metrics and continuously adjusts Kafka consumer batch sizes

• A load monitoring service synthesizing CPU load, heap usage, consumer lag, and processing latency into a single actionable load score

• A feedback control loop wiring metrics collection → load scoring → batch size adjustment → effect measurement

• Micrometer gauges exposing batch size dynamics so Grafana shows the system tuning itself in real time

Why This Matters

Yesterday you profiled the ingestion pipeline and found where time goes. Today we fix the most insidious bottleneck that profiling alone cannot reveal: the batch size you chose at deployment time is wrong by definition.

A static batch of 500 records performs beautifully at 5% CPU load. At 85% CPU — during a traffic spike, a GC storm, or a co-located service competing for cores — that same batch size turns your consumer into a latency landmine. Records queue up in Kafka, end-to-end latency inflates, and the downstream SLA breach arrives before any alert fires.

Netflix’s Mantis platform, Uber’s Kafka infrastructure team, and LinkedIn’s original Kafka consumers all converged on the same lesson: the optimal batch size is a function of current system state, not a constant. Static configuration is a performance anti-pattern disguised as simplicity.

Adaptive batching is the practical application of control theory to distributed systems. You are building a feedback loop — not writing configuration. That shift in mental model is the architectural insight this lesson delivers.

System Design Deep Dive

1. The Failure Mode of Static Batching

Static batch sizes embed a hidden assumption: the system operating point does not change. In production, it changes constantly. Batch sizing involves three competing objectives:

Objective Prefers Why Throughput Large batches Fewer broker round-trips, better compression ratios Latency Small batches Records processed sooner, lower end-to-end delay Stability Small batches Prevents OOM, GC pressure, processing timeouts under load

Static configuration can optimize for one operating point. Adaptive batching tracks the Pareto frontier across all three as conditions change.

Anti-pattern to avoid: picking a “safe” small batch size and calling it done. A batch of 10 records at 5% load processes ~10x fewer records per poll cycle than necessary, starving throughput and inflating Kafka consumer lag for no reason.

2. The Feedback Control Architecture

Adaptive batching is a closed-loop control system. The output (batch size) influences the system state (load), which feeds back into the controller input (load score):

[JVM + Kafka Metrics] → [LoadMonitorService] → [load score: 0.0–1.0]
                                                          │
[current batch size] ← [AdaptiveBatchController] ←───────┘
         │
[Kafka Consumer] ──→ [BatchMetrics] ──→ feeds back into LoadMonitor

The controller has three key parameters:

• MIN_BATCH_SIZE (floor, e.g. 10): prevents throughput collapse under extreme load

• MAX_BATCH_SIZE (ceiling, e.g. 500): prevents memory exhaustion during low-load periods

• Smoothing factor α (e.g. 0.30): exponential moving average that prevents oscillation

Proportional-only control without smoothing causes oscillation — the system overshoots, corrects too aggressively, overshoots again. The smoothing factor damps response and prevents thrashing between batch sizes on every poll cycle. Think of it as the derivative term in a PID controller.

3. Load Score Composition

A single scalar load score is more useful than reacting to individual metrics in isolation. CPU alone is a poor signal — a GC pause shows as 100% CPU, but the right response (reduce batch) is identical to high processing latency from an unrelated cause. Compositing metrics makes the controller robust to noisy individual signals:

loadScore = (cpuLoad            × 0.40)
          + (heapUsageRatio     × 0.30)
          + (normalizedLag      × 0.20)
          + (normalizedProcTime × 0.10)

Weight rationale: CPU and heap are the primary resource constraints for batch processing (0.70 combined). Consumer lag drives urgency — when lag grows, the system is falling behind regardless of local resource state (0.20). Processing time per record measures throughput efficiency directly (0.10).

Key architectural insight: Consumer lag is a leading indicator of downstream SLA breach. When lag accelerates, reducing batch size counter-intuitively helps: smaller batches complete faster, allowing the broker fetch loop to execute more iterations per second, catching up with lower per-iteration memory pressure.

4. Adjustment Strategy and the Dead Zone

if loadScore > 0.75:   # High load — shrink batch
    target = currentBatch × 0.80

elif loadScore < 0.40 and processingMs < TARGET_LATENCY_MS:
    target = currentBatch × 1.20  # only grow if latency budget allows

else:
    target = currentBatch          # dead zone — hold steady

# Exponential smoothing
newBatch = (int)(currentBatch × 0.70 + target × 0.30)
newBatch = clamp(newBatch, MIN_BATCH, MAX_BATCH)

The dead zone (0.40–0.75 load) is intentional and critical. Without it, the controller hunts continuously between grow and shrink on every evaluation cycle. In production, stability within an acceptable band is more valuable than chasing the theoretical optimum every 10 seconds.

5. Spring Kafka Integration — The Mechanical Challenge

Kafka’s ConcurrentKafkaListenerContainerFactory creates KafkaMessageListenerContainer instances at startup. Adjusting max.poll.records dynamically requires reaching into the container via the KafkaListenerEndpointRegistry, which Spring exposes as a managed bean.

The critical constraint: the new batch size takes effect on the next poll cycle, not mid-batch. This is correct behavior — you never want batch size changing mid-processing. The controller evaluates every 10 seconds via @Scheduled, which means batch size adjustments flow through on the next poll after the scheduled evaluation fires.

Trade-off: Running the controller at 1-second intervals would be more responsive but adds overhead on every evaluation cycle. At 10 seconds, the system can absorb a 10-second load spike before responding. Size this interval against your SLA recovery time objective.

Github Link :

https://github.com/sysdr/sdc-java-p/tree/main/day72/day72-adaptive-batching

Implementation Walkthrough

LoadMonitorService — Reading System State

The service collects four signals. The expensive operation — AdminClient consumer lag query — is TTL-cached at 5 seconds to avoid hammering the broker on every load sample:

@Service
public class LoadMonitorService {
    private volatile CachedLag lagCache = new CachedLag(0L, Instant.EPOCH);

    public SystemLoadSnapshot getCurrentLoad() {
        double cpuLoad   = osMXBean.getCpuLoad();
        double heapRatio = getHeapUsageRatio();
        long   lag       = getCachedConsumerLag(); // TTL=5s

        return SystemLoadSnapshot.builder()
            .cpuLoad(cpuLoad)
            .heapUsageRatio(heapRatio)
            .consumerLag(lag)
            .timestamp(Instant.now())
            .build();
    }
}

Failure mode: when AdminClient.listConsumerGroupOffsets() fails, the service returns a conservative high-lag sentinel, biasing the controller toward smaller batches. Always fail toward lower throughput, never toward OOM.

AdaptiveBatchController — The Control Loop

@Service
public class AdaptiveBatchController {
    private static final int    MIN_BATCH = 10;
    private static final int    MAX_BATCH = 500;
    private static final double HIGH_LOAD = 0.75;
    private static final double LOW_LOAD  = 0.40;

    private final AtomicInteger currentBatchSize = new AtomicInteger(100);

    @Scheduled(fixedDelay = 10_000)
    public void evaluate() {
        SystemLoadSnapshot snap = loadMonitor.getCurrentLoad();
        double score = computeLoadScore(snap);
        int    next  = applyControlLaw(score, snap.getLastProcessingMs());
        currentBatchSize.set(next);
        emitMetrics(score, next); // Micrometer gauge: log.consumer.batch.size.current
    }

    private int applyControlLaw(double score, long processingMs) {
        double raw = currentBatchSize.get();
        if (score > HIGH_LOAD)                             raw *= 0.80;
        else if (score < LOW_LOAD && processingMs < 200L) raw *= 1.20;
        int smoothed = (int)(currentBatchSize.get() * 0.70 + raw * 0.30);
        return Math.clamp(smoothed, MIN_BATCH, MAX_BATCH);
    }
}

AdaptiveLogConsumer — Closing the Loop

Each listener batch records its actual size and wall-clock processing time, feeding back into LoadMonitorService:

@KafkaListener(topics = "${kafka.topic.log-events}",
               containerFactory = "adaptiveBatchListenerContainerFactory")
public void consume(List<ConsumerRecord<String, String>> records, Consumer<?, ?> consumer) {
    long start    = System.currentTimeMillis();
    int  batchUsed = records.size();

    processBatch(records);

    long elapsed = System.currentTimeMillis() - start;
    // This single line closes the feedback loop
    loadMonitor.recordBatchCompletion(new BatchMetrics(batchUsed, elapsed, ...));
}

The AdaptiveBatchConfig wires a ContainerCustomizer that updates max.poll.records on the running container before each poll, pulling the current value from AdaptiveBatchController.getCurrentBatchSize().

Production Considerations

GC pressure and batch sizing: Large batches under memory pressure trigger G1GC concurrent marking, which appears as CPU spikes and feeds back into the load score as “high load” — correctly triggering batch reduction. This is the control loop working as intended.

Kafka broker alignment: MAX_BATCH_SIZE × avg_record_size must not exceed max.partition.fetch.bytes (default 1MB). At 500 records × 2KB average = 1MB exactly — validate this alignment or the broker silently truncates fetches, making your effective ceiling lower than configured.

What to alert on: Alert on currentBatchSize == MIN_BATCH sustained for >5 minutes — the system is overwhelmed and adaptive tuning has hit its floor. Alert on batch size oscillation (high variance over a 2-minute window) — indicates the smoothing factor is too low or the dead zone is too narrow.

Observability: The Micrometer gauge log.consumer.batch.size.current becomes the most informative single metric in your dashboard. Watch it alongside kafka.consumer.fetch-manager.records-lag-max and jvm.gc.pause.total — the correlation between GC pressure and batch size reduction should be visible within one controller evaluation cycle.

Implementation Guide :

This guide covers how to build, run, observe, and verify the Day 72 adaptive batching system. All commands assume you are in the project root directory (the folder containing docker-compose.yml, start.sh, and pom.xml).

Prerequisites

Tool Version Purpose Java JDK 17+ (21 recommended) Build Spring Boot modules Maven 3.8+ Package JARs Docker 24+ Run Kafka, Postgres, Redis, Prometheus, Grafana, apps Docker Compose v2+ Orchestrate the stack curl any Health checks and API calls python3 3.8+ Integration test helper scripts

Ensure these ports are free before starting: 8081, 8082, 3000, 9091, 5432, 6379, 9092, 2181.

Way 1 — One-command startup (recommended)

chmod +x start.sh stop.sh cleanup.sh load-test.sh integration-tests/test-adaptive-batching.sh
./start.sh

start.sh will:

1. Build all Maven modules (mvn clean package -DskipTests)

2. Start the Docker Compose stack with a fresh build

3. Wait until producer, consumer, Prometheus, and Grafana are healthy

4. Seed demo log events so dashboard metrics are non-zero

Way 2 — Manual step-by-step

# 1. Build JARs
mvn clean package -DskipTests

# 2. Start infrastructure and apps
docker compose up -d --build

# 3. Wait ~60–90 seconds, then verify health
curl http://localhost:8081/actuator/health
curl http://localhost:8082/actuator/health
curl http://localhost:9091/-/ready
curl http://localhost:3000/api/health

# 4. Send a test event
curl -X POST http://localhost:8081/api/logs \
  -H "Content-Type: application/json" \
  -d '{"service":"demo","level":"INFO","message":"hello","environment":"test"}'

# 5. Run integration tests
./integration-tests/test-adaptive-batching.sh

# 6. Run a load demo
./load-test.sh light    # batch size should grow
./load-test.sh spike    # light → heavy → recovery (best for Grafana)

Way 3 — Run apps locally (without Docker for Java services)

Use this when you want to debug Java code directly. Infrastructure still runs in Docker.

# Start only infrastructure
docker compose up -d zookeeper kafka kafka-init postgres redis prometheus grafana

# Terminal 1 — producer
cd log-producer
KAFKA_BOOTSTRAP_SERVERS=localhost:9092 mvn spring-boot:run

# Terminal 2 — consumer
cd log-consumer
KAFKA_BOOTSTRAP_SERVERS=localhost:9092 \
POSTGRES_HOST=localhost REDIS_HOST=localhost \
mvn spring-boot:run

Way 4 — Run unit tests only

mvn clean test

Tests cover SystemLoadSnapshot scoring and LogEventController request handling.

Stopping and cleaning up

Script What it does ./stop.sh Stops all project containers; keeps Docker volumes (data persists) ./cleanup.sh Stops stack, removes project volumes/images, prunes dangling Docker resources ./cleanup.sh --full Same as above plus docker system prune -a --volumes (removes all unused images/volumes)

./stop.sh
# or full cleanup before git commit / disk reclaim:
./cleanup.sh

Viewing output

Service health

curl http://localhost:8081/actuator/health   # Producer
curl http://localhost:8082/actuator/health   # Consumer

Raw Prometheus metrics

# Consumer adaptive batching metrics
curl -s http://localhost:8082/actuator/prometheus | grep log_consumer

# Producer throughput
curl -s http://localhost:8081/actuator/prometheus | grep log_producer

Prometheus UI

Open

http://localhost:9091

and query:

• log_consumer_batch_size_current

• log_consumer_load_score

• rate(log_consumer_records_processed_total[1m])

• rate(log_producer_messages_sent_total{status="success"}[1m])

Grafana dashboard (primary observability)

Item Value URL http://localhost:3000/d/day72-adaptive-batch Login admin / admin (local demo credentials) Refresh 5 s (live)

Dashboard panels:

1. Batch Size — Self-Tuning — log_consumer_batch_size_current (should move between 10–500)

2. Load Score [0–1] — log_consumer_load_score

3. Records Processed / sec — consumer throughput rate

4. Avg Batch Processing Time (ms) — per-batch latency

5. JVM Heap Usage % — consumer heap pressure

6. Producer Send Rate — events published per second

Application logs

docker compose logs -f log-consumer
docker compose logs -f log-producer
docker compose logs -f kafka

Working Code Demo:

Success criteria

Use this checklist to confirm the system is working correctly.

Startup

• curl http://localhost:8081/actuator/health returns {"status":"UP"}

• curl http://localhost:8082/actuator/health returns {"status":"UP"}

• ./integration-tests/test-adaptive-batching.sh exits 0 with all checks passed

• Exactly one container per service (docker ps --filter name=day72-adaptive-batching)

Data flow

• POST /api/logs returns 202 with "status":"queued"

• POST /api/logs/batch returns "queued": N matching batch size

• log_consumer_records_processed_total increases after sending events

• log_producer_messages_sent_total{status="success"} increases

Adaptive batching behavior

• log_consumer_batch_size_current is not stuck at 0 (initial value ~100)

• Under light load (./load-test.sh light), batch size grows toward max (500)

• Under heavy load (./load-test.sh heavy), batch size shrinks toward min (10)

• log_consumer_load_score stays in range 0.0 – 1.0 and responds to load changes

• Batch size changes appear in Grafana within one evaluation interval (~10 s)

Dashboard (non-zero, live values)

• Batch Size panel shows a line above zero

• Load Score gauge shows a value between 0 and 1

• Records/sec panel shows activity during ./load-test.sh

• Producer Send Rate panel shows activity during load test

• JVM Heap % shows a percentage (typically 15–40 % under load)

• Panels auto-refresh every 5 s without manual reload

Example passing metric snapshot

After ./load-test.sh light for 30+ seconds:

log_consumer_batch_size_current     > 100
log_consumer_load_score             between 0.0 and 1.0
log_consumer_records_processed_total > 0
rate(log_consumer_records_processed_total[1m]) > 0
rate(log_producer_messages_sent_total{status="success"}[1m]) > 0

Troubleshooting

Symptom Likely cause Fix Port already in use Previous stack still running ./cleanup.sh then ./start.sh Producer returns 500 Kafka not ready or Snappy/native lib issue Wait 60 s; rebuild with ./start.sh Grafana panels empty No traffic yet or Prometheus scrape delay Run ./load-test.sh light; wait 20 s Duplicate containers Stack started twice ./cleanup.sh then ./start.sh Prometheus on 9090 conflict Host Prometheus already bound This project uses host port 9091

Configuration reference

Key tuning parameters live in log-consumer/src/main/resources/application.yml under adaptive.batch:

Parameter Default Effect evaluation-interval-ms 10000 Control loop frequency high-load-threshold 0.75 Shrink batch when score exceeds this low-load-threshold 0.40 Grow batch when score below this min-size 10 Minimum max.poll.records max-size 500 Maximum max.poll.records target-latency-ms 200 Only grow if batch processing is faster than this

Quick reference

./start.sh                                          # start everything
./integration-tests/test-adaptive-batching.sh       # verify end-to-end
./load-test.sh spike                                # demo adaptive behavior
open http://localhost:3000/d/day72-adaptive-batch   # Grafana dashboard
./stop.sh                                           # stop (keep data)
./cleanup.sh                                        # stop + prune Docker

Scale Connection

Confluent Cloud dynamically tunes fetch.min.bytes and consumer throughput quotas based on broker-observed partition load — the same feedback principle at infrastructure scale. LinkedIn’s original Kafka design paper (2011) identified consumer batch sizing as the primary lever for throughput/latency trade-offs. Uber’s real-time data platform team published in 2019 that their consumer fleet auto-scales batch sizes across time zones as daily load patterns shift. What you are building today is a software-layer implementation of what cloud Kafka platforms implement at the broker layer.

Next Steps

Day 73 adds Redis caching layers so query-heavy log analytics stop hammering PostgreSQL on every dashboard refresh.

When Datadog surfaces an outage, engineers don’t read millions of lines—they rely on classification, clustering, correlation, forecasting, root-cause graphs, and fix recommendations working as one pipeline. Splunk’s ML Toolkit and Elastic’s Observability AI follow the same pattern: enrich first, then reason across signals. This lesson wires seven analytics layers into one platform.

By the end you’ll have:

• NLP enrichment extracting intent, entities, and sentiment from free-text logs

• ML classification predicting severity and category

• Clustering discovering patterns and flagging anomalies

• Cross-source correlation linking related events

• Ensemble forecasting with alert levels on response-time metrics

• Root-cause analysis with causal graphs on incidents

• Troubleshooting recommendations from similar past incidents

This project integrates analytics patterns from prior lessons without modifying those source directories.

Why this matters

• PagerDuty AIOps groups related alerts before paging—your correlation engine does this at log-ingest scale.

• Google Borgmon / Monarch forecast capacity from time-series—your ensemble forecaster mirrors that on response_time.

• Elastic ML clusters rare log templates—your KMeans + DBSCAN pipeline surfaces the same anomaly signals.

• New Relic incident intelligence ranks probable root causes—your RCA engine uses causal ordering and confidence scoring.

• GitHub Copilot for SRE suggests fixes from similar tickets—your recommendation engine uses TF-IDF similarity, not black-box magic.

Analytics is not one model—it is a staged pipeline where each layer adds context the next layer needs.

Core concepts

Architecture

Single FastAPI service on port 8095. All analytics modules run in-process—same coordination logic as a distributed production stack without extra containers for learning.

Read more

• Local SQLite log buffer that persists logs during network outages

• Connectivity monitor that detects online/offline transitions in real time

• Store-and-forward sync engine with priority-based flushing

• Central FastAPI receiver with deduplication and sequence reassembly

• React dashboard showing edge device sync status and buffer depth

The Problem Nobody Talks About

Most logging tutorials assume a stable network. Edge devices don’t get that luxury.

An Amazon Flex driver’s handheld scanner logs 400 package-scan events before hitting a dead zone between warehouses. A wind turbine sensor records vibration anomalies for 6 hours before the maintenance truck’s cellular modem comes back in range. A hospital medication cart logs 200 dispense events in an elevator with zero signal.

In all three cases, the logs still need to arrive at the central system—complete, ordered, and deduplicated.

This is the store-and-forward problem, and it’s fundamentally different from what Days 31–36 covered with RabbitMQ. There, the broker was always reachable. Here, the broker might not exist from the device’s perspective for hours.

Core Architecture

Three distinct zones interact in this system:

Edge Zone — The device itself. A lightweight Python agent runs continuously, writing incoming log events to a local SQLite buffer regardless of network state. Each log gets a monotonically increasing local sequence number and a device-scoped UUID.

Sync Engine — A background thread watches the connectivity monitor. The moment the network returns, it reads unsynced logs from the buffer ordered by priority tier (ERROR > WARN > INFO > DEBUG), batches them into compressed payloads, and ships them upstream. On acknowledgment, it marks records as synced.

Central Receiver — A FastAPI endpoint accepts batches from any number of edge devices. It runs a deduplication check using the device ID + sequence number composite key, merges out-of-order arrivals using the embedded timestamp, and commits clean records to the central log store.

How Connectivity Awareness Works

The connectivity monitor doesn’t use a ping to 8.8.8.8 — that’s brittle. Instead it attempts a lightweight HTTP HEAD request to the central receiver’s /health endpoint. This confirms both network reachability and that the receiver is operational.

State transitions matter here:

OFFLINE → ONLINE   : trigger immediate sync of entire buffer
ONLINE  → OFFLINE  : switch to local-only write mode, stop sync attempts
ONLINE  → ONLINE   : periodic sync tick (every 30 seconds)

The edge agent never drops logs in any state. The buffer is the source of truth on the device.

SQLite as the Edge Buffer

Why SQLite and not a file or in-memory queue? Three reasons:

1. Crash safety — WAL mode gives you atomic writes. If the device powers off mid-write, no corruption.

2. Priority queries — SELECT * FROM logs WHERE synced=0 ORDER BY priority DESC, seq ASC LIMIT 100 is ten characters of SQL, not 200 lines of custom heap code.

3. Deduplication — The device_id + seq unique constraint prevents double-writes on the device side.

Schema is deliberately minimal:

CREATE TABLE edge_logs (
    seq       INTEGER PRIMARY KEY AUTOINCREMENT,
    device_id TEXT NOT NULL,
    ts        REAL NOT NULL,
    level     TEXT NOT NULL,
    payload   TEXT NOT NULL,
    synced    INTEGER DEFAULT 0,
    priority  INTEGER DEFAULT 3
);

Sequence Reassembly at the Receiver

When three devices each send batches simultaneously after coming back online, the receiver sees interleaved sequence numbers. Tesla’s vehicle telemetry team solved this exact pattern: each vehicle is its own ordering namespace. You never sort across devices—only within a device’s own sequence stream.

The receiver maintains a per-device last_received_seq watermark. Gaps in the sequence trigger a 10-second wait window before committing, allowing retransmissions to fill holes. After the window, any still-missing sequences get flagged as “gap detected” in the central log metadata.

What You’ll See Running

Terminal shows the edge agent cycling through states — writing logs locally, detecting network loss, buffering silently, reconnecting, then flushing the backlog in priority order. The React dashboard shows each simulated device as a card: buffer depth, sync percentage, last seen timestamp, and connectivity state badge (green/red).

The central server logs show batch arrivals with deduplication counts — confirming that even if a device sends the same batch twice (network retry), records appear exactly once in the central store.

Integration with the Bigger System

This edge collector slots in directly below Week 25’s executive dashboard. The C-level views built on Day 176 now gain a new data source: edge devices that were previously invisible during connectivity gaps. The sync engine’s synced_at timestamp lets the dashboard distinguish real-time data from backfilled historical data — a distinction that matters enormously in operations analytics.

Day 178 builds on today’s sync engine by adding compression and delta encoding to the batch payloads, reducing the bytes transferred per log by up to 80%.

Github Link:

https://github.com/sysdr/course-p/tree/main/day177/day177_edge_log_collector

Implementation Guide Step-to-Step

Local edge log buffer with store-and-forward sync to a central FastAPI receiver, plus a React (Vite) dashboard served from the API when frontend/dist is built.

There are no API keys in this demo; run it only on trusted networks or behind your own auth/TLS.

Requirements

• Python 3.11+ recommended (3.12 works with the pinned deps in requirements.txt)

• Node.js 20+ and npm (only if you build or develop the dashboard)

• Docker and Docker Compose plugin (optional, for container run)

Install Python dependencies:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Build the dashboard

cd frontend
npm install
npm run build
cd ..

The API serves / from frontend/dist/index.html and bundles under /assets/ after a successful build.

Development server (alternate):

cd frontend && npm run dev

Uses

http://localhost:5173

and calls the API at

http://127.0.0.1:8100

.

Run (local)

From this directory:

./start.sh

• API + dashboard:

http://127.0.0.1:8100/

• OpenAPI: http://127.0.0.1:8100/docs

Stop:

./stop.sh

Tests

source venv/bin/activate
pytest tests/ -v --tb=short

Docker

cd docker
docker compose up --build

Compose builds from the parent directory (context: .. in docker-compose.yml). Runtime SQLite and logs use the mounted ../logs volume.

Tear down and free unused Docker disk:

./cleanup.sh

Cleanup script

cleanup.sh runs ./stop.sh, docker compose down for this project (when Docker is available), then docker container|network|image|builder prune to drop unused objects. Review flags before running on shared machines; it does not run docker system prune -a (no aggressive removal of all unused images).

Working Code Demo:

Success Criteria

By the end of this build you’ll have:

• Edge agent that writes logs to local SQLite with zero data loss during simulated network outages

• Sync engine that detects connectivity and flushes the buffer in priority order

• Central receiver that deduplicates and merges batches from multiple devices

• Dashboard showing per-device buffer depth and sync status in real time

Assignment

Extend the edge agent with a configurable retention policy: logs older than N hours that haven’t synced should be promoted to priority 1 (highest) and, if still unsynced after an additional M hours, written to a local compressed archive file and removed from the SQLite buffer to prevent unbounded disk growth.

Solution hints: Add a created_at column to the schema. Run a background task every 15 minutes that queries WHERE synced=0 AND created_at < (now - N_hours) and updates their priority. A second pass queries WHERE synced=0 AND created_at < (now - N+M_hours), serializes them to a .jsonl.gz archive using Python’s gzip module, then deletes those rows.

Next: Day 178 — Making log transport bandwidth-efficient with delta encoding and adaptive compression for constrained networks

• JVM profiling harness using async-profiler + JFR to find CPU/memory hotspots in the ingestion path

• Batching and compression tuning across the Kafka producer, consumer, and HTTP ingest layer

• Thread-pool right-sizing with measured queue saturation and latency percentiles

• Before/after benchmark dashboard — Prometheus metrics exposed so the improvement is quantifiable, not anecdotal

Why This Matters

Every distributed log pipeline eventually hits a performance wall. The wall isn’t random — it’s almost always the same three culprits: serialization overhead eating CPU, undersized thread pools causing artificial queuing, and naïve per-event I/O that could be batched. Netflix’s Vector team traced 40 % of their ingestion CPU to Jackson serialization before switching to a binary codec. Uber’s M3 pipeline saw 3× throughput gains purely from Kafka producer batch tuning without touching business logic.

The insight that separates senior engineers from mid-level ones: you don’t optimize what you guess — you optimize what the profiler proves. Premature optimization is a maintenance tax with no guaranteed dividend. Today we instrument first, measure under realistic load, identify the actual bottleneck, and then apply a targeted fix. Then we measure again. That loop — profile → hypothesis → change → validate — is the discipline that makes improvements stick.

System Design Deep Dive

1. The Ingestion Pipeline as a Series of Queues

Think of the pipeline as linked queues: HTTP thread pool → deserialization → Kafka producer internal batch buffer → Kafka broker → consumer thread pool → persistence write pool. Little’s Law governs every stage: throughput = concurrency / latency. When one stage saturates, back-pressure ripples upstream. Profiling tells you which queue is the bottleneck.

Anti-pattern: tuning all stages equally. This wastes engineering time and often creates new bottlenecks by overpowering downstream stages.

Read more

Production backends rarely process logs synchronously inside the HTTP request that generated them. Under load, tying observability to the critical path increases tail latency, hides failures behind timeouts, and makes it impossible to absorb traffic spikes without dropping data. Message queues decouple ingestion from processing—and this Spring Boot application shows how that decoupling composes into a deployable system.

What We Built

• A single Spring Boot 3.2 application (Week5IntegratedApplication) hosting seven composable messaging modules: RabbitMQ topic routing, Kafka producer semantics, consumer groups, manual acknowledgment with idempotency, content-based fan-out, dead-letter classification, and priority-tier processing

• Profile-driven infrastructure toggles (app.kafka.enabled, app.rabbitmq.enabled, app.redis.enabled) so the same JAR runs on a laptop with in-memory brokers or against a full Docker stack

• Resilience4j circuit breakers on Kafka producers and log processing, Spring Retry on transient failures, and SQL-backed DLQ records that survive broker outages

• An operations dashboard at

http://localhost:8080/

• aggregating Micrometer counters, module metrics, and a one-click demo that exercises every pipeline

Why This Matters

At Netflix, Uber, or Datadog scale, log pipelines ingest billions of events daily. No team writes directly to Elasticsearch from a servlet thread. Producers publish durable events; brokers buffer them; consumers work at a sustainable rate with explicit reliability contracts—acknowledgments, retries, dead-letter routing, and priority tiers. This project implements those contracts in one JVM so you can trace how they interact rather than studying each pattern in isolation.

System Design Deep Dive

Topic exchange routing decouples producers from topology. The RabbitMQ module publishes Day31LogEvent instances through RabbitTemplate when brokers are enabled, or through Day31InMemoryBrokerService locally—simulating logs-critical, logs-processing, and logs-monitoring queues with routing keys that encode severity and category. Producers never hard-code consumer topology; bindings decide destination—a direct analogue to production systems that hide queue names behind exchange configuration.

Kafka producer tuning exposes reliability trade-offs. Day32KafkaProducerService offers sync, async, and fire-and-forget send paths guarded by a Resilience4j circuit breaker. Producer config sets acks=all and idempotent sends via Week5KafkaConfig. A custom Day32LogPartitioner routes high-severity events to dedicated partitions—an operational concern when you need ordered processing per tenant without starving the rest of the topic.

At-least-once delivery requires business-layer idempotency. Day34LogProcessingService combines Spring Retry with hybrid deduplication via Day34IdempotencyKeyRepository and optional Redis SETNX. Committing an offset or ACK only tells the broker you took responsibility once; deduplication is what makes retries safe.

DLQ design needs taxonomy, not a single bucket. Day36DeadLetterQueueService classifies failures as transient (retry topic), permanent (DLQ), or ambiguous (retry then DLQ), always persisting to Day36DlqRecordRepository before optional Kafka mirroring—so operators query failure state even when the cluster is down.

Read more

• A C-level executive dashboard that pulls business KPIs from your distributed log processing pipeline

• Real-time revenue impact scoring, SLA compliance views, and system health indexes surfaced as board-ready metrics

• Role-segmented views: one backend aggregation engine, three audience lenses (CEO, CTO, CFO)

• React frontend with auto-refreshing metric cards and trend sparklines — no charting library needed

The Hidden Problem Nobody Talks About

Your distributed log system already knows which requests timed out, which payment flows failed, and which services are degrading. The tragedy is that this intelligence stays locked in Grafana dashboards that only on-call engineers open at 2 AM.

Executives make resource allocation decisions — hiring, infrastructure spend, roadmap priority — with almost no visibility into what the logs already know. Salesforce famously surfaced this problem internally when they discovered their support ticket volume spiked every Monday morning because their weekend batch jobs were silently corrupting user records. The signal was in the logs; it just never reached anyone who could act on it.

Today you close that gap.

Core Concept: Log-to-KPI Translation

Raw logs aren’t business metrics. The translation layer is what makes this lesson non-trivial.

What logs contain:
HTTP status codes, response times, user IDs, endpoint names, service identifiers, error payloads.

What executives need:
Availability percentage, revenue-at-risk from degraded services, customer-impacting incident count, SLA breach rate, mean-time-to-recovery.

The translation pipeline works in three stages:

1. Log Classifier — tags each log event with a business domain (payment, auth, checkout, search) and impact type (revenue-critical, ux-degrading, infrastructure)

2. KPI Aggregator — rolls up classified events into 1-minute, 15-minute, and 1-hour windows; computes percentages, counts, and weighted impact scores

3. Dashboard Renderer — serves role-specific views that filter and frame the same aggregated data for CEO (outcome), CTO (system), and CFO (cost/risk) audiences

Architecture

The system extends your Day 175 customer experience monitoring output. The log stream feeds a FastAPI aggregation service that maintains rolling window metrics in memory (using Python’s collections.deque). A React SPA polls the API every 15 seconds and renders three role-specific panels.

Key design choice: No time-series database. Rolling windows held in memory keep the implementation self-contained and fast for a single-node demo — exactly the pattern DoorDash used for their internal ops dashboard before they scaled it to Prometheus.

Read more

• Audit Event Consumer: Kafka consumer that ingests and deduplicates compliance-relevant events into a dedicated audit store

• Multi-Format Export Engine: Strategy-pattern exporters producing PDF, CSV, and JSON reports for regulatory consumption

• Scheduled Report Generator: Cron-driven GDPR, SOC2, and PCI-DSS report automation with async generation and status tracking

• Compliance Report API: REST endpoints for on-demand report generation, status polling, and authenticated file download

Why This Matters

Every company handling user data operates under compliance frameworks—GDPR, SOC2, PCI-DSS, HIPAA. When an auditor arrives (or a regulator sends a letter), the difference between a 3-hour response and a 3-week scramble is whether your compliance data is pre-collected and queryable or scattered across operational databases.

The naive approach—running ad-hoc SQL queries against your production database when audit season arrives—fails at scale. Uber processes 50M+ trips daily; their operational database can’t absorb analytical compliance queries. Netflix handles thousands of GDPR data-access requests monthly via dedicated compliance infrastructure. Stripe’s audit system is completely isolated from its payment processing path.

The pattern: stream audit events into a compliance-specific read model continuously, then generate reports asynchronously from that model. This decouples compliance data collection from report generation, and both from your operational systems.

System Design Deep Dive

1. Event Sourcing as Your Audit Foundation

Compliance reporting requires an immutable, append-only audit trail. Every user data access, deletion, authentication event, and admin action becomes an event with a unique ID, actor, timestamp, and outcome.

The critical property: audit events must be immutable after creation. This is non-negotiable for regulatory validity. Your audit store should never support UPDATE on event records—only INSERT and SELECT. Compliance frameworks treat mutable audit logs as evidence of tampering.

AuditEvent {
    id: UUID (Kafka message key - used for deduplication)
    userId, action, resourceType, resourceId
    eventTime, ipAddress, outcome, metadata
    serviceOrigin  // which microservice emitted this
}

Trade-off: Event sourcing means your compliance store grows indefinitely. Plan for partitioned tables by month and archival to cold storage after 7 years (the SOC2/GDPR retention floor).

2. CQRS: Separate Your Compliance Read Model

Read more

When Cloudflare ingests 10M+ HTTP logs per second, they don’t write one event at a time—they batch, compress, and skip duplicates before touching disk. Datadog caches hot queries in memory tiers. ClickHouse picks columnar storage because analytics scans columns, not rows. This lesson wires seven performance layers into one platform.

By the end you’ll have:

• Pipeline profiler with bottleneck detection

• Self-tuning adaptive batching

• L1 LRU query cache

• Row / columnar / hybrid storage selection

• Bloom filter deduplication

• Delta encoding with zlib compression

• Adaptive worker scaling

Why this matters

• Kafka producers batch messages because each network round-trip costs milliseconds—your adaptive batcher learns the sweet spot automatically.

• Redis sits in front of Postgres for hot keys—your L1 cache does the same for repeated log queries.

• S3 lifecycle rules move cold data to cheaper tiers—your storage optimizer picks row vs columnar based on access patterns.

• Cassandra uses bloom filters to skip SSTable reads—your bloom layer skips duplicate log fingerprints at ingest.

• InfluxDB delta-encodes timestamps—your encoder stores only what changed since the last baseline.

Performance is not one trick—it is a stack of small decisions that compound.

Core concepts

Architecture

Single FastAPI service on port 8095. All performance modules run in-process with SQLite persistence.

Read more

What We’re Building Today

• Parse raw application logs into structured user sessions

• Compute CX metrics: page load time, error encounter rate, session abandonment, funnel completion

• Build a Python 3.11 FastAPI backend that aggregates these metrics in real-time

• React dashboard showing live CX health at a glance

• Integrate cleanly into the log pipeline built across prior weeks

Why This Problem Is Hard

When Shopify’s checkout page added 300ms of latency during Black Friday 2021, it cost merchants millions before anyone noticed — because no single error fired. The signal was buried in the aggregate pattern of user sessions quietly abandoning mid-funnel. Standard error-rate monitoring saw nothing. Customer experience monitoring would have caught it in minutes.

Logs tell you what happened. CX monitoring tells you what the user experienced.

Core Concepts

Session Stitching — Log lines arrive with user IDs and timestamps but no session boundaries. We detect sessions by grouping events per user with a 30-minute inactivity timeout. This mirrors how Google Analytics and Amplitude define sessions.

Funnel State Machine — Every product has a conversion funnel (e.g., landing → product_view → cart → checkout → purchase). We model this as a state machine where each log event advances or exits a session’s state. Abandoned sessions are those that entered but never exited through purchase.

Percentile Latency, Not Averages — Average page load time hides the 5% of users on slow connections who see 8-second loads. We compute p50, p90, p95, p99 — the same metrics Stripe and Cloudflare publish in their status pages.

Error Encounter Rate — The fraction of sessions that hit at least one 5xx or client-reported JS error. A healthy system keeps this under 0.5%. Amplitude’s engineering team found that users who encounter even one error have 3× the churn rate.

Architecture

The system has three layers:

Ingestion Layer — A LogIngestor reads structured JSON logs from your existing pipeline (the Kafka/Redis stream from Week 5). Each log line carries: user_id, session_id, event_type, page, latency_ms, status_code, timestamp.

Computation Layer — A SessionAggregator maintains in-memory session windows per user. When a session closes (timeout or purchase), it emits a SessionSummary object. A MetricsComputer consumes summaries and updates rolling percentile buckets using a T-Digest algorithm — O(1) memory regardless of data volume.

Serving Layer — FastAPI exposes /metrics/cx returning the current CX health snapshot. React polls every 10 seconds and renders a dashboard with trend sparklines.

Data Flow

Raw Log Line
    → SessionAggregator (stitch by user_id + 30-min gap)
        → SessionSummary (funnel_stage, errors_seen, load_times[], completed)
            → MetricsComputer (rolling p50/p95/p99, error_rate, abandonment_rate)
                → Redis (5-min sliding window snapshots)
                    → FastAPI → React Dashboard

State transitions per session: STARTED → BROWSING → INTENT (cart) → CHECKOUT → CONVERTED or → ABANDONED at any stage.

Key Implementation Pieces

Session Aggregator — Keyed on user_id, emits session on 30-min idle or explicit session_end event. Uses a min-heap of expiry times to efficiently close stale sessions without scanning all active ones.

# Pseudo-code
class SessionAggregator:
    def ingest(self, log_line: dict) -> SessionSummary | None:
        session = self.active[log_line['user_id']]
        session.update(log_line)
        if session.is_expired() or log_line['event'] == 'session_end':
            return self.close(session)

T-Digest Percentile Tracker — A lightweight implementation of the T-Digest algorithm gives accurate percentile estimates with ~1KB of memory per metric. No need for heavy libraries.

Funnel Abandonment Detection — Each session carries its highest funnel stage reached. At close time, sessions that reached INTENT or CHECKOUT but not CONVERTED are flagged as high-value abandonments — the metric product teams care most about.

What the Dashboard Shows

Four metric cards: p95 Page Load, Error Encounter Rate, Funnel Completion Rate, Abandonment by Stage. A time-series sparkline for each updates every 10 seconds. Color thresholds turn cards amber/red when metrics breach SLO bounds — the same pattern Datadog uses for its SLO widgets.

Real-World Grounding

Amazon’s CX team famously measured that 100ms of added latency reduced revenue by 1%. Their monitoring pipeline is exactly this: session stitching from distributed logs → funnel analysis → real-time dashboards — all derived from logs, not from invasive client-side tracking that users block.

Integration with Week 25

This module consumes the same enriched log stream the A/B framework (Day 174) reads from. Tomorrow (Day 176), your executive dashboard will pull directly from the /metrics/cx endpoint you build today — giving C-level stakeholders a one-pager built on the same underlying data.

Github Link:

https://github.com/sysdr/course-p/tree/main/day175/day175_cx_monitoring

Implementation Guide Step-to-Step

Prerequisites

• Python 3.10+ (python3 or python3.11).

• pip inside a virtual environment (recommended).

Optional:

• Docker and Docker Compose Plugin for containerized runs.

Install and run (local)

From this directory (day175_cx_monitoring/):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Start the API and dashboard:

uvicorn src.api.server:app --host 0.0.0.0 --port 8175

Or use helpers (creates venv if needed, installs deps, runs tests, starts server):

chmod +x start.sh stop.sh cleanup.sh   # once
./start.sh

Stop:

./stop.sh

URLs:

• Dashboard and API root:

http://localhost:8175/

• Metrics JSON: http://localhost:8175/metrics/cx

• Health: http://localhost:8175/health

Tests

source .venv/bin/activate
python -m pytest tests/ -v --tb=short

Dependencies for tests (pytest, pytest-asyncio, httpx) are listed in requirements.txt.

Docker

From docker/ (Compose build context is the project root):

cd docker
docker compose up --build

The service listens on host port 8175 mapped to container port 8175.

Tear down and Docker cleanup

./cleanup.sh

This stops the local HTTP service, brings down this project’s Compose stack (removes local images built for cx-monitor), and runs non-destructive docker ... prune -f cleanups.

To also remove all unused Docker images globally (including non-dangling):

DOCKER_PRUNE_ALL_UNUSED_IMAGES=1 ./cleanup.sh

Remove local Python/ephemeral artefacts yourself when needed (rm -rf .venv .pytest_cache, delete *.pyc / __pycache__); see .gitignore for what should stay out of version control.

Success Criteria

By end of day, your system should:

• Correctly stitch 10,000 simulated log events into sessions

• Compute p95 load time within 2% of true value

• Show abandonment rate broken down by funnel stage in the React dashboard

• Handle new log lines with < 5ms processing latency per event

Your mission: build the monitoring layer that turns a wall of log lines into the user story your product team actually needs to make decisions.

Working Code Demo:

Next: Day 176 — Building executive dashboards that surface these CX metrics alongside infrastructure health for C-level views

• GDPRComplianceService — orchestrates multi-store deletion with saga pattern

• Kafka Tombstone Producer — crypto-shredding and null-record compaction for append-only logs

• Cross-store Erasure Pipeline — coordinated deletion across PostgreSQL, Redis, and Kafka

• Immutable Audit Trail — tamper-proof deletion records for regulatory proof

Why This Matters

GDPR Article 17 gives users the legal right to have their personal data erased — within 30 days, across every system that holds it. A €20M fine (or 4% of global revenue) is the cost of getting this wrong.

The engineering problem is deeper than it sounds. Your log processing system is built on append-only principles: Kafka topics don’t support deletes, PostgreSQL rows cascade across foreign keys you may not have mapped, and Redis caches duplicate data from a dozen upstream sources. The architecture you’ve built over the past 68 days was optimized for write throughput and durability — exactly the wrong defaults for a deletion workflow.

This is where distributed systems design gets legally consequential. Netflix, Spotify, and LinkedIn all had to retrofit GDPR compliance onto event-sourced architectures that were never designed for erasure. The techniques they developed — crypto-shredding, deletion sagas, and data lineage registries — are what we implement today.

Read more

Every production observability platform eventually discovers that storing logs is not a CRUD exercise—it is a distributed systems problem. Writes must survive node failures. Partitions must exist before traffic arrives. Coordinators must know who leads. Queries must fan out without blocking. And replicas that drift must repair themselves before operators lose trust in the data.

What We Built

• A single Spring Boot application hosting a complete distributed log storage stack: replication gateways, partitioned ingestion, consistent-hash routing, Raft leadership, gossip membership, scatter-gather queries, tunable quorums, and anti-entropy repair

• RocksDB-backed local storage with quorum replication (W=2 of 3 replicas) and Resilience4j circuit breakers on gateway paths

• PostgreSQL composite partitioning (daily ranges × 256 source-hash sub-partitions) with optional Kafka ingestion and batch writes

• An operations dashboard at / aggregating Micrometer metrics, cluster membership, and a one-click demo that exercises every subsystem

Why This Matters

At Netflix or Datadog scale, log platforms ingest billions of events per day. No single node holds the data. No single query scans every shard. The engineering challenge is composing replication, partitioning, consensus, and repair into something operators can reason about under failure. This project implements that composition in one JVM—sharing H2/PostgreSQL persistence, optional Redis topology, and Prometheus export—so you can see how production patterns interact rather than studying them in isolation.

System Design Deep Dive

Multi-node replication with quorum writes. The storage cluster persists LogEntry records in RocksDB and propagates ReplicationRequest payloads to peers via RestTemplate. A write succeeds only when storage.replication.write-quorum acknowledgments arrive within the configured timeout. The trade-off is explicit: lower quorum improves availability during partitions; higher quorum improves durability.

Consistent hashing for load balance. ConsistentHashRing places 150 virtual nodes per physical node using MurmurHash3, minimizing key movement when nodes join or leave—unlike naive modulo sharding that reshuffles everything. DistributionMetricsService exposes storage.distribution.balance_score so operators catch hot spots before they become incidents.

Raft for ordered leadership. RaftNode runs election timeouts (150–300 ms) and leader heartbeats on a dedicated ScheduledExecutorService—sub-second timing that Spring’s @Scheduled pool is not tuned for. Only the leader accepts writes; followers redirect until leadership stabilizes after failover.

Tunable consistency via quorums. QuorumService honors ConsistencyLevel (ONE, QUORUM, ALL) with version-vector conflict resolution. ONE minimizes latency; ALL maximizes durability at the cost of write availability during outages.

Distributed Log Storage Platform Architecture

Read more

• Ingest experiment event logs from the distributed pipeline

• Assign users deterministically to control/treatment groups via hash-based bucketing

• Compute statistical significance (z-test, Welch’s t-test) on conversion metrics

• Surface live results through a React dashboard with confidence intervals and lift %

• Auto-register every experiment that appears in the log stream

Why A/B Testing Lives in the Log Pipeline

When Booking.com runs 1,000 simultaneous experiments, the raw signal is log data—page views, clicks, bookings—streaming through their event pipeline. Attaching the analysis framework directly to that pipeline means zero ETL delay: results update as logs arrive, not the next morning after a batch job.

Our log processor already tags every event with user_id, session_id, timestamp, and event_type. We add two fields: experiment_id and variant. The rest is statistics.

Core Concepts

Deterministic Assignment Users must land in the same variant on every visit. A simple hash solves this: hash(user_id + experiment_id) % 100 < split_pct → treatment. No database lookup, no race condition, same result every time.

Metric Aggregation Per variant we track impressions, conversions, and revenue inside a time-windowed accumulator. The window matches the experiment’s active period so stale events don’t pollute results.

Statistical Testing Two-sample z-test for proportions (conversion rate) and Welch’s t-test for continuous metrics (revenue per user). p-value and 95% confidence interval update every 30 seconds. When p < 0.05 and the CI excludes zero, the result is significant.

Sequential vs Fixed-Horizon Fixed-horizon tests enforce a minimum sample size before results are readable—blocking the peeking bias that inflates false-positive rates. Sequential testing (Airbnb’s method) uses alpha-spending to let teams check daily without inflating error. We support both modes via a config flag.

Architecture

Log Stream ──► Experiment Router ──► Metric Aggregator
                     │                      │
              Assignment Store        Stats Engine (every 30s)
                                           │
                                    Result Store (SQLite)
                                           │
                                    FastAPI REST Layer (:8174)
                                           │
                                    React Dashboard

Read more

• RetentionPolicyEngine — a configurable, rule-based system that maps log categories to retention windows

• TieredStorageManager — automatic promotion/demotion of log data across hot → warm → cold storage tiers

• RetentionEnforcerJob — a scheduled, idempotent enforcement job that purges expired partitions safely

• PolicyMetricsCollector — observability layer reporting retention lag, purge throughput, and storage reclamation rates

Why This Matters

Every byte you store costs money. Every byte you keep longer than legally required is a liability. Data retention is where economics, compliance, and system design collide.

At Uber’s scale, the ride-events topic alone generates terabytes per day. Without automated retention enforcement, storage costs compound weekly. Netflix’s content access logs must be purged within defined windows to satisfy GDPR and CCPA — a manual process would require a dedicated team and still be error-prone.

The naive approach — a cron job that DELETEs rows matching WHERE created_at < NOW() - INTERVAL '90 days' — will destroy your database under production load. A full table scan locks rows, spikes I/O, and triggers autovacuum storms in PostgreSQL. The production pattern is fundamentally different: you don’t delete data, you expire partitions.

Tomorrow we layer GDPR “right to be forgotten” on top of today’s time-based retention. Today’s architecture is deliberately designed to make that addition surgical rather than invasive.

System Design Deep Dive

Read more

When Datadog ingests payment logs, access is not a free-for-all—RBAC gates who sees what. Stripe encrypts PAN fields at ingest; Splunk redacts before analysts search. GDPR erasure in Shopify removes a buyer’s trail across shards. This lesson wires seven security layers into one platform.

By the end you’ll have:

• JWT authentication with RBAC on log namespaces

• AES-256-GCM field encryption for PII

• Regex redaction before display

• Hash-chained immutable audit trails

• Automated retention policies

• GDPR right-to-erasure workflow

• SOX / HIPAA / GDPR compliance report export

Why this matters

• AWS CloudTrail records who touched which resource—your audit chain mirrors that immutability model.

• HashiCorp Vault encrypts secrets at the field level—same idea as encrypting email and ssn in log JSON.

• Elastic Security redacts PAN before indexing—your redaction engine runs on the read path.

• SOC 2 auditors ask for retention proof—your policy engine deletes or archives on schedule.

• Meta GDPR tooling erases user data across stores—your erasure coordinator tracks locations and verifies completion.

Security is not a checkbox—it is a pipeline every log byte passes through.

Core concepts

Architecture

Single FastAPI service on port 8094. All security modules run in-process with SQLite persistence.

Read more