Content-addressable storage slices files into chunks, hashes each one, and stores a single copy per hash. Fixed-size chunking — cut every 8KB — is brittle: insert one byte and every boundary downstream moves, changing the hash even though the content didn’t. Content-defined chunking picks boundaries from the bytes themselves using a rolling hash, like scanning for a fingerprint instead of counting to 8192. Only the chunk touching the edit changes; the rest match their old hashes.

Do you know whether your backup system uses fixed-size or content-defined chunking?

Most people assume a better dedup ratio just means smaller chunks. The reality: halving chunk size can shrink storage another 20–30%, but roughly doubles the chunk index — pushing metadata that once fit in RAM onto disk, and slowing every restore.

The full Deep Dive on content-addressable storage lands Week 7, covering how chunk-size tuning shifts cost across hot, warm, and cold tiers.

Instead of updating a block in place, the filesystem appends the new version to a running log and marks the old copy dead. Segments fill with a mix of live and dead blocks. A background cleaner finds the deadest segments, copies out what’s still live, and frees the rest — cleaning cost climbs fast as the disk fills, since less dead space remains per segment.

Do you know what percentage of your disk is currently marked as dead-but-not-yet-reclaimed?

Most people assume a log-structured filesystem is always faster than an in-place one. Under sustained random writes near capacity, segment cleaning can push effective write amplification to roughly 2 to 6x — the disk rewrites several bytes for every byte the app actually sent.

This topic is in the Deep Dive queue — the Scan gives you the tension, the full article gives you the production reality.

A local NVMe SSD replies in about 100 microseconds. Route that same read through iSCSI — SCSI commands wrapped in TCP — and you’re paying 1 to 2 milliseconds, ten to twenty times slower. Teams buy all-flash arrays expecting local-SSD speed, then wonder why p99 latency barely moved. The flash was never the bottleneck. The translation layer was.

NVMe was designed for drives sitting next to the CPU, so it skips the layers SCSI needs for tape drives and printers. NVMe-oF puts that same lean command set directly on the wire — over RDMA, TCP, or Fibre Channel — instead of repacking it into SCSI first. No translation step, either direction.

Diagram note: shows the failure moment (Part 2) as a before/after — one extra box (SCSI translate) is the entire gap between millisecond and microsecond storage. A flow diagram was chosen over a chart because the win is structural, not statistical: it’s about a hop disappearing, not a value shifting.

Do you know whether your storage fabric runs native NVMe-oF, or still translates every command back into SCSI at the target?

Most people assume NVMe-oF needs exotic RDMA-capable NICs to be worth it. The reality: NVMe/TCP over ordinary Ethernet now lands within roughly 20-30 microseconds of RDMA in many production stacks — close enough that most teams skip RDMA entirely.

This topic is in the Deep Dive queue. The Scan gives you the tension. The full article will give you the production reality.

Tomorrow: RDMA and kernel bypass.

That’s the whole tension. Block storage exposes a raw array of fixed-size blocks, addressed by number, and any block can be overwritten independently — this is what filesystems and databases are built on. Object storage exposes whole, versioned blobs addressed by key, and there is no partial-overwrite API — change one byte, and the system writes a brand new object. Think of block storage as a filing cabinet where you open one drawer without touching the rest; object storage is a sealed crate you must fully repack to change anything inside.

Do you know which of your write-heavy paths — logs, indexes, queues — are currently landing on object storage instead of block storage?

Most people assume object storage is just “cheaper, slower disk.” The reality is that its per-object write latency, roughly 100–200ms, comes almost entirely from metadata and index updates, not data transfer — so a workload doing many small updates can run an order of magnitude slower than the identical pattern on block storage, even though object storage’s raw throughput on large sequential objects is excellent.

This topic is in the Deep Dive queue. The Scan gives you the tension. The full article will give you the production reality.

Somebody on your team disaggregates storage and compute. The pitch is simple: scale each independently, save money on idle capacity, stop over-provisioning compute nodes just to keep disks nearby. Three months later, a query that used to take 40 milliseconds is taking 4 seconds, and nobody can find the bug, because there isn’t one. The system is working exactly as designed. It’s just answering a question nobody asked: is this workload one where you should send the computation to the data, or the data to the computation?

For thirty years, the answer was obvious. Data is heavy, code is light. You ship the query to the disk, not the disk to the query. MapReduce built an entire era of big data infrastructure on that one assumption. But disaggregation breaks the assumption it was built on, and most teams don’t notice until the bill or the latency graph tells them.

The Assumption That Stopped Being True

“Move compute to data” was never really a principle. It was a shortcut, and it only worked because moving data used to be slow. A 1 Gbps top-of-rack switch and a spinning disk made data movement the expensive operation by a wide margin, so you avoided it whenever you could. The rule of thumb calcified into architecture: colocate compute and storage on the same box, keep the network out of the hot path, and if you must talk to another machine, send it a small request and get a small answer back.

Disaggregated systems remove the box. Compute and storage now live on different machines connected by a fabric — NVMe over Fabrics (NVMe-oF), RDMA, or increasingly CXL for memory-level access — and that fabric is no longer the bottleneck it used to be. A modern RDMA link over 100 or 200 Gbps Ethernet moves data at a rate that starts to rival local memory bandwidth. The old shortcut doesn’t hold. Sometimes shipping the bytes really is cheaper than shipping the job.

This isn’t a story about the network getting “fast.” It’s a story about the cost curve for two different operations crossing each other, and the crossing point depending on the shape of the workload, not on some fixed rule you can write once and forget.

The Trade You’re Actually Making

Every remote read is really a decision between two paths, and the decision hinges on three things: how much data the operation touches, how selective the operation is, and how fast the fabric between the two sides actually is.

Path one: ship the function. You send a small piece of logic — a filter predicate, an aggregation, a scan bound — to wherever the data lives, and it runs there. Only the result crosses the network. This is what pushdown predicates, storage-side filtering, and “smart” SSDs with onboard compute are all doing. The cost is roughly one round trip plus execution time, and the payload over the wire stays tiny no matter how much data the operation touches internally.

Path two: ship the bytes. You pull the data across the fabric and run the computation wherever your compute pool is elastic that day. The cost is roughly payload size divided by fabric bandwidth, plus setup overhead. For a highly selective read — one row out of a billion — this is wasteful. For a large sequential scan or a join that touches most of a table anyway, this can win, because modern fabrics move sequential data fast enough to make the “expensive network” assumption obsolete.

Read more

Triple replication on a 10-petabyte dataset burns roughly 30 petabytes of raw disk. Switch to a (10,4) erasure code and the same data needs around 14 petabytes — cheaper, but one node failure now triggers a rebuild that reads ten surviving shards to reconstruct a single chunk. That storm is what bites teams during a second failure mid-rebuild.

Replication keeps full copies of every object — simple to read, expensive to store. Erasure coding splits an object into k data chunks plus m parity chunks, so any k of the k+m chunks rebuild the original — close to how RAID 6 survives two disk failures with two extra disks, not a full mirror. Rebuilding one chunk means pulling k chunks over the network, not one replica.

Do you know how many shards your storage reads across the network to rebuild one lost chunk right now?

Most people assume erasure coding is unambiguously cheaper to run. The reality: a single node failure in an EC(10,4) scheme can pull around 10x more rebuild traffic than a 3x-replicated system recovering the same data, since reconstruction needs k healthy shards instead of one copy.

The full Deep Dive on data locality in disaggregated systems is coming in Week 6, covering the rebuild-bandwidth math and where the erasure coding break-even point actually sits.

You write a value to memory. It’s there. You pull the power cord. It’s gone. That’s the deal you signed up for the day you learned what RAM stood for.

Persistent memory broke that deal on purpose. For about six years, you could buy a DIMM that plugged into a normal memory slot, ran at close to DRAM speed, and kept your data after a power loss. Intel called it Optane. It promised to close the biggest gap in computing — the one between how fast your CPU can think and how slow your disk can remember. Then, in 2022, Intel killed it.

This lesson is about why PMEM mattered enough to build, why it was hard enough to use that most teams never touched it, and why the idea didn’t actually die — it just moved into CXL, which is where we left off last time.

The 100,000x Gap Nobody Wanted to Admit

Here’s the number that started this whole category of hardware. A DRAM access takes roughly 100 nanoseconds. A flash SSD access, even a fast NVMe one, takes somewhere around 50 to 100 microseconds once you count the driver, the queue, and the device firmware. That’s a gap of roughly 500 to 1,000x. Spinning disk was worse — closer to 100,000x.

Every database, every cache, every distributed log was built to hide that gap. Write-ahead logs batch writes so you fsync less often. Caches try to keep hot data out of the gap entirely. B-trees and LSM-trees, which we covered earlier in this act, exist largely because random access to that slow tier is so expensive that you have to be clever about when you touch it.

All of that cleverness is a workaround. It assumes the gap is fixed and permanent — a law of physics you design around, not a problem you solve. PMEM was the first serious attempt to just shrink the gap instead. Optane’s read latency landed around 300 to 400 nanoseconds — call it 3 to 4x slower than DRAM, not 500x. Write latency was close to DRAM on writes, because unlike flash, there was no erase-before-write cycle to hide.

Suddenly the question wasn’t “how do I avoid touching slow storage” — it was “what if slow storage was fast enough that I didn’t have to avoid it.”

How PMEM Lets the CPU Skip the Storage Stack Entirely

A normal write to disk goes through a stack: your application calls write(), the kernel copies data into a page cache buffer, the block layer schedules an I/O, the driver talks to the device, and eventually bytes land on flash. Every one of those layers adds latency, and every one exists because the CPU can’t directly address anything on the other side of a block device.

PMEM removes the “block device” part of that sentence. It sits on the memory bus, addressed by the CPU the same way DRAM is — you can point a regular pointer at it and dereference it. Intel’s App Direct mode exposed PMEM as a memory-mapped region: your application mmap()s a file backed by the PMEM device, and from that point on, reading and writing it is a plain load or store instruction. No syscall, no page cache copy, no block layer, no driver queue.

This is the part that made people excited: you could build a hash table, a log, or a B-tree that lived directly in persistent memory, with no serialization step and no separate “save to disk” phase. The data structure and its durable copy were the same bytes.

Read more

Every SSD keeps a flash translation layer — a lookup table matching the logical address your OS asks for to the physical NAND cell it actually lives on. That table sits in DRAM for speed. Cut power completely and the table is gone with it. Power back on, and the drive has to replay checkpoint logs off the NAND itself before it can safely serve a single read — like firing the librarian every night and re-shelving the whole building each morning.

Do you know how many seconds your storage array takes to reach steady-state latency after a cold power-on?

Most people assume powering off an idle array saves energy. The reality is a full power cycle forces an FTL rebuild plus a garbage-collection catch-up burst — a spike that can pull more peak power in the first two minutes than the drive’s low-power idle state, roughly 2–5mW, would have used over several hours.

This topic is in the Deep Dive queue. The Scan gives you the tension. The full article will give you the production reality.

The fix is treating metadata as its own sharded, distributed problem, separate from where the actual bytes physically sit. Object keys get hashed and spread across many metadata partitions, each backed by an LSM-tree-style store built for fast writes and range scans. A request first resolves through this metadata layer to find which physical node holds the data, then fetches the bytes as a second, separate step. It’s like a library catalog sitting apart from the shelves — you query the catalog, then walk straight to the exact shelf instead of searching every aisle

Do you know whether a LIST or rename operation in your object store is O(1), or does it slow down as the bucket grows?

Most people assume renaming an object in cloud storage is instant. The reality: it’s usually implemented as copy-then-delete under the hood, so renaming a 50GB object can take as long as re-uploading it — and if the process fails halfway, you can end up paying for both the old and new copies.

The full Deep Dive on metadata sharding at object-store scale is coming in Week 6. It covers how partition splitting and rebalancing work when a single shard gets too hot.

A page near your promotion threshold migrates twice in one scan interval — up to DRAM, back down, up again — each trip copying it across the interconnect you’re trying to protect. At roughly 3 accesses per epoch, a page hovering between 2 and 4 migrates almost every cycle. The system built to save bandwidth ends up spending more of it than the workload alone ever would.

Exclusive tiering treats a page as living in one place — promoting it evicts the slow-tier copy, so every crossing pays full migration cost. Copy-on-promote copies the page up but leaves the original untouched below for a grace period. If access reverses, the system discards the fast-tier copy and reads the original — no re-migration.

Do you know what fraction of pages promoted in your memory tiers this week were demoted again within a single scan cycle?

Most people assume moving a page to a faster tier is pure upside. The reality: it now occupies capacity in both tiers until reclaimed — near full capacity, that trades thrashing for memory pressure instead.

The full Deep Dive lands in Week 15, covering the reclaim-rate race that turns this safety net into an OOM risk.

Tomorrow: eBPF kernel observability.

Take a 128-core box running a thread-heavy service. Page faults spike under load, and throughput doesn’t scale — it falls off a cliff around 32 threads. Every core is waiting on the exact same lock, even though they’re touching completely different parts of memory. You added cores. You didn’t add concurrency.

The bottleneck is usually a single lock guarding the whole address space, not the memory itself. A concurrent interval skiplist fixes this by keying each memory region (a VMA — virtual memory area) as a range in a probabilistically balanced, multi-level list. Threads walk down through skip pointers toward the range they actually need, so two threads touching different regions never fight over the same node. It’s like a building with express elevators to different floors instead of one lobby line for everyone.

Do you know whether your runtime or kernel still takes one global lock on every page fault, regardless of which memory range is touched?

Most people assume more cores just means more throughput. The reality is that Linux held a single mmap_lock semaphore for the entire address space on every fault until per-VMA locking landed around kernel 6.4 in 2023 — before that, threads touching unrelated memory still queued behind each other, so adding cores could make things slower, not faster.

The full Deep Dive on concurrent interval skiplists is coming in Week 14. It covers how range deletion and rebalancing stay lock-free under live traffic.

Tomorrow: eBPF kernel bypass.

What the diagram shows: the left panel is the failure mode (Part 2) — every thread converges on one mmap_lock, so contention scales with thread count, not with how much of the address space they’re actually touching. The right panel is the mechanism (Part 3) — the skiplist lets threads branch toward their own range and never meet. A before/after layout was chosen because the whole tension of this Scan is “same threads, same memory, wildly different throughput,” and that’s a contrast, not a sequence.

You solved the capacity problem. Then a new one showed up wearing the same clothes.

Pool your memory across hosts with CXL and the spreadsheet math looks great. Ten machines, each with 128 GB of local DRAM sitting half-idle, now share a 2 TB pool that any of them can reach. Utilization goes up. Procurement is happy. Then week two arrives, and one host’s p99 latency triples for no reason anyone can find in its own metrics. It isn’t out of memory. It isn’t out of CPU. It’s waiting — for a memory controller that three other hosts are also hammering at that exact moment.

This is the part the CXL pitch decks skip. Pooling memory doesn’t remove contention, it relocates it. You went from “my memory is mine” to “my memory is a resource other people can starve me out of,” and now something has to decide, every few microseconds, whose request goes first. That something is the remote memory scheduler, and how it makes that decision is the actual engineering problem hiding behind the marketing term “disaggregated memory.”

Why More Capacity Didn’t Buy You More Headroom

Local DRAM has one consumer: the CPU socket it’s wired to. Contention exists, but it’s bounded — you’re fighting your own cores, and the memory controller knows exactly who’s asking.

Pooled memory has an unknown number of consumers, and they don’t coordinate. Host A doesn’t know Host B just kicked off a batch job that’s streaming 40 GB through the shared fabric. From Host A’s point of view, its own workload didn’t change, its own queries didn’t change, but its memory access latency just went from roughly 100 nanoseconds to somewhere north of 300. The capacity is there. The bandwidth to reach it, at the moment you need it, is not guaranteed — it’s shared, and sharing without arbitration is just contention with better marketing.

This is the same failure mode you’d get if ten teams shared one database with no connection pooling, except it happens at the nanosecond timescale of memory access instead of the millisecond timescale of a query. You can’t put a human in that loop. You need something making the call before the request even leaves the CPU.

The Tiering Decision Nobody Sees

The scheduler’s actual job is deciding, continuously, where each page of memory should physically live: in fast local DRAM, or out in the slower shared pool.

It does this with a hotness signal — access counters, sampled reference bits, sometimes hardware telemetry from the CXL controller itself — that estimates which pages get touched often and which sit cold. Hot pages get migrated toward local DRAM, where access is fast and uncontended. Cold pages get pushed out to the remote pool, where latency is worse but nobody’s paying that latency cost anyway because nobody’s reading them.

Layered on top of that is bandwidth arbitration at the fabric level. The CXL switch, or the software layer managing it, partitions the pool’s available bandwidth across the hosts attached to it — something like a QoS class per host or per device, so one runaway consumer can’t starve the rest. Think of it as two separate scheduling problems stacked on each other: one deciding where a page lives, the other deciding how much of the pipe each device gets to reach the pages that live remotely. Get either one wrong and the other one’s work doesn’t matter.

Read more

CXL pooling works by carving a shared memory device into address ranges, then binding each range to one host through Bind and Unbind commands issued by the Fabric Manager. The switch itself just forwards requests — it checks whether a request’s address falls inside the range currently bound to that host. Think of the switch as a mail room that only delivers to the box number it’s told is yours; it never asks whether the clerk wrote the right number down. If the Fabric Manager’s bookkeeping is wrong, the switch enforces the wrong boundary with total confidence

Diagram note: the left panel shows a correctly partitioned CXL memory pool — two hosts, two non-overlapping Device Physical Address ranges. The right panel shows the failure moment: a Fabric Manager bookkeeping error hands overlapping ranges to both hosts, and the switch enforces that overlap as if it were correct. This is shown visually rather than described in text because the failure is spatial — you have to see the ranges collide to understand that the switch itself never detects anything wrong.

Do you know whether your CXL pool’s Fabric Manager runs on hardware-attested firmware, or as an ordinary software process outside your trust boundary?

Most people assume CXL memory isolation is a hardware guarantee, like ECC or a TLB lookup. The reality is that in CXL 2.0 deployments, the entire tenant boundary lives in Fabric Manager software — hardware-enforced range integrity only arrives with CXL 3.x features that most production pools haven’t turned on yet.

The full Deep Dive on CXL memory isolation is coming in roughly Week 4. It covers how CXL 3.x’s hardware-enforced integrity changes the trust model, plus the Fabric Manager audit checklist teams use in production.

Your Kafka broker runs fine for six hours. Then throughput drops by half and P99 latency jumps from 2ms to 40ms. Nothing crashed. No alert fired for disk failure. You check iostat and see the drive is busy — really busy — doing work you didn’t ask it to do.

That work is garbage collection, and it’s the SSD doing something you can’t see, on your behalf, whether you want it to or not. Every flash drive you’ve deployed for logs, write-ahead logs, or LSM-tree storage engines has been quietly fighting this problem. Zoned Namespace SSDs, ZNS for short, are the first mainstream fix that doesn’t just make the fight faster — it removes the fight entirely, by handing you control the drive used to keep for itself.

Here’s the thesis worth sitting with: ZNS isn’t a faster SSD. It’s a narrower one. It trades general-purpose flexibility for a guarantee that matches exactly how append-only systems already write data. That narrowing is what makes it fast, and it’s also what makes it hard to adopt — you can’t just drop it in.

The Drive Has Been Lying About Being Empty

A normal SSD lets you write to any logical block address at any time, just like a hard drive. Underneath, that’s a fiction. Flash memory can’t overwrite a page in place — it has to erase a whole block first, and erase blocks are much bigger than the pages you write. So the SSD’s firmware, called the Flash Translation Layer or FTL, maps your logical writes to physical locations, and quietly relocates old data to free up erase blocks. This background relocation is garbage collection, and the ratio of physical bytes written to logical bytes you asked for is the write amplification factor, or WAF.

For random-write workloads, WAF of 3 to 5 is common. Every byte you write, the drive writes three to five bytes internally. That extra writing burns through flash endurance — each cell tolerates a few thousand program/erase cycles before it wears out — and it steals bandwidth from your actual I/O. The FTL doesn’t coordinate with your application. It doesn’t know that your LSM-tree compaction just made a huge swath of the drive logically dead. It finds out the slow way, by scanning, and it picks the worst moment to catch up.

Append-only workloads make this worse, not better, in a subtle way. You’d think sequential writes would be gentle on an SSD. They are, for the write itself. But deletes in an LSM-tree or a Kafka log aren’t sequential — they’re compaction and retention, deleting old segments while new ones keep landing. The FTL sees a drive that looks randomly fragmented even though your application logic was perfectly orderly.

Zones Turn the Drive Into the Log You Already Wanted

ZNS restructures the drive’s address space into zones — typically 96MB to a few hundred MB each — and imposes one rule per zone: you can only write sequentially, starting from a write pointer that advances with each write, and you can only reclaim space by resetting the entire zone at once. No random writes within a zone. No partial invalidation. You either haven’t written to a byte yet, or you’ve written it and it stays until the whole zone resets

This sounds like a restriction, and it is — but it’s a restriction that matches what an LSM-tree, a write-ahead log, or a Kafka segment already does. Your application was writing sequentially and deleting in bulk anyway. ZNS just asks you to say so up front. In exchange, the FTL’s job collapses. There’s no indirection table mapping arbitrary logical addresses to arbitrary physical ones, because within a zone, logical and physical order match. There’s no background garbage collection scanning for dead pages, because you tell the drive exactly when a zone is dead — you reset it. The drive doesn’t guess. You decide.

The practical effect: WAF drops toward 1.0. You write a byte, the drive writes a byte. Endurance goes up because the drive isn’t multiplying your writes internally. Latency variance drops because there’s no surprise background compaction competing with your foreground I/O for the same NAND die.

The Write Pointer Doesn’t Forgive Mistakes

This is where ZNS stops being a free upgrade and starts being an architecture decision. The sequential-write-only rule doesn’t live in the drive’s convenience layer — it’s enforced at the protocol level. Try to write out of order within a zone and the command fails. Your application, not the FTL, now owns placement, and that ownership comes with new failure modes nobody warns you about until they hit staging.

The first one is active zone limits. A drive only supports a limited number of zones open for writing at once — often somewhere in the dozens, not hundreds. If your storage engine wants to write to more zones concurrently than the drive allows, you queue, or you redesign your write path to batch differently. This is invisible in a small test and very visible under real multi-tenant load.

The second is the gap between zone capacity and zone size. A zone might be nominally 256MB but only accept 240MB of actual writes — the difference is reserved for the drive’s internal parity and reliability mechanisms. Applications that assume the full nominal size overshoot and get write failures at a boundary they didn’t know existed.

The third, and the one that causes the most production pain, is that a zone reset is all-or-nothing. You can’t reclaim 80% of a zone because 80% of its data is dead — you wait until the whole zone is dead, or you copy the surviving 20% out first. This means your compaction and garbage collection logic, the stuff you already wrote for your LSM-tree, needs a second layer that’s aware of zone boundaries, not just key ranges. Get this wrong and you either waste capacity holding onto zones that are mostly dead, or you spend write bandwidth copying live data around — which is exactly the write amplification you adopted ZNS to avoid.

What Write Amplification Near 1.0 Actually Looks Like

In production, the signature of a healthy ZNS deployment is boring, and boring is the point. WAF sits close to 1.0 instead of drifting up over the drive’s lifetime as a traditional SSD fills and fragments. P99 write latency stays flat under sustained load instead of showing periodic spikes every few minutes when garbage collection kicks in — those spikes on a conventional SSD often show up as latency that’s 10 to 20 times the median, lasting a few hundred milliseconds at a time. Drive-reported wear leveling metrics, visible through nvme smart-log, show program/erase cycle counts climbing roughly in step with your actual logical writes instead of running well ahead of them.

On the tooling side, you’ll be working with libzbc or the Linux kernel’s native zoned block device support, and if you’re not writing your own storage engine from scratch, you’re likely picking up ZNS through something that already speaks it — RocksDB has a ZenFS backend, and a few Kafka-adjacent log stores have added zoned support. Most teams don’t hand-roll zone management; they adopt a storage engine that’s already made the trade-offs described above, and their job becomes choosing zone size and active zone limits that match their write concurrency, not writing an FTL replacement themselves.

Where Zones Fit in the Stack You’re Already Building

ZNS is a hardware-level expression of an idea you’ve already met if you’ve read the LSM-tree lesson in Act 2: separate sequential writing from random reads, and let compaction do the reorganizing work in bulk instead of in place. ZNS just moves that idea down a layer, so the drive itself refuses to let you violate it. It also pairs naturally with NVMe-oF from the last lesson — a disaggregated storage target built on zoned drives gets both the network flexibility of shared storage and the endurance profile of a drive that’s never fighting its own garbage collector.

The open question worth carrying forward: as more of the storage stack becomes zone-aware by default, does the FTL disappear entirely, or does it just move up into your application, where you now own decisions the drive used to hide from you? Either way, you’re the one making them now.

Next in this series

RDMA picks up the networking side of the low-latency storage story — how you read and write memory across a network without the kernel touching every packet, and why that matters once your zoned drives are sitting behind a fabric instead of in the same chassis.

Track A, Act 4, Lesson 6 — RDMA: Bypassing the Kernel for Microsecond Access — available now in the paid archive.

There’s no controller routing those writes. The client itself RDMA-writes the same value to two or three remote nodes in parallel, using one-sided operations that skip the remote CPU. It waits for a quorum ack — say 2 of 3 — before telling the app the write succeeded. Recovery means picking any surviving node; nobody elects a leader first.

Do you know how many independent failure domains your writes land on before your app gets an ack?

Most people assume disaggregating memory makes recovery harder, since state now lives elsewhere. The reality is recovery gets faster — a client already holding RDMA connections to 3 nodes can fail over in microseconds, with no coordinator to re-elect first.

This topic is in the Deep Dive queue. The Scan gives the tension; the full article gives the production reality.

To stop that, the migration is treated like a transaction, not a memcpy. The kernel write-protects the page, so any write during the copy triggers a fault instead of landing silently. It copies the bytes, flips the page table entry to the new location, then replays the faulted write on top of the copy. Every core with that page cached in its TLB (the CPU’s map of virtual-to-physical addresses) has to be told to drop the old mapping before the swap counts as done.

Do you know how many page migrations per second your CXL tier triggers under peak load, and how many of those stall on a write fault?

Most people assume moving a page between memory tiers is basically a fast copy. The reality is that a single migration includes a TLB shootdown across every core touching that page — a synchronous stop-the-world moment measured in microseconds — and a hot enough working set can spend more time migrating than it saves by living in fast memory.

The full Deep Dive on CXL tiered memory is coming in Week 14. It covers the hotness-tracking heuristics that decide which pages get migrated in the first place.

Tomorrow’s Scan continues Act 4 — Track A. (Forward link omitted — connection field not provided in inputs.)

Load testing won’t save you here. Synthetic traffic is a guess dressed up as a number. It hits the endpoints you remember to test, with payloads you imagined, in a distribution you invented. Real traffic has the malformed header from a five-year-old mobile client someone forgot to deprecate, the request from a user with 40,000 items in their cart, the timezone edge case that only exists twice a year. You cannot invent your way to those inputs. You have to watch them happen.

Shadow traffic solves this by duplicating real requests and sending the copy somewhere the user never sees. The original request goes through the normal path and gets the normal response. The copy goes to your new service, and its response goes nowhere except a log. The user experiences nothing. You get a live diff between old and new behavior, generated from actual production entropy instead of your imagination.

The Gap Between “Works in Staging” and “Works”

Staging lies to you in a specific, predictable way: it has staging traffic. Staging traffic is traffic your team generated on purpose, which means it only exercises the paths your team thought to exercise. Production traffic is generated by everyone else, which means it exercises everything.

This gap matters more as systems get more stateful and more interconnected. A pure function is easy to test — give it inputs, check outputs, done. A service that reads from three caches, calls two downstream APIs, and makes a decision based on a feature flag rollout percentage is not a pure function. Its behavior depends on the state of the world at the moment the request arrives, and staging’s world is not production’s world. The only traffic that reliably reproduces production’s world is production’s traffic.

Canary deployments get you part of the way there — route 5% of real users to the new version, watch the error rate. But a canary still puts real users at risk. If your new recommendation service crashes on cart sizes over 10,000, a canary finds that bug by breaking a real customer’s checkout. Shadow traffic finds the same bug with zero blast radius, because the response from the new service is never returned to anyone.

How the Fork Actually Happens

The mechanism is simpler than people expect. At the proxy or mesh layer — the same layer we covered in the last lesson — a rule intercepts matching requests and fires two copies. The primary copy proceeds normally: it goes to the current production service, and its response goes back to the client on the critical path. The mirror copy goes to the shadow service asynchronously, off the critical path, and its response is discarded or logged for comparison. The client’s request latency should never include any part of what happens to the mirror.

Read more

A circuit breaker at the edge watches error rates per PoP (point of presence), not globally. Past a threshold, that PoP flips from closed — traffic flows — to open — requests fail fast — for a fixed window, usually 5 to 30 seconds. Then it goes half-open and probes with one request before reopening. One guard per entrance, deciding alone.

Do you know whether your edge layer trips per region, or does one slow origin get hammered by every PoP at once?

Most engineers assume circuit breakers share state across your stack. The reality: edge breakers almost always trip per-PoP with no shared state — one region can stay wide open while others keep hammering a failing origin until config propagates.

The full Deep Dive on edge circuit breaking arrives Week 1. It covers sizing per-PoP thresholds so one region’s failure doesn’t get amplified by the rest.

Tomorrow: exactly-once delivery.

Somewhere in your company there’s a spreadsheet. It maps tenant IDs to database shards. Someone updates it by hand when a new shard goes live. Every service that talks to the database has a copy of the routing logic baked in — a if tenant_id % 16 == shard_id buried in a config file, or worse, in application code across forty microservices.

Then you need to add shard seventeen. Now you’re not deploying a database change. You’re coordinating forty deployments, in the right order, hoping nobody ships stale routing logic that sends a write to the wrong place. This is the actual failure mode, not a hypothetical one: a service holds onto an old routing table for six hours after a resharding event, and it quietly writes rows to a shard nobody reads from anymore.

Proxy-based sharding exists to make this someone else’s problem — specifically, one dedicated layer’s problem, instead of every service’s problem at once.

Where Consistent Hashing Stops Helping

Consistent hashing, which we covered last time, solves the math. It tells you which shard a key belongs to, and it keeps that mapping stable when you add or remove nodes. What it doesn’t solve is where that logic lives.

If every client library embeds its own copy of the hash ring, you’ve distributed a single source of truth across every codebase that touches the database. Each one needs the same ring version, the same shard count, the same virtual node configuration. Drift between them isn’t a bug you catch in code review — it’s a silent data placement error that surfaces weeks later as a customer complaining their data “disappeared.”

A proxy collapses that fan-out. One layer owns the routing table. Every client talks to the proxy the same way it would talk to a single database. The proxy decides, on every single query, which shard actually gets it.

One Door, All the Shards Behind It

Picture a connection pooler that also happens to know your data layout. That’s roughly what a sharding proxy is. It speaks your database’s wire protocol — MySQL’s protocol, Postgres’s, whatever your clients already expect — so nothing upstream needs a new driver or a new SDK.

When a query comes in, the proxy parses just enough of it to extract the shard key. Usually that’s a WHERE tenant_id = ? clause, or a value bound to a known parameter position. It runs that key through the same consistent hashing scheme from last lesson, gets back a shard ID, and forwards the query to the right backend connection. The response comes back through the proxy and out to the client, which never learns that a decision got made on its behalf.

The part worth sitting with: the proxy is now a single, small, well-tested piece of code making a decision that used to live in forty places. When you reshard, you update the proxy’s routing table. You don’t touch application code at all.

Read more

The trick: QUIC identifies a connection by a Connection ID, not by your IP and port. Every packet carries this CID, so when your phone jumps networks and gets a new IP, the server sees the same CID arrive from a different address and matches it to the existing session instead of treating it as a new client. Before it trusts that new path with real bandwidth, it fires a random PATH_CHALLENGE value at the new address and waits for that same value to come back in a PATH_RESPONSE — proof you actually control that endpoint, not someone spoofing it.

Do you know how many round trips your QUIC-based service spends validating a migrated path before it trusts it with full bandwidth?

Most people assume connection migration is instant. The reality: the server resets the congestion window and caps throughput on every new path until PATH_RESPONSE confirms it — treating migration like a brand-new, unproven route, not a resumed one.

The full Deep Dive on QUIC Connection Migration is coming in Week 6. It covers what happens when migration is abused for NAT-rebinding attacks and how load balancers keep CID routing consistent across their backend pool.

Tomorrow: consistent hashing under rebalancing.