Every Postgres table has at least one B-tree index — the primary key. Understanding the physical layout of that tree — pages, keys, pointers, splits — explains every index tuning decision you will ever make. This is the issue most engineers say they should have read first.

An engineer asked me why their query was slow after they added an index. EXPLAIN showed the planner was using the index, the query was returning 1 row, and yet it took 40ms on a table with 2 million rows. The answer was on the B-tree’s fourth level: the table had been created with a UUID primary key, which caused random page splits on every insert and bloated the index to four levels with poor fill rates. Understanding this required understanding what a B-tree page split actually is.

What a B-tree index physically is

A B-tree index is a file of 8KB pages, structured as a tree. The file lives alongside the heap file in the Postgres data directory. Every page in the index file is one of three types:

Root page: The entry point. Every index has exactly one root. A new index starts as a single root page that is also a leaf. The root’s location is stored in the index’s meta page.

Internal pages: Non-leaf pages containing keys and pointers to child pages. Each internal page can hold roughly 340 entries for an integer key (8KB page ÷ ~24 bytes per entry). The more entries an internal page holds, the shorter the tree.

Leaf pages: The bottom level. Each leaf page entry contains the indexed key value and a ctid — the heap tuple physical address (page_number, line_pointer). Leaf pages are linked in key order via forward and backward pointers. This linking is what makes B+ trees (what Postgres actually uses) efficient for range scans.

Read more

Postgres stores your data row-by-row. DuckDB, Redshift, and ClickHouse store it column-by-column. Same data, fundamentally different I/O cost for different query types. This is the single most consequential storage design decision in modern database engineering.

A data analyst runs SELECT SUM(revenue) FROM sales on a 10-million-row table. The sales table has 40 columns. revenue is one of them — 8 bytes each, totalling 80MB for all 10 million values.

In Postgres, that query reads approximately 800MB from disk. All 40 columns of all 10 million rows must be loaded — 8KB pages containing complete rows — even though only one column is needed.

In a columnar store like DuckDB or Redshift, the same query reads approximately 80MB — just the revenue column. Ten times less I/O.

This is not a Postgres bug or limitation. It is the fundamental consequence of the row storage format, and understanding why makes the right database choice obvious for any given workload.

Row storage: optimised for rows

Postgres stores each row’s columns contiguously on the same 8KB page. When a query reads row #4,827, Postgres loads the page containing that row and has immediate access to all its columns — id, customer_name, product_id, revenue, discount, timestamp, all 40 of them — in a single page read.

This is perfect for OLTP workloads: SELECT * FROM orders WHERE id = 42 loads one page and returns one complete row. UPDATE orders SET status='shipped' WHERE id = 42 modifies one tuple on one page and writes one WAL record. The row is the atomic unit of access.

Read more

An engineer optimised every aspect of a query — added the right index, reduced column count, ran ANALYZE. The query still took three times longer than the EXPLAIN estimated. The culprit: a JSONB column averaging 15KB per row. Every row read from the heap was a TOAST pointer, triggering an invisible join to a second table for the actual value.

TOAST (The Oversized-Attribute Storage Technique) is Postgres’s mechanism for storing values larger than approximately 2KB out-of-line, in a separate pg_toast table. It’s transparent — you never write TOAST explicitly — but its performance implications are very much not transparent.

Why TOAST exists: the 8KB page constraint

From Issue #015: Postgres reads and writes in 8KB pages. A page header uses 24 bytes. Line pointers use ~4 bytes each. That leaves roughly 8,100 bytes for actual row data. If a single column value is 50KB, it cannot fit on a page.

The solution TOAST uses: if a row would exceed roughly 2KB (the TOAST_TUPLE_THRESHOLD), Postgres compresses and/or moves the large values out of the heap page and into a dedicated TOAST table. What remains in the heap is an 18-byte TOAST pointer: a reference containing the TOAST table OID, a chunk ID, and the original value size.

Read more

A senior engineer I work with discovered something that permanently changed how they designed schemas. They were profiling a slow endpoint and found the database was reading 14,000 pages to serve a query returning 200 rows. Each page is 8KB. That is 112MB of I/O for a 200-row result.

The table had 47 columns. The query selected 3 of them. But since row storage puts all columns of a row on the same page, reading those 3 columns forced Postgres to load every page containing the 200 matching rows — each page packed with all 47 columns of all rows on that page.

The fix was a covering index. But understanding why the covering index helped — and why the original query was expensive — required understanding what a page actually is.

The 8KB page: Postgres’s atomic storage unit

Postgres never reads or writes individual rows. It reads and writes pages — fixed-size blocks of 8,192 bytes (8KB). Every heap table, every index, every system catalog is a sequence of these pages. When you execute SELECT * FROM employees WHERE id = 42, Postgres locates the page containing that row and loads the entire 8KB page into shared_buffers, even if the row itself is only 200 bytes.

This is the page size contract: the operating system’s unit of I/O aligns with Postgres’s page size, meaning each read maps to one or a small number of OS operations. Making pages smaller would mean more I/O operations for sequential scans; making them larger would waste memory for small tables. 8KB is the default, compiled-in constant.

Read more

There is a pattern in storage systems so fundamental that once you see it, you see it everywhere:

1. Keep an append-only log. Never modify history.

2. Periodically snapshot the current state. Call this a checkpoint.

3. On startup, load the snapshot, then replay only what came after it.

Postgres does this with checkpoints and WAL replay. Redis does this with RDB snapshots and AOF replay. Bitcask does this with its “keydir” snapshot and log replay. The specific names differ. The pattern is identical. This issue implements it in about 80 new lines on top of Part 2.

After a checkpoint on a 300,000-record log, startup drops from 613ms to 202ms — a 3× speedup. More importantly, the startup time no longer scales with total log history. It scales only with records written since the last checkpoint. Checkpoint every 10,000 writes and startup is bounded regardless of how long the system has been running.

What a checkpoint stores

Snapshot file format:
  [MAGIC: 4 bytes]           "BBKV" — identifies this as our format
  [checkpoint_offset: 8 bytes]  byte position in log when snapshot was taken
  [entry_count: 4 bytes]     number of index entries
  for each entry:
    [key_len: 4 bytes]
    [key bytes]
    [value_offset: 8 bytes]  same as what's in self._index
    [value_len: 4 bytes]

It’s a complete serialisation of self._index plus the log byte offset at the moment checkpoint() was called. That offset is the crucial piece — it tells us exactly where to seek in the log on startup to replay only the new records.

Read more

In Part 1 we built a crash-safe key-value store. get(”user:42”) worked correctly and survived process kills. It was also catastrophically slow — O(n) for every lookup. Today we fix that with a 12-line structural change that makes reads O(1). No new disk format. No new files. Just a dict.

Part 1 ended with a working, crash-safe key-value store that stored records in an append-only log and recovered from crashes by scanning for the last valid CRC. The correctness story was solid. The performance story was not. To look up a single key in a 20,000-record log, the store read every record from the beginning to find the latest match. 500 random lookups took 15.5 seconds.

The fix is embarrassingly simple in hindsight, which is exactly why it’s worth implementing yourself before reading about it: keep a dictionary in memory that maps every live key to the byte offset where its value lives in the log. A lookup becomes offset, length = index[key] followed by one file.seek(offset) and one file.read(length). No scan. No iteration. One dict lookup and one seek.

Read more

In 2012, engineers at a startup discovered their config store had been silently corrupting data for three months. The code wrote a JSON file on every config update. It worked perfectly across 4 million writes in staging. On the first power cut in production, the file was half-written. JSON parse failed. Service refused to start. Every engineer on the team had read about durability. None of them had built the primitive that makes it real.

By the end of this issue you will understand exactly what went wrong, why the fix is a single design pattern, and how that pattern underlies every durable storage system you have ever used — from SQLite to Postgres to RocksDB.

The problem with writing directly to a file

When your code calls write() on a file and the call returns success, the data is not on disk. It is in the Linux kernel’s page cache — a pool of RAM managed by the OS, separate from your process memory. The kernel decides when to flush it to actual storage. Usually this happens within seconds. But if the machine loses power before that flush, your data is gone. The OS confirmed the write succeeded. The disk never saw it.

This is not a bug. It is how write() is specified. Performance would collapse if every write() forced a physical disk write.

Databases need stronger guarantees. They use fsync() — a system call that blocks until the OS confirms the data has physically reached the storage device. Expensive: 500µs–5ms on NVMe, 10–30ms on spinning disk. But it is the only way to guarantee durability.

fsync() solves the write-to-cache problem. But it introduces a new problem: what if the machine crashes halfway through writing a record? You now have a partially-written file on disk. Corrupted. Unrecoverable.

Read more

A payments company ran a daily reconciliation query for eight months before anyone noticed it was wrong. The query aggregated transaction totals by region and compared them against bank statements. Every day it produced numbers. Every day the finance team signed off on them. Every day it was silently excluding transactions from three regions.

The bug was a single NULL in the region_id column of a lookup table — a row that had been inserted by a data migration script eighteen months earlier. The query used NOT IN with a subquery. The NULL made the entire NOT IN clause evaluate to UNKNOWN for every transaction. The finance team was reconciling against numbers that excluded roughly 4% of daily transaction volume.

Eight months of wrong numbers. One NULL. Zero errors. Zero warnings.

This is not a Postgres quirk or a MySQL gotcha. It is the mathematically correct behaviour of SQL’s three-valued logic — a system where comparisons involving NULL produce neither TRUE nor FALSE but a third value called UNKNOWN. Understanding why this happens, and where it happens, is the difference between writing queries that work and writing queries that look like they work.

Why NULL exists at all

CODD introduced NULL in his relational model to represent information that is missing or inapplicable. A customer with no phone number on file. An employee whose salary hasn’t been set yet. An order with no shipping date because it hasn’t shipped.

Two-valued logic — TRUE and FALSE — has no way to represent this. If you store 0 for a missing salary, you’ve confused “no salary” with “zero salary.” If you store an empty string for a missing email, you’ve confused “no email” with “an empty email address.” NULL is a different thing: it is the explicit representation of “we don’t know.”

CODD’s decision was correct and mathematically elegant. But it means SQL operates on three truth values, not two:

• TRUE — the proposition is known to be true

• FALSE — the proposition is known to be false

• UNKNOWN — the proposition involves at least one NULL; we cannot determine its truth

Every WHERE clause filter, every JOIN condition, every CHECK constraint evaluates to one of these three values. Rows are included in results only when the predicate evaluates to TRUE. UNKNOWN rows are discarded — silently, identically to FALSE.

This is the mechanism behind the reconciliation bug. The NOT IN predicate evaluated to UNKNOWN for every row. UNKNOWN rows are discarded. Result: empty.

The five places NULL behaves differently

The challenge is that NULL’s semantics are not consistent across SQL contexts. The same value — NULL — is handled differently depending on where it appears.

The diagram builds the five contexts in sequence, showing both the behaviour and the most common trap in each.

Comparisons ( = , != , > , < ): Any comparison involving NULL produces UNKNOWN. NULL = NULL is UNKNOWN, not TRUE. NULL = 5 is UNKNOWN. NULL != 5 is UNKNOWN. The only operators that handle NULL specially are IS NULL and IS NOT NULL — these are not comparisons, they’re existence tests, and they always return TRUE or FALSE.

-- This never matches any row, even rows where col IS NULL:
SELECT * FROM orders WHERE status = NULL;

-- This correctly finds NULL rows:
SELECT * FROM orders WHERE status IS NULL;

GROUP BY: NULL is grouped together. All rows with NULL in the GROUP BY column land in the same group, as if NULL equals NULL. This is the opposite of comparison behaviour, where NULL does not equal NULL. The inconsistency is intentional — grouping requires partitioning rows; without this rule, every NULL would be its own group.

SELECT region_id, COUNT(*) FROM orders GROUP BY region_id;
-- Rows with NULL region_id appear as a single group with region_id = NULL

UNIQUE constraints: Multiple NULLs are allowed in a UNIQUE column. The reasoning: since NULL does not equal NULL, two NULLs are not considered duplicates. A column with a UNIQUE constraint can contain many NULL rows. If you want to prevent this, add NOT NULL alongside UNIQUE.

ORDER BY: NULL sorts LAST in ascending order, FIRST in descending order by default. Control this explicitly with NULLS FIRST or NULLS LAST:

ORDER BY salary ASC NULLS LAST   -- 1, 2, 3, NULL
ORDER BY salary DESC NULLS LAST  -- 3, 2, 1, NULL
ORDER BY salary ASC NULLS FIRST  -- NULL, 1, 2, 3

Aggregate functions: Every aggregate function except COUNT(*) silently ignores NULL values. SUM, AVG, MIN, MAX, COUNT(column) — all of them exclude NULL rows from their computation without warning. COUNT(*) counts all rows including those with NULLs. COUNT(salary) counts only rows where salary is not NULL.

The AND and OR truth tables

Before the traps, the mechanics. Three-valued logic has precise rules for combining UNKNOWN with TRUE and FALSE:

ABA AND BA OR BTRUETRUETRUETRUETRUEFALSEFALSETRUETRUEUNKNOWNUNKNOWNTRUEFALSEUNKNOWNFALSEUNKNOWNUNKNOWNUNKNOWNUNKNOWNUNKNOWN

Two entries are surprising:

FALSE AND UNKNOWN = FALSE — regardless of what UNKNOWN resolves to, ANDing with FALSE gives FALSE. So the result is definitively FALSE.

TRUE OR UNKNOWN = TRUE — regardless of what UNKNOWN resolves to, ORing with TRUE gives TRUE.

These rules matter because SQL WHERE clauses are boolean expressions. A predicate like salary > 80000 AND department_id = 5 evaluates both sides and combines them with AND. If either side is UNKNOWN, the combined result follows the AND table above.

And NOT: NOT UNKNOWN = UNKNOWN. Negating an unknown truth is still unknown.

The trap every engineer hits: complementary predicates that don’t cover everything

Here is a query that looks like it returns all rows:

SELECT * FROM employees
WHERE salary > 50000 OR salary <= 50000;

It doesn’t. Employees with NULL salary satisfy neither predicate — both evaluate to UNKNOWN, UNKNOWN OR UNKNOWN = UNKNOWN, and UNKNOWN rows are excluded. An employee with no recorded salary is invisible to this query.

The correct version: WHERE salary > 50000 OR salary <= 50000 OR salary IS NULL.

This is why filtering on a nullable column requires explicit NULL handling. The complement of salary > 50000 is not salary <= 50000. It is salary <= 50000 OR salary IS NULL.

Try this now

Run this on any Postgres database. It demonstrates all four anomalous truth table entries in one query:

SELECT
    TRUE  AND UNKNOWN AS "T AND U",   -- UNKNOWN (surprising)
    FALSE AND UNKNOWN AS "F AND U",   -- FALSE   (surprising)
    TRUE  OR  UNKNOWN AS "T OR U",    -- TRUE    (surprising)
    FALSE OR  UNKNOWN AS "F OR U",    -- UNKNOWN (surprising)
    NOT UNKNOWN       AS "NOT U";     -- UNKNOWN

Postgres returns the values directly — no table needed. Notice the two entries that most engineers get wrong: FALSE AND UNKNOWN returns FALSE (not UNKNOWN), and TRUE OR UNKNOWN returns TRUE (not UNKNOWN).

The paid section covers the production traps that follow directly from these rules — including the reconciliation bug from the opening and exactly why NOT IN with a NULL subquery empties your result set.

What’s in this issue’s paid section

✦ The NOT IN / NULL trap — the full mechanism, the reconciliation bug explained, and the exact fix
✦ NULL in aggregate functions — the silent denominator problem with AVG, COUNT, and SUM
✦ The four NULL-safe operators: COALESCE, NULLIF, IS DISTINCT FROM, NULLS FIRST/LAST
✦ GROUP BY NULL behaviour and when it produces wrong business metrics
✦ Full hands-on experiments — 4 reproducible SQL demos with annotated output
✦ Production NULL audit checklist — 7 SQL queries to find NULL bugs in your existing schema
✦ Query Lab annotated solution from Issue #010
✦ Downloadable cheat sheet: NULL behaviour reference card

$15/month · $120/year · cancel anytime

The NOT IN / NULL trap — the full mechanism

This is the bug that emptied eight months of reconciliation reports. Here it is in minimal form:

CREATE TEMP TABLE regions (id INT, region_id INT);
INSERT INTO regions VALUES (1, 10), (2, 20), (3, NULL);

CREATE TEMP TABLE transactions (id INT, region_id INT, amount NUMERIC);
INSERT INTO transactions VALUES (1, 10, 500), (2, 20, 300), (3, 30, 200);

-- Intended: find transactions in regions not marked active
SELECT * FROM transactions
WHERE region_id NOT IN (SELECT region_id FROM regions);
-- Returns: (0 rows)

The subquery returns {10, 20, NULL}. For each transaction row, SQL expands NOT IN as:

-- For transaction with region_id = 10:
10 NOT IN (10, 20, NULL)
= 10 != 10 AND 10 != 20 AND 10 != NULL
= FALSE AND TRUE AND UNKNOWN
= FALSE
-- Row excluded (correct — region 10 is in the list)

-- For transaction with region_id = 30:
30 NOT IN (10, 20, NULL)
= 30 != 10 AND 30 != 20 AND 30 != NULL
= TRUE AND TRUE AND UNKNOWN
= UNKNOWN
-- Row excluded (WRONG — should be included, region 30 is not in the list)

Every row where the match against NULL is the deciding factor becomes UNKNOWN and is excluded. The query returns nothing even when it should return rows.

Fix 1 — Filter NULL from subquery:

SELECT * FROM transactions
WHERE region_id NOT IN (
    SELECT region_id FROM regions WHERE region_id IS NOT NULL
);
-- Returns: (1 row) — transaction with region_id = 30

Fix 2 — Use NOT EXISTS (immune to NULL, often better plan):

SELECT t.* FROM transactions t
WHERE NOT EXISTS (
    SELECT 1 FROM regions r WHERE r.region_id = t.region_id
);
-- Returns: (1 row) — correct, and the planner uses an anti-join

NOT EXISTS works because it tests for the existence of matching rows, not equality. A NULL in the subquery cannot match any value from the outer query’s row.

Always prefer NOT EXISTS over NOT IN when the subquery touches nullable columns. It is safer, and the query planner often generates a more efficient plan (anti-join vs nested loop with filter).

NULL in aggregate functions — the silent denominator

The diagram shows a 7-row employee table where 2 employees have NULL salary. It builds the aggregate results one by one — COUNT(*), COUNT(salary), SUM, AVG, MAX, MIN — showing which NULLs are included and which are silently excluded.

The dangerous one is AVG. Given salaries of {90k, 85k, NULL, 92k, 78k, NULL, 72k}:

• COUNT(*) = 7 (counts all rows)

• COUNT(salary) = 5 (excludes 2 NULL rows)

• SUM(salary) = 417,000 (excludes NULLs)

• AVG(salary) = 417,000 ÷ 5 = 83,400 — not 417,000 ÷ 7 = 59,571

The AVG denominator is the count of non-NULL rows, not total rows. If you mean “average salary across all employees, treating unrecorded salaries as zero,” you must write:

-- Treating NULL salary as 0:
SELECT SUM(COALESCE(salary, 0)) / COUNT(*) AS true_avg FROM employees;
-- = 417,000 / 7 = 59,571

-- Or equivalently:
SELECT AVG(COALESCE(salary, 0)) FROM employees;

If you mean “average salary among employees with a recorded salary,” then AVG(salary) is correct — but document this intent explicitly in a comment, because the next engineer to read the query will assume it covers all rows.

The GROUP BY NULL problem:

SELECT region_id, SUM(amount) FROM orders GROUP BY region_id;

Orders with NULL region_id form their own group, reported as:

region_id | sum
----------+------
       10 | 5000
       20 | 3200
     NULL | 1800   ← orders with no region assigned

If the business intent is “all orders belong to exactly one region,” this NULL group is data quality problem. If the intent is “some orders have no region,” this group is correct. The query cannot tell you which — you must know the business semantics.

Audit: SELECT COUNT(*) FROM orders WHERE region_id IS NULL — if this returns anything unexpected, you have unassigned orders silently accumulating in your aggregate totals.

The four NULL-safe operators

COALESCE(val, fallback, ...) — Returns the first non-NULL argument. The standard way to replace NULL with a default:

COALESCE(salary, 0)        -- 0 when salary is NULL
COALESCE(phone, email, 'no contact')  -- first non-NULL contact method

NULLIF(val, other) — Returns NULL if val = other, otherwise returns val. Prevents division-by-zero and converts sentinel values to NULL:

100 / NULLIF(divisor, 0)        -- NULL instead of division-by-zero error
NULLIF(status, 'unknown')       -- NULL when status = 'unknown'

IS DISTINCT FROM / IS NOT DISTINCT FROM — NULL-safe equality comparison. Unlike =, these return TRUE or FALSE even when NULL is involved:

NULL IS DISTINCT FROM NULL     -- FALSE  (same unknown = not distinct)
NULL IS DISTINCT FROM 5        -- TRUE   (NULL differs from 5)
5 IS NOT DISTINCT FROM 5       -- TRUE   (same value)
5 IS NOT DISTINCT FROM NULL    -- FALSE  (5 differs from unknown)

Use this when you need to compare two nullable columns and want NULL = NULL to be TRUE — the opposite of regular = behaviour.

NULLS FIRST / NULLS LAST — Explicit control over NULL sort position. Never rely on the default — it varies by sort direction. Always specify explicitly when NULLs appear in columns you sort by:

ORDER BY salary ASC  NULLS LAST   -- 72k, 78k, 85k, 90k, 92k, NULL, NULL
ORDER BY salary DESC NULLS LAST   -- 92k, 90k, 85k, 78k, 72k, NULL, NULL

Read more

When you upgrade to a paid subscription, you’re added to the bytes-and-btrees-labs GitHub organisation within a few minutes. You’ll see every repository listed below — past and future — with full read access.

The vault grows by 3–4 assets every month. Founding Members get permanent access at today’s price, regardless of future price changes.

database-internals-vault

One folder per issue. Each folder contains the production scripts and SQL files referenced in the paid section of that issue.

Issue #007 — Atomicity at the byte level

• wal_demo.py — Simulates the WAL write sequence during COMMIT, ROLLBACK, and crash recovery. Zero dependencies. Shows exactly what’s in memory vs on disk at each step.

• fsync_benchmark.sh — Measures fsync latency on your actual hardware using pg_test_fsync or a pure bash fallback. Tells you your hardware’s commit latency and recommends the right wal_sync_method.

Issue #008 — Durability & fsync

• verify_write_cache.sh — Audits every block device for dangerous write cache configuration. Reports which drives are misconfigured and the exact fix command.

• durability_checklist.md — 8-item durability checklist with verify commands and fix commands for every Postgres durability setting.

Issue #009 — Isolation levels

• isolation_demo.sql — All four isolation anomalies (dirty read, non-repeatable read, phantom read, write skew) reproduced in one file. Copy-paste each section into psql.

• ssi_lock_inspector.sql — Observe Postgres’s SIREAD locks during active SERIALIZABLE transactions. Shows SSI’s internal dependency tracking.

Issue #010 — SQL & relational algebra

• query_equivalence_proofs.sql — Four algebraic equivalences demonstrated with EXPLAIN output. Shows the planner’s selection pushdown rewrite in action.

• null_trap_demos.sql — All four NULL traps (NOT IN null subquery, complement gaps, AVG denominator, LEFT JOIN WHERE) reproduced with minimal test data and fixes.

Issue #011 — NULLs & three-valued logic

• null_audit.sql — 7-query audit suite that finds NULL bugs in your production schema. Read-only, safe to run in production. Requires pg_stat_statements for full results.

production-checklists

Standalone checklists that don’t depend on a specific issue. Run these on any Postgres server.

• postgres-durability-checklist.md — 8 items covering fsync, synchronous_commit, wal_sync_method, full_page_writes, write cache, WAL storage, checkpoint tuning, and replication durability.

• isolation-level-audit.sql — Finds active transactions and their isolation levels, long-running transactions causing MVCC bloat, lock-holding queries, and tables missing integrity constraints.

• index-health-check.sql — Finds unused indexes, bloated indexes, missing indexes (via seq_scan ratio), duplicate indexes, and calculates index-to-table size ratios.

query-lab-solutions

Every Query Lab puzzle solution in annotated SQL format. Each solution file includes:

• The puzzle restated

• 2–3 valid approaches with explanations

• Which approach is preferred and why

• The specific concept from the issue it tests

build-it-yourself (coming with Issue #012–014)

The Build It Yourself series ships line-by-line annotated implementations:

• BIY #1: A crash-safe key-value store (487 lines of Python) — WAL, hash index, crash recovery

• BIY #2: A minimal SQL query engine — AST parser, logical plan, hash join executor

• BIY #3: A replication log with primary and replica

How to get access

Subscribe at $15/month or $120/year. Within a few minutes you’ll receive an email asking for your GitHub username. Enter it and you’re added to the organisation automatically.

To claim permanent access at today’s price: upgrade to Founding Member ($240/year). 50 spots available.

Subscribe to access the vault

© Bytes & B-trees · Substack · bytesbtrees.substack.com

Welcome. This newsletter is a structured 3-year curriculum on how databases actually work — not how to use them, but how they’re built, why they make the decisions they make, and what happens inside the engine when your queries run.

Every issue goes deep on exactly one concept. Not a roundup. Not bullet points. One concept, from first principles, with working code, animated diagrams, and production context.

Here’s what you need to know to get the most out of it.

Who this is for

Engineers who use databases daily and want to understand what’s happening underneath. The sweet spot is senior backend engineers, architects, and SREs who can write complex queries but couldn’t confidently explain why a particular query plan was chosen, what MVCC is actually doing, or when to use SERIALIZABLE isolation.

If that’s you, you’re in the right place.

If you’re completely new to databases, start with Issue #001 and work forward. The curriculum is designed to be progressive — each issue builds on the ones before it.

Free vs paid

Free subscribers get the full core concept every issue — typically 1,200–2,000 words explaining the mechanism, one animated diagram, and a Query Lab puzzle. This is genuinely complete. You’ll understand the concept.

Paid subscribers ($15/month · $120/year) get the full article including:

• Production failure modes — the specific ways the concept breaks in real systems

• 3–4 hands-on experiments with annotated output

• A configuration checklist with verify and fix commands

• Annotated Query Lab solution from the previous issue

• Access to the private GitHub vault (see below)

The difference is the gap between knowing what something is and being able to use it safely in production.

The GitHub vault

Every paid issue ships a vault asset: a Python script, SQL file, or configuration guide that goes into a private GitHub repository. When you subscribe, you’re invited to the bytes-and-btrees-labs GitHub organisation and get access to everything in the vault — including all past assets, not just this week’s.

What’s currently in the vault:

• Issue #007: WAL write sequence demo (Python, zero deps) + fsync benchmark script

• Issue #008: Write cache verification script + 8-item durability checklist

• Issue #009: All 4 isolation anomalies reproducible in one SQL file + SSI lock inspector

• Issue #010: Query equivalence proofs + NULL trap demonstrations

• Issue #011: 7-query NULL audit suite (find NULL bugs in your production schema)

• Production checklists: index health check + isolation level audit + durability checklist

The vault grows by 3–4 assets every month.

Start with these five free issues

If you’re not sure where to begin, these five issues are the most referenced and the most shared:

1. Issue #001 — Data, storage & memory: first principles. Starts at the bottom of the stack and explains why disk access speed shapes every database design decision.

2. Issue #007 — Atomicity at the byte level. Traces a COMMIT from application to disk write by write. Explains what the WAL actually does and why ROLLBACK doesn’t need to touch disk.

3. Issue #008 — Durability: what fsync guarantees. The three ways your database’s durability claim can be silently false — and how to verify yours isn’t.

4. Issue #009 — Isolation levels visualised. All four anomalies, animated, with the exact SQL that reproduces each one and the isolation level that prevents it.

5. Issue #010 — SQL and relational algebra. Why the planner can safely rewrite your queries, and the algebraic reason NULL behaves the way it does.

The Query Lab

Every issue ends with a puzzle that requires applying the week’s concept to answer. The puzzle is free — post your answer in the comments. The annotated solution (with 2–3 valid approaches explained) is in the following issue’s paid section.

If you want to check your reasoning on past puzzles, the full solution archive is in the vault.

One thing that would help me

If you know exactly one engineer who would find this useful, forward them this issue. That’s genuinely the only growth mechanism this publication uses. No paid ads. No sponsor deals. Just word of mouth.

See you Sunday.

© Bytes & B-trees · Substack · bytesbtrees.substack.com

A senior engineer at a fintech company spent two days debugging a reporting discrepancy. Their query looked right. The table had data. The filters made sense. But the result was consistently empty. No error. No warning. Just zero rows.

The query was this:

SELECT name FROM customers
WHERE region_id NOT IN (
    SELECT region_id FROM regions WHERE active = false
);

One region had a NULL in region_id. That single NULL made the entire NOT IN clause evaluate to UNKNOWN for every customer row — returning nothing. Two days of debugging for one NULL in a subquery that nobody knew was there.

This isn’t a bug. It’s the mathematically correct behaviour of SQL’s three-valued logic — a direct consequence of how NULL is defined in relational algebra. The engineer didn’t have a query problem. They had a knowledge gap: they’d learned SQL syntax without the algebra underneath it.

That algebra is what this issue covers. Not because you need to write mathematical notation — you don’t. But because when you understand the six operations that SQL is built from, queries become transparent. You’ll see immediately why that NOT IN query returns nothing, why the planner moves your WHERE clause before the JOIN, and why some “optimisations” your colleagues suggest are algebraically guaranteed to give the same result.

What relational algebra actually is

CODD didn’t design SQL. He designed the relational algebra — a set of six operations on relations (tables) that can express any fact retrievable from relational data. SQL was designed later as a human-readable syntax that maps onto those operations.

The six operations are:

σ (sigma) — Selection. Filter rows based on a predicate. σ_salary > 80000 (employee) is every row where salary exceeds 80,000. In SQL: WHERE salary > 80000.

π (pi) — Projection. Choose which columns to include. π_name, salary (employee) returns only name and salary. In SQL: SELECT name, salary.

⋈ — Join. Combine two relations on a matching condition. In SQL: JOIN ... ON.

∪ — Union. Combine two result sets with identical schemas. In SQL: UNION.

− — Difference. Rows in one relation that don’t appear in another. In SQL: EXCEPT.

ρ (rho) — Rename. Give a relation or attribute a new name. In SQL: AS.

That’s the complete vocabulary. Every SQL query you’ve ever written — no matter how complex, with subqueries, CTEs, window functions, and recursive joins — is a composition of these six operations. The query planner translates your SQL into an algebra tree and then finds the most efficient physical execution of that tree.

The diagram shows each operation applied to an employee table, with results appearing alongside. The key observation in the last stage: operations compose. π name (σ salary > 80000 (employee)) applies selection first, then projection — which is exactly what SELECT name FROM employee WHERE salary > 80000 does. The planner sees your SQL, builds this algebra expression, and then decides whether to apply the selection before or after loading the full table.

Get Access to GitHub Link

Why the algebra explains query optimisation

Here is the insight that makes database query planning make sense: algebraic equivalences let the planner legally rewrite your query into a faster form.

Two algebra expressions are equivalent if they always produce the same result on any valid input. The planner maintains a library of these equivalences — hundreds of rewrite rules — and applies them before choosing a physical execution strategy.

The most important one for everyday SQL:

Selection pushdown: σ_condition(R ⋈ S) is equivalent to σ_condition(R) ⋈ S

In English: filtering rows before joining is mathematically the same as filtering rows after joining — but filtering first means fewer rows enter the join, making the join dramatically cheaper.

This is why the planner moves WHERE clauses before JOIN operations. Not as a heuristic. Not as a “best practice.” As an algebraic guarantee. The result is identical either way, and one path is faster.

Other equivalences the planner applies:

-- These two queries are algebraically identical:
SELECT * FROM orders WHERE status = 'shipped' AND amount > 100;
SELECT * FROM orders WHERE amount > 100 AND status = 'shipped';
-- Predicate order in WHERE is irrelevant. Planner reorders for index use.

-- These are also equivalent:
SELECT DISTINCT customer_id FROM orders;
SELECT customer_id FROM orders GROUP BY customer_id;
-- Both express π with duplicate elimination. Planner may choose either plan.

-- Subquery to join rewrite (planner does this automatically):
SELECT name FROM customers
WHERE id IN (SELECT customer_id FROM orders WHERE amount > 500);
-- Equivalent to:
SELECT DISTINCT c.name FROM customers c
JOIN orders o ON c.id = o.customer_id
WHERE o.amount > 500;

When you understand that these rewrites are algebraic equivalences — not approximations — you stop worrying whether the planner will “do the right thing.” It will, because the right thing is mathematically defined.

The fundamental unit of relational algebra: the relation

A relation is not just “a table.” Formally, a relation is a set of tuples — and sets have no duplicates and no ordering. These two properties have direct SQL consequences:

No ordering: A bare SELECT has no guaranteed row order. This surprises engineers who see consistent ordering in testing and assume it’s guaranteed. It isn’t. Without ORDER BY, a query can return rows in any order the planner chooses — and that order can change when you add an index, update statistics, or upgrade Postgres. Always use ORDER BY when order matters.

No duplicates: A true relation has no duplicate rows. SQL allows duplicates (it’s a multiset, technically) — which is why SELECT DISTINCT exists, and why CODD’s Rule 12 is violated by SQL at a theoretical level. For most practical purposes this doesn’t matter. For queries that depend on set semantics (UNION vs UNION ALL, EXCEPT vs EXCEPT ALL), it matters significantly.

How the query lifecycle uses algebra

Every SQL query passes through this pipeline before a single row is read:

1. Parse — Your SQL string is tokenised and parsed into a parse tree. Syntax errors are caught here.

2. Analyse — Names are resolved: employee → relation OID 16385, salary → attribute 4 of that relation. Type checking happens. Semantic errors are caught here.

3. Rewrite — View definitions are inlined. Rule-based rewrites are applied. If your query references a view, the view’s definition is substituted into the query tree here.

4. Plan (logical) — The parse tree is converted into a logical algebra tree. Equivalence rewrites are applied — selection pushdown, join reordering, subquery flattening. The planner generates multiple candidate trees.

5. Plan (physical) — Each logical algebra operation is assigned a physical implementation: should this join use hash join, merge join, or nested loop? Should this selection use an index scan or sequential scan? Should this sort use an explicit sort node or an index scan that returns rows pre-sorted? The planner estimates costs using table statistics and picks the cheapest physical plan.

6. Execute — The physical plan runs. Rows are returned.

Steps 4 and 5 are where the algebra matters. The planner can only rewrite the logical tree safely because the rewrites are algebraic equivalences. If they weren’t, the planner would need to ask you for permission before each rewrite — and SQL would be far less usable.

Try this right now

Run this on any Postgres database with data:

EXPLAIN (FORMAT TEXT)
SELECT e.name, d.dept_name
FROM   employee e
JOIN   department d ON e.dept_id = d.id
WHERE  e.salary > 80000;

Look at the output. Find the line that says Filter: (salary > 80000). Note whether that filter appears above or below the join node. In almost every case, Postgres will have pushed the filter below the join — applying it to the employee table before the join happens.

Now try:

EXPLAIN (FORMAT TEXT)
SELECT e.name, d.dept_name
FROM   (SELECT * FROM employee WHERE salary > 80000) e
JOIN   department d ON e.dept_id = d.id;

Compare the plans. They’re identical. Postgres recognises that both SQL forms produce the same algebra tree after selection pushdown — and generates the same plan for both. That’s the algebra at work.

The paid section explains what the other nodes in this plan mean, how to read the cost estimates, and — importantly — the three cases where Postgres cannot push selection below a join, why, and how to rewrite your query to help it.

What’s in this issue’s paid section

✦ Three-valued logic and NULL — the complete truth tables with every trap explained
✦ The SQL algebra tree diagram — how Postgres builds and rewrites your query plan
✦ The five equivalence rules you can apply yourself to rewrite slow queries
✦ When selection pushdown fails — three cases and how to fix them
✦ Full hands-on experiments: query equivalences, NULL behaviour, EXPLAIN analysis
✦ Production checklist: 7 NULL-related bugs to audit in your existing queries
✦ Query Lab annotated solution from Issue #009
✦ Downloadable cheat sheet: SQL ↔ relational algebra reference card

$15/month · $120/year · cancel anytime Subscribe → bytesandbrees.substack.com

Three-valued logic — where the fintech bug lives

SQL has three truth values: TRUE, FALSE, and UNKNOWN. UNKNOWN arises whenever NULL is involved in a comparison. This isn’t a Postgres quirk — it’s the formal definition of NULL in relational theory, designed to represent “unknown or inapplicable information.”

The diagram builds the AND, OR, and NOT truth tables for three-valued logic, then reveals the three traps in sequence.

The AND table has one surprising entry: FALSE AND UNKNOWN = FALSE. Why? Because regardless of what UNKNOWN resolves to (true or false), anding with FALSE gives FALSE. So FALSE AND UNKNOWN is definitely FALSE. But TRUE AND UNKNOWN = UNKNOWN — because if the unknown part turns out to be false, the whole expression is false.

The OR table has the symmetric surprising entry: TRUE OR UNKNOWN = TRUE. Because regardless of what UNKNOWN resolves to, oring with TRUE gives TRUE.

These rules are internally consistent and mathematically correct. They’re also the source of bugs that have been in production codebases for years.

Trap 1: NULL = NULL is UNKNOWN, not TRUE

SELECT * FROM orders WHERE cancelled_at = NULL;  -- always 0 rows
SELECT * FROM orders WHERE cancelled_at IS NULL;  -- correct

The equality operator applied to NULL produces UNKNOWN. SQL’s WHERE clause only returns rows where the predicate evaluates to TRUE — UNKNOWN rows are excluded, same as FALSE. IS NULL is the correct predicate; it’s a separate operator that specifically tests for null.

This trap is so common that some SQL clients warn you. Many don’t.

Trap 2: NOT IN with a NULL in the subquery

This is the bug from the opening story. Recall:

SELECT name FROM customers
WHERE region_id NOT IN (
    SELECT region_id FROM regions WHERE active = false
);

If any row in the subquery returns NULL for region_id, the NOT IN predicate evaluates to UNKNOWN for every row in customers — because:

region_id NOT IN (1, 2, NULL)
= region_id != 1 AND region_id != 2 AND region_id != NULL
= region_id != 1 AND region_id != 2 AND UNKNOWN
= UNKNOWN (for any value of region_id)

The fix: either explicitly exclude NULLs from the subquery, or use NOT EXISTS instead (which is immune to NULL):

-- Fix 1: exclude NULLs from subquery
SELECT name FROM customers
WHERE region_id NOT IN (
    SELECT region_id FROM regions
    WHERE active = false AND region_id IS NOT NULL
);

-- Fix 2: NOT EXISTS (preferred — immune to NULL, often better plan)
SELECT name FROM customers c
WHERE NOT EXISTS (
    SELECT 1 FROM regions r
    WHERE r.region_id = c.region_id
      AND r.active = false
);

The NOT EXISTS version is also frequently faster because the planner can use an anti-join, which short-circuits as soon as it finds one matching row.

Trap 3: COUNT(*) vs COUNT(column)

SELECT COUNT(*),       -- counts all rows, including NULLs
       COUNT(salary),  -- counts only non-NULL salary values
       AVG(salary)     -- NULLs excluded from average denominator
FROM employees;

If 3 of 100 employees have NULL salary, COUNT(*) returns 100, COUNT(salary) returns 97, and AVG(salary) divides by 97 — not 100. This makes AVG behave like “average salary among employees who have a recorded salary” not “average salary across all employees.” If you mean the latter, use AVG(COALESCE(salary, 0)).

The SQL algebra tree — what the planner actually builds

The diagram builds the algebra tree for a JOIN query with a WHERE clause and ORDER BY, one node at a time — base tables at the leaves, operations stacking toward the root. The critical annotation appears at step 3: the selection node (σ salary > 80,000) is shown at the leaves, below the join — the result of selection pushdown.

The planner builds the initial tree with selection at the top (where your WHERE clause is syntactically), then applies the pushdown rewrite. This single rewrite can make a query 10–100x faster when the filtered table is large.

Three cases where pushdown fails

The planner cannot push selection below a join in these cases:

Case 1 — The predicate references both tables:

-- Cannot push: predicate uses columns from both tables
SELECT * FROM employee e JOIN department d ON e.dept_id = d.id
WHERE e.salary > d.avg_salary;
-- The join must happen first; you can't filter on d.avg_salary before joining

Case 2 — The predicate uses a volatile function:

-- Cannot push: random() is evaluated per-row and must stay above the join
SELECT * FROM employee e JOIN department d ON e.dept_id = d.id
WHERE e.score > random() * 100;
-- If pushed below, each table side would get different random() values

Case 3 — OUTER JOIN semantics:

-- Cannot always push: LEFT JOIN null-fills right side; WHERE on right side
-- effectively converts to an INNER JOIN, which changes semantics
SELECT e.name, d.dept_name
FROM employee e LEFT JOIN department d ON e.dept_id = d.id
WHERE d.budget > 100000;
-- This already is an inner join in effect; Postgres converts it
-- But if you meant to keep employees without departments, write it differently

Understanding these cases lets you write queries that give the planner room to optimise — and lets you diagnose cases where a plan isn’t as fast as expected.

Read more

Consider a banking application running a nightly report that sums all account balances. The report runs as a single long read transaction. While it runs, thousands of transfer transactions are committing — debiting one account, crediting another. If the report reads account A before its transfer is processed and account B after, it captures an inconsistent snapshot: A’s old balance plus B’s new balance. The sum is wrong. No data was corrupted. No transaction failed. The isolation level just wasn’t strong enough for the workload.

This is a real class of bug that appears in production systems regularly. It’s not dramatic — it doesn’t destroy data, it just produces wrong numbers that take days or weeks to notice. And it’s completely preventable once you understand what isolation levels actually control.

There are four isolation levels in the SQL standard. Each one makes a specific set of anomalies impossible. Each one costs more in locking or tracking overhead. Most applications are running at the wrong level for their workload — either too weak (producing wrong results) or stronger than necessary (throttling concurrency without benefit). This issue maps every anomaly, every level, and the precise mechanism Postgres uses to implement the strongest one.

What isolation actually means

Before anomalies, let’s be precise about what isolation is trying to achieve.

Ideally, concurrent transactions would behave as if they ran one at a time in some serial order. If transaction T1 and T2 run concurrently, the result should be identical to either “T1 then T2” or “T2 then T1.” When this holds, we say the execution is serialisable.

Full serialisability is achievable but expensive — it requires the database to track enough information to detect when two concurrent transactions would produce a result that’s impossible in any serial order, and abort one of them. The SQL standard defines weaker levels that relax this guarantee in exchange for better concurrency.

The four levels, from weakest to strongest:

The matrix builds up row by row — each anomaly appearing alongside which levels prevent it. Reading it column by column: READ UNCOMMITTED prevents nothing. READ COMMITTED (Postgres’s default) prevents only dirty reads. REPEATABLE READ prevents dirty reads and non-repeatable reads. SERIALISABLE prevents all four anomaly classes, including the subtle write skew.

Get Access to GitHub Link

Anomaly #1: The dirty read

A dirty read occurs when transaction T1 reads data written by transaction T2, and T2 subsequently rolls back. T1 has read data that, from the database’s perspective, never officially existed.

The animated diagram shows a clean example:

1. T2 begins and updates balance = 900 (was 1,000). T2 has not committed.

2. T1 reads balance — sees 900.

3. T2 rolls back. Balance is officially 1,000 again.

4. T1 proceeds having operated on a value that was never committed.

The damage depends on what T1 did with that read. If it just displayed balance = 900 and the user immediately refreshed, the worst outcome is mild confusion. If T1 used the dirty value to calculate a transfer amount, approve a loan, or write an audit record, the consequences compound.

The fix: READ COMMITTED. At this level, a transaction can only read rows where the writing transaction has committed. T1’s read of T2’s row will either wait for T2 to commit or rollback, or will see the pre-T2 version — never the dirty intermediate state.

Postgres behaviour: Postgres never allows dirty reads — even at READ UNCOMMITTED, Postgres internally behaves as READ COMMITTED. This is standards-compliant: the SQL standard says READ UNCOMMITTED may allow dirty reads, not that it must. Postgres chose to eliminate dirty reads at all levels.

Anomaly #2: The non-repeatable read

A non-repeatable read occurs when T1 reads the same row twice within a single transaction and gets different values — because T2 updated and committed that row between T1’s two reads.

The diagram shows a reporting transaction:

1. T1 begins and reads balance = 1,000 for account 42.

2. T2 updates balance to 800 and commits.

3. T1 reads balance again — sees 800.

T1 is internally inconsistent. If this is a financial report, it captured account 42 at two different times. The report’s totals are wrong even though no transaction committed invalid data.

This is surprisingly common in real applications. An ORM that lazy-loads relationships can hit this: the same entity is fetched twice in the same request (once directly, once via a join), and the second fetch reflects an update that happened between the two loads.

The fix: REPEATABLE READ. At this level, T1’s snapshot is taken at the start of the transaction and held fixed. Every read within T1 sees the same version of every row — the version that existed when T1 began — regardless of commits that happen during T1’s lifetime.

Postgres implementation: Postgres implements REPEATABLE READ using MVCC snapshot isolation. When T1 begins, Postgres records the current transaction ID as T1’s snapshot boundary. Any row modified by a transaction with a higher ID (i.e., committed after T1 began) is invisible to T1. The old row version is kept alive in the heap specifically so T1 can read it. This is why VACUUM must avoid removing row versions that active transactions might still need — long transactions prevent cleanup of old versions and cause table bloat.

Anomaly #3: The phantom read

A phantom read is subtler than a non-repeatable read. Instead of the same row returning different values, a query returns a different set of rows when run twice.

The canonical example:

-- T1: find all accounts with balance > 500
SELECT * FROM accounts WHERE balance > 500;
-- Returns: {Alice: 1000, Carol: 750}

-- T2: INSERT INTO accounts VALUES ('Dave', 600); COMMIT;

-- T1: same query again
SELECT * FROM accounts WHERE balance > 500;
-- Returns: {Alice: 1000, Carol: 750, Dave: 600}  ← phantom!

T1 ran the same query twice. The second result set has a row that didn’t exist in the first. Dave is the phantom — a row that appeared from nowhere, inserted by a committed concurrent transaction.

REPEATABLE READ prevents the non-repeatable read anomaly (re-reading the same row), but in many databases it does not prevent phantom reads — because the newly inserted row was never read by T1 before, so there’s no “previously seen version” to lock it to.

Postgres is unusual here: Postgres’s REPEATABLE READ implementation actually prevents phantom reads too, because Postgres uses snapshot isolation rather than traditional range locking. T1’s snapshot was taken before Dave’s INSERT — Dave’s row doesn’t exist in T1’s snapshot, and T1 will never see it, even on a re-query. This is stricter than the SQL standard requires for REPEATABLE READ and is one reason Postgres’s REPEATABLE READ is considered particularly strong.

Traditional databases (MySQL InnoDB): MySQL prevents phantom reads at REPEATABLE READ using next-key locks — range locks that prevent other transactions from inserting into the range covered by a query. This is correct but more restrictive than Postgres’s approach, because it locks not just the rows you’ve read but the gaps between them.

Anomaly #4: Write skew — the hardest one

This is the anomaly that most engineers haven’t encountered by name, even when they’ve been bitten by it. It doesn’t involve reading stale data or phantom rows. Both transactions read correct, committed, up-to-date data. Both transactions commit successfully. And yet the result is impossible if either transaction had run alone.

The animated diagram shows the hospital on-call scenario:

Business rule: At least one doctor must be on call at all times.

State: Alice and Bob are both on call (count = 2).

T1 (Alice’s client): Reads count of on-call doctors → 2. Since count ≥ 2, safe to go off call. Deletes Alice from on-call table. Commits.

T2 (Bob’s client): Reads count of on-call doctors → 2. Since count ≥ 2, safe to go off call. Deletes Bob from on-call table. Commits.

Result: Zero doctors on call. Business rule violated. Both transactions read correct data. Both committed without error. No dirty reads. No phantom rows. And yet the combined result is wrong.

The key insight is timing: both T1 and T2 read the count before either deletion was committed. Each saw 2 and independently concluded it was safe to proceed. Neither could see what the other was about to do.

If the transactions had run serially — Alice first, then Bob — Bob’s read would have seen count = 1 and would have blocked or rejected the deletion. The serial result is safe. The concurrent result is not. This is the definition of a serialisation anomaly: a result that’s impossible in any serial ordering.

Other write skew examples that appear in real systems:

-- Meeting room double-booking:
-- T1: SELECT count(*) FROM bookings WHERE room=5 AND time='09:00' -- returns 0
-- T2: SELECT count(*) FROM bookings WHERE room=5 AND time='09:00' -- returns 0
-- T1: INSERT INTO bookings(room, time, user) VALUES (5, '09:00', 'Alice')
-- T2: INSERT INTO bookings(room, time, user) VALUES (5, '09:00', 'Bob')
-- Both commit → room 5 double-booked at 09:00

-- Username uniqueness with application-level checks:
-- T1: SELECT count(*) FROM users WHERE username='alice' -- returns 0
-- T2: SELECT count(*) FROM users WHERE username='alice' -- returns 0
-- T1: INSERT INTO users(username) VALUES ('alice')
-- T2: INSERT INTO users(username) VALUES ('alice')
-- Both commit → two users named 'alice' (if no UNIQUE constraint)

The second example is exactly why UNIQUE constraints must live in the database (CODD Rule 10), not in application code. A database UNIQUE constraint is enforced atomically. Application-level uniqueness checks are vulnerable to write skew.

The fix: SERIALISABLE isolation. At this level, the database detects that T1 and T2 have a dependency cycle (T1’s decision to proceed depended on data that T2 was about to modify, and vice versa) and aborts one of them. The application retries the aborted transaction, which now sees the committed result of the other and makes the correct decision.

How Postgres implements SERIALISABLE: SSI

Postgres uses an algorithm called Serialisable Snapshot Isolation (SSI), introduced in PostgreSQL 9.1. SSI is remarkable because it provides full serialisability without the coarse locking that traditional serialisable implementations require.

Traditional serialisable isolation works by locking everything that’s read, preventing concurrent writers from touching it. This is safe but severely limits concurrency — a long-running read-only transaction blocks all writers on the rows it has read.

SSI works differently. It lets transactions run concurrently using Postgres’s normal MVCC snapshot isolation, but it tracks read-write dependencies between transactions. When it detects a cycle of dependencies that would make the execution non-serialisable, it aborts one of the involved transactions (the “pivot” transaction).

The dependency tracking uses SIREAD locks — a new kind of lock that doesn’t block other transactions but records “transaction X has read this row.” These locks are much lighter than write locks and allow concurrent reads without blocking.

What SSI tracks:

If T1 reads a row that T2 later writes (rw-antidependency T1→T2)
And T2 reads a row that T1 later writes (rw-antidependency T2→T1)
→ This is a dangerous cycle → abort one transaction

In the hospital example:

• T1 reads the on-call count (will be modified by T2’s DELETE)

• T2 reads the on-call count (will be modified by T1’s DELETE)

• Both have read-write antidependencies on each other

• SSI detects the cycle and aborts one

• The other completes successfully

• The retried transaction sees count = 1 and correctly refuses to go off call

The cost of SSI: SIREAD locks consume memory. A long-running SERIALISABLE transaction that reads millions of rows can accumulate millions of SIREAD locks. Postgres manages this with lock summarisation — consolidating many per-row locks into a per-page or per-relation lock — which may produce false positives (unnecessary aborts) but never false negatives (missed violations).

The max_pred_locks_per_transaction parameter controls the lock table size. If SERIALISABLE transactions are aborting more than expected, check whether lock summarisation is causing overly aggressive conflict detection.

Choosing the right isolation level

The practical question: what level should your application use?

READ COMMITTED (Postgres default) — Correct for most OLTP workloads where each transaction is short, operates on a small number of rows, and doesn’t have read-modify-write logic that depends on other concurrent transactions. The reporting anomaly at the top of this issue is the sign you need to go higher.

REPEATABLE READ — Correct for long-running reads that must see a consistent snapshot: financial reports, data exports, analytics queries running against OLTP data. Also correct when you have explicit optimistic locking — read a row, check a version column, update only if the version hasn’t changed. In Postgres, REPEATABLE READ is surprisingly strong (it also prevents phantoms) and rarely causes performance issues.

SERIALISABLE — Necessary when your application has read-then-write logic that depends on the reads being stable: the on-call problem, meeting room bookings, maintaining invariants across multiple rows. If your application implements business rules by checking a condition before writing, and that condition involves rows that other transactions might also be checking and writing, you need SERIALISABLE.

The common mistake is using READ COMMITTED with application-level uniqueness or business-rule checks. That’s vulnerable to write skew. The correct fix is either:

1. Move the constraint into the database (UNIQUE, CHECK, FOREIGN KEY constraints are write-skew-safe)

2. Use SERIALISABLE isolation

3. Use explicit locking (SELECT ... FOR UPDATE) to prevent concurrent modification of the rows the decision depends on

Per-transaction isolation in Postgres:

-- Set isolation for a single transaction:
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;

-- Or set the session default:
SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL REPEATABLE READ;

-- Or set for the whole database:
ALTER DATABASE mydb SET default_transaction_isolation = 'serializable';

The phantom that Postgres prevents but the standard doesn’t require

One specific clarification worth making explicit before the next issue:

The SQL standard says REPEATABLE READ may allow phantom reads. It doesn’t say it must. Postgres’s REPEATABLE READ prevents phantoms because its snapshot isolation implementation is stricter than the standard’s minimum.

This creates a subtle portability issue: if you write an application that relies on REPEATABLE READ preventing phantoms, it will work correctly on Postgres but may exhibit phantom reads on MySQL InnoDB (which uses range locking rather than snapshot isolation for REPEATABLE READ). MySQL’s range locking does prevent phantoms through a different mechanism, so in practice both databases are safe — but the reason each is safe differs.

When writing cross-database code or switching databases, verify the isolation semantics, not just the level name.

Read more

In March 2018, a widely-read blog post from PostgreSQL contributor Tomas Vondra described something alarming: on Linux, if a page is evicted from the OS page cache while dirty, and then a subsequent write to that page fails, Postgres might not learn about the failure. The page might be silently discarded. And the next fsync() on that file might return success — because from the kernel’s perspective, there was nothing left to sync.

The PostgreSQL project spent months investigating. The conclusion was that Postgres, like most database software, had been making an assumption about fsync() semantics that was only partially true on Linux. A design change was made in PostgreSQL 12. But the investigation surfaced something more general and more unsettling: fsync() returning success is not the same as data being physically durable. The gap between those two things is where data loss lives.

This issue maps that gap precisely. We’ll cover the five levels of the durability stack between your application’s write() call and actual non-volatile storage, the three specific failure modes that cause fsync() to lie, and how to configure your database server to close each one.

What fsync() actually guarantees

Let’s start from the specification. The POSIX definition of fsync() states:

The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes.

“Transferred to the storage device associated with the file.” Not “written to non-volatile cells.” Not “protected from power failure.” Transferred to the storage device — meaning it has left the OS’s control and been handed to the device.

What the device does with it is not part of the POSIX specification.

This is the gap. fsync() guarantees that data has left the OS page cache and been sent to the storage device. Whether the storage device has written it to actual non-volatile storage — NAND flash cells, magnetic platter domains — depends on the device’s internal behaviour, which varies by hardware and configuration.

The durability ladder

Before the three lies, we need the full picture of where data can live between a write() call and non-volatile storage.

The animated diagram builds the five levels from most volatile to most durable, with a “power failure kills everything above this line” marker placed after level four.

Level 1 — Application memory: Your application’s heap. Lost on process crash, OOM kill, or SIGKILL. Data that has been write()‘d has left this level.

Level 2 — OS page cache: Kernel-managed RAM. Survives a process crash but not a kernel panic or power failure. write() returns success when data reaches this level. The OS will flush this to disk eventually, but “eventually” is not “now.”

Level 3 — Drive write cache (volatile DRAM): Many storage devices — HDDs and consumer SSDs — have a small internal RAM buffer where they stage writes before actually committing them to persistent storage. This speeds up write throughput dramatically (the drive can acknowledge the write instantly while scheduling the actual physical write for later). If power is cut while data is in this buffer, it’s gone.

Level 4 — Drive write cache (capacitor-backed or flash-backed): Enterprise SSDs and RAID controllers with a Battery Backup Unit (BBU) or supercapacitor-protected write cache store data in a non-volatile buffer that can survive a power outage long enough to flush to persistent storage on the next power cycle. This is “mostly safe” — but the BBU can degrade, the capacitor can fail, and the controller can crash.

Level 5 — Non-volatile storage cells: Data physically written to NAND flash cells or magnetic platter domains. Survives power loss. This is true durability. Getting data here is the entire point of the fsync() call.

The power failure boundary sits between levels 3 and 5 depending on your hardware. fsync() guarantees that data has left level 2. Whether it reaches level 5 before power can cut it is what the three lies are about.

Get Access to GitHub Link

Lie #1: Disk write cache enabled without power-loss protection

Consumer-grade HDDs and SSDs ship with write caching enabled by default. When the OS calls fsync() and the device acknowledges it, the device may be reporting that the data has reached its own internal RAM buffer — not that it has been written to persistent cells.

The acknowledgement is technically truthful at one level: the data has left the OS page cache and is in the drive’s control. But it is not in non-volatile storage. A sudden power cut will discard it.

How to verify this is happening on your system:

# Check if write cache is enabled on your drive:
sudo hdparm -I /dev/sda | grep -i "write cache"
# Output: * Write cache

# Check if it's actually enabled:
sudo hdparm -W /dev/sda
# Output: write-caching = 1 (on) -- this is the dangerous state

How to fix it:

# Disable the write cache (survives reboots only with hdparm.conf):
sudo hdparm -W 0 /dev/sda

# Verify:
sudo hdparm -W /dev/sda
# write-caching = 0 (off) -- fsync() now waits for actual cell writes

The performance cost is real. Disabling the write cache means every fsync() blocks until the storage device has physically written the data. On a consumer SSD, the difference is often 2–5x slower for write-heavy workloads. On an HDD, the difference can be 10–20x.

This is why enterprise SSDs matter for database servers: they have capacitor-backed write caches that are non-volatile. The write cache can remain enabled (for performance), and the data is still safe if power is cut — the capacitor provides enough energy to complete the flush.

For virtual machines and cloud instances: AWS EBS volumes, GCP Persistent Disks, and Azure Managed Disks all provide durable write semantics — they handle the write cache properly at the storage layer. You don’t need to disable write caching on cloud block storage. But you should verify the documentation of any storage provider before assuming this.

Lie #2: RAID controller with degraded or failed BBU

RAID controllers sit between the OS and the physical drives. When a write comes in, a RAID controller typically acknowledges it to the OS once the data is in the controller’s own cache — not once it’s been distributed to the drives. This dramatically improves write performance (the application isn’t waiting for multiple drives to write) and provides some protection (the controller can retry failed drive writes from its cache).

The battery backup unit (BBU) — or supercapacitor-based equivalent — is what makes this cache non-volatile. If the BBU is healthy and the power cuts, the controller keeps power long enough to flush its cache to the drives.

The problem: BBUs degrade. A RAID controller battery rated for three years may lose significant capacity by year two. When the BBU is degraded, the controller’s firmware detects this and switches to “write-through” mode — acknowledging writes only after they’ve been committed to the drives, not the cache. This is the correct safe behaviour, but it’s also 3–5x slower, and in some firmware implementations, it happens silently without alerting the DBA.

Even worse: some older RAID controllers, when the BBU is charging after a replacement, temporarily switch to write-through mode. A DBA replaces a degraded battery, and for the 24–48 hours the new battery charges, database write performance suddenly drops dramatically.

How to monitor BBU health:

# For MegaRAID controllers:
sudo megacli -AdpBbuCmd -GetBbuStatus -aALL | grep -E "Battery State|Relative Charge"

# For HPE Smart Array:
sudo ssacli ctrl all show status | grep -A5 "Cache"

# In your monitoring stack, alert on:
# - BBU charge below 85%
# - Write-through mode enabled
# - BBU replacement needed

What to do when BBU fails: Either replace it immediately and accept temporary performance degradation in write-through mode, or configure the database with synchronous_commit = off temporarily to reduce fsync frequency — acknowledging the increased (small) data loss window during the transition.

Lie #3: The Linux page cache eviction bug (and its relatives)

This is the most subtle lie, discovered in the Postgres investigation mentioned at the top of this issue.

On Linux, the OS page cache is not just a write buffer — it also caches file data for reads. When a write comes in for a page that’s already in the page cache, the write updates the cached page. When memory pressure is high, the kernel may evict dirty pages from the page cache to reclaim memory — before they’ve been written to disk.

Under normal operation, this is fine. The kernel tracks dirty pages and ensures they’re written to disk (via pdflush/kworker) before eviction, or marks them as needing a write and tracks them.

But here is the precise failure scenario that the Postgres investigation exposed:

1. Postgres writes dirty page to OS page cache (dirty)

2. Memory pressure causes the kernel to decide to evict this page

3. The kernel attempts to write the page to disk as part of eviction

4. The disk write fails (e.g., transient I/O error, storage temporarily unavailable)

5. On some Linux kernel versions: the page is evicted from the cache. The in-memory dirty state is discarded. The error is recorded in the page state — but not necessarily surfaced to any process that subsequently calls fsync() on that file

6. Postgres calls fsync() on the WAL file or data file

7. fsync() returns success — because from the kernel’s current view, there are no dirty pages for this file to sync

8. Postgres has been told the data is durable. The data was never written.

This was a kernel-level behaviour that PostgreSQL’s WAL code had implicitly depended on not occurring. PostgreSQL 12 added explicit checks: Postgres now marks files with pending writes and will re-issue writes if it detects the kernel lost track of them. The kernel behaviour was also addressed in Linux kernel patches.

Why this matters beyond the specific bug: It illustrates that the contract between database software and the OS storage stack is not as airtight as engineers assume. Every layer in the durability stack makes assumptions about the layers below it. When any assumption breaks — hardware caches data that it shouldn’t, kernels lose track of dirty pages, virtual storage layers introduce unexpected buffering — data can be silently lost even when every API call returned success.

The defence: Use O_DIRECT (bypasses the page cache entirely), monitor disk I/O errors at the OS level, and keep your Linux kernel updated. For critical data, verify that your storage stack is end-to-end correct using tools like Jepsen (which we’ll cover in issue #143).

Read more

Here is a fact that should immediately raise a question: a write() system call is not atomic. You can write 8KB to a file and have the system crash after the first 4KB have been persisted. The storage hardware itself is not atomic — it has no concept of “complete this set of writes or undo all of them.” And yet, when Postgres tells you a transaction committed, you can be absolutely certain that either all of its changes are reflected in the database or none of them are. Not most of them. Not the first three of five updates. All, or none.

How is a non-atomic property — atomicity — built from hardware primitives that aren’t atomic?

The answer is one of the most elegant pieces of systems engineering in all of database design. It doesn’t require any hardware support beyond what we described in issue #001. It doesn’t require locking data pages. It doesn’t require any special hardware atomic instructions. It requires only the write-ahead log, two types of information stored in each log record, and a clear protocol for which writes happen in which order.

This issue traces a transaction from BEGIN to COMMIT (and from BEGIN to ROLLBACK) at the byte level — every write, every fsync, every piece of state that must exist for the guarantee to hold.

Subscribe now

The transaction state machine

Before we go to bytes, we need the conceptual model. Every transaction moves through a precise sequence of states. The guarantee of atomicity is that no intermediate state is ever observable to another transaction.

The animated diagram builds the state machine step by step:

ACTIVE — The transaction is executing SQL statements. It’s reading and writing rows. Everything is happening in memory (shared_buffers) and in WAL buffers. Nothing is on disk yet. From the outside, none of these changes are visible.

PARTIALLY COMMITTED — The transaction has executed its last statement. All WAL records for the transaction have been written to the WAL buffer in memory. The system is about to fsync.

COMMITTED — The WAL has been fsynced to disk. The COMMIT record exists in durable storage. This is the point of no return. The transaction’s changes will survive any subsequent crash. And only now — not before — are the changes visible to other transactions.

ABORTED — Either the application issued ROLLBACK, or an error occurred, or the fsync failed. The transaction is being unwound. No changes from this transaction should be visible to anyone, ever.

DONE — Resources released, locks dropped, transaction ID freed for reuse.

The critical insight is the gap between PARTIALLY COMMITTED and COMMITTED. In that gap, the transaction has finished executing, but its changes are not yet durable and not yet visible. The fsync is what bridges the gap. Before the fsync, the transaction is in a liminal state — it’s done from the application’s perspective, but the database is still making it permanent.

The application blocks at COMMIT until the fsync completes and the state transition to COMMITTED is confirmed. This is why commit latency on an HDD is 10–30ms: the application is waiting for physical confirmation from the storage device.

What is actually written to the WAL

The WAL is a sequential append-only file. Every change to every data page generates one or more WAL records. The records are written in strict order. Let’s look at exactly what’s inside one.

LSN (Log Sequence Number) — 8 bytes. A monotonically increasing identifier for this position in the WAL stream. LSNs are how Postgres tracks “how far has this replica replicated?” and “from which point does recovery need to replay?” Every WAL record has a unique LSN. You can query the current WAL LSN in Postgres with SELECT pg_current_wal_lsn().

Transaction ID (XID) — 4 bytes. Which transaction produced this record. Postgres uses 32-bit transaction IDs, which is why transaction ID wraparound (the famous “XID wraparound” issue) is a real operational concern — running out of 32-bit XIDs would mean the database could no longer distinguish old from new.

Record type — 1 byte. HEAP_INSERT, HEAP_UPDATE, HEAP_DELETE, XLOG_XACT_COMMIT, XLOG_XACT_ABORT, and others. The record type tells the recovery process what operation to replay or undo.

Relation OID and block number — Which table (by object identifier) and which 8KB page within that table was modified.

Before-image (old tuple) — The complete state of the row before this change. Variable length. This is the undo information. If the transaction must be rolled back, Postgres applies this image to restore the row to its pre-transaction state.

After-image (new tuple) — The complete state of the row after this change. Variable length. This is the redo information. If the system crashes after the WAL record is written but before the data page is written to disk, recovery replays this image to reconstruct the correct data page state.

CRC — A 32-bit checksum over the entire record. On recovery, Postgres verifies the CRC before replaying any record. A corrupted WAL record stops recovery rather than applying corrupt data.

The presence of both before-image and after-image in the same record is the key to atomicity. The after-image enables redo (crash recovery forward). The before-image enables undo (rollback). Both directions of atomicity — completing and cancelling — are possible from a single log record.

The exact commit sequence: what happens in what order

Let’s trace a specific transaction to its individual write operations. The transaction:

BEGIN;

UPDATE accounts SET balance = balance - 100 WHERE id = 42;

UPDATE accounts SET balance = balance + 100 WHERE id = 99;

COMMIT;

Here is every write, in order, with nothing omitted:

Step 1 — BEGIN received. A transaction ID (XID) is allocated from shared memory. A BEGIN WAL record is written to the in-memory WAL buffer. The data pages for accounts 42 and 99 are loaded into shared_buffers from disk (if not already cached). Nothing has been written to disk.

Step 2 — First UPDATE executes. Postgres reads the current tuple for account 42 from shared_buffers. It creates a new tuple in the same or a nearby page with balance = old_balance - 100. Both tuples now exist in the page (this is MVCC — the old version for readers, the new version for this transaction). A WAL record is written to the in-memory WAL buffer: before-image is the old account 42 tuple, after-image is the new account 42 tuple.

Step 3 — Second UPDATE executes. Same process for account 99. WAL record written to buffer.

Step 4 — COMMIT received. This is where atomicity is established:

a) Write COMMIT WAL record to WAL buffer

b) Flush WAL buffer to WAL file on disk (write() + fsync())

← THIS IS THE BLOCKING OPERATION. Application waits here.

c) fsync() returns: storage device confirms data is durable

d) COMMIT returns to the application: “Transaction committed”

The data pages for accounts 42 and 99 in shared_buffers are still dirty — they haven’t been written to the data files on disk. The background writer process will eventually flush them, but not in the commit path.

Step 5 — Data pages flushed asynchronously. The background writer process or the checkpoint process flushes the dirty pages to their data files. This happens after the commit and is not visible to the committing transaction. It may happen milliseconds or minutes after the commit, depending on load.

Now here is the critical question: if the machine crashes between Step 4 and Step 5 — after the WAL is durable but before the data pages are written to disk — what happens on recovery?

Postgres opens the WAL file, finds the COMMIT record for XID 7421, sees that there is a committed transaction whose data page changes are not yet reflected in the data files, and replays the after-images from the WAL records for each UPDATE. The data pages end up in exactly the state they would have been in had the machine not crashed. The transaction is complete. Nothing is lost.

This is redo recovery. The WAL is always written before the data pages. This ordering — WAL before data — is the core invariant of write-ahead logging. It’s why the mechanism is called write-ahead: the log is always written ahead of the data.

How ROLLBACK works without a second disk write

Now let’s trace a rollback:

BEGIN;

UPDATE accounts SET balance = balance - 100 WHERE id = 42;

-- Something goes wrong

ROLLBACK;

After the UPDATE, the situation is:

• shared_buffers has the modified page for account 42 (balance = 900 in the new tuple)

• The WAL buffer has a WAL record with before-image (1000) and after-image (900)

• The data file on disk still has the original value (balance = 1000) — the async write hasn’t happened yet

When ROLLBACK is issued:

Step 1 — Postgres writes an ABORT WAL record to the WAL buffer. This marks the transaction as aborted. It does not need to be fsynced immediately (though it will be written to disk eventually) — an uncommitted transaction’s changes are never visible to other transactions, so there’s no data loss if the ABORT record is lost in a crash.

Step 2 — Postgres reads the before-image from the WAL buffer (in memory — no disk read required). For account 42, the before-image is balance = 1000.

Step 3 — Postgres writes the before-image back into the shared_buffers page, overwriting the new tuple’s data. The page now has balance = 1000 again — the pre-transaction state.

Step 4 — The new tuple (balance = 900) is marked as dead. VACUUM will eventually reclaim its space.

That’s it. ROLLBACK is fast because:

1. The data file was never touched — there’s nothing to undo there

2. The before-image is in the WAL buffer, which is in memory

3. Restoring the before-image is a single memory write

No disk read. No disk write (beyond the ABORT WAL record). No seeking through the data file. Rollback’s cost is proportional to the number of rows modified, not to the size of the table.

The misconception about ROLLBACK: Many engineers assume ROLLBACK “reverses” the writes to disk. It doesn’t. It reverses the writes to shared_buffers. The disk was never involved. The WAL’s before-image is the mechanism that makes this possible — it stores the undo information alongside the redo information in the same record, so both directions of recovery are available without any additional state.

Why the WAL must be fsynced before data pages

There is a precise ordering requirement in write-ahead logging that, if violated, destroys the atomicity guarantee. Let’s examine what would happen if data pages were written to disk before the WAL was fsynced.

Scenario: Postgres writes the modified account 42 data page to disk (balance = 900), then crashes before the WAL is fsynced.

On recovery: Postgres reads the data file. It sees balance = 900 for account 42. It looks at the WAL for any un-replayed transactions. It finds... nothing (the WAL wasn’t fsynced). Postgres concludes: “This data is consistent — there are no WAL records to replay.” But the data is not consistent. Account 42’s balance was changed by a transaction that was never committed. The change is permanent, and there’s no record of what the original value was.

This is a violation of atomicity: a transaction’s changes are visible even though the transaction was never committed.

The only way to prevent this is the WAL-before-data ordering:

1. WAL records for the transaction are written and fsynced

2. Only after the WAL is durable can data pages be written to disk

Postgres enforces this through a protocol called WAL-logged page writes. Before any dirty page is flushed to disk, Postgres checks that all WAL records that describe changes to that page have been written to disk. This is tracked per page using the page’s LSN: the page stores the LSN of the most recent WAL record that describes a change to it. The background writer will not flush a page to disk until the WAL has been flushed to at least that page’s LSN.

The full_page_writes parameter in Postgres tightens this further: after a checkpoint, the first modification to a page writes the entire page image to the WAL (not just the change). This is the full-page write. It ensures that even if a page is torn during writing (half-written to disk, then crash), the WAL contains the complete before-and-after state needed to reconstruct the page correctly.

Savepoints: atomicity within atomicity

SQL supports savepoints — named points within a transaction to which you can partially roll back:

BEGIN;

UPDATE inventory
SET quantity = quantity - 10
WHERE sku = 'WIDGET';

SAVEPOINT after_inventory;

UPDATE orders
SET status = 'confirmed'
WHERE id = 9901;

-- This fails with a foreign key violation

ROLLBACK TO SAVEPOINT after_inventory;

-- The inventory update survives; the orders update is undone

COMMIT;

Savepoints are implemented exactly like full transaction rollback, but scoped. When a savepoint is created, Postgres records the current WAL LSN as the savepoint boundary. If a rollback to savepoint is requested, Postgres applies before-images from WAL records written after that LSN — only those records, not the ones written before the savepoint.

The WAL’s sequential structure makes this straightforward: records before the savepoint LSN are left alone; records after the savepoint LSN are undone using their before-images. The transaction continues as if the rolled-back portion never happened.

This is also why long transactions that modify many rows are expensive to roll back — Postgres must apply before-images for every WAL record written during the aborted portion. The cost is proportional to the work done, not to the table size, but for a transaction that has modified a million rows, that’s a million before-image applications.

The synchronous_commit lever

There’s a performance escape hatch that’s worth understanding in this context: synchronous_commit.

When synchronous_commit = on (the default), Postgres waits for the WAL fsync before returning COMMIT to the application. This is the full guarantee: committed transactions survive any crash.

When synchronous_commit = off, Postgres returns COMMIT to the application before the WAL is fsynced. The WAL will be fsynced shortly afterward (within wal_writer_delay, default 200ms), but the application gets its commit confirmation first.

The risk: if the machine crashes in the window between the application receiving COMMIT and the WAL being fsynced, that transaction’s changes are lost. The database will not be corrupted — the WAL is consistent, just missing the very recent transactions. But those transactions are gone, despite the application having been told they committed.

For workloads where occasional data loss is acceptable (logging, metrics, analytics ingestion), synchronous_commit = off can increase write throughput dramatically — it turns what would be 1,000 sequential fsyncs per second (each blocked, each waiting for storage confirmation) into a much smaller number of batched fsyncs.

For any workload involving financial data, user-visible state, or any operation where “committed” must mean durably committed, leave synchronous_commit = on.

The parameter can even be set per-session or per-transaction, letting you mix durability levels within a single application:

-- For this session only: high-durability mode

SET synchronous_commit = on;

BEGIN;

INSERT INTO payments VALUES (...);

COMMIT;  -- blocks until fsynced

-- For this session only: high-throughput mode

SET synchronous_commit = off;

BEGIN;

INSERT INTO page_views VALUES (...);

COMMIT;  -- returns immediately, fsync happens within 200ms

The practical mental model for atomicity

Atomicity is not a property of storage hardware. It’s a property of protocol.

The WAL protocol achieves atomicity through three rules:

1. Write WAL records before data pages

2. Write COMMIT to WAL (and fsync) before acknowledging commit to the application

3. On recovery, replay all WAL records for committed transactions; apply before-images for all uncommitted transactions

If these three rules are followed, any crash at any point leaves the database in a state where every visible transaction is fully committed and every non-visible transaction has left no trace. The atomicity guarantee holds regardless of when the crash occurs — before the WAL is written, mid-WAL write, after the WAL but before the data pages, mid-data-page write, at any point.

This is why “add a WAL” is not just a database feature — it’s the engineering solution to the fundamental impossibility of atomic writes on hardware that doesn’t support them natively. And it’s why almost every storage system that needs durability — databases, distributed file systems, operating system journals, even the internal state machines of SSDs — uses some variant of this pattern.

Read more

By the early 1980s, relational databases were commercially successful. Oracle was selling software. IBM had DB2. Ingres had a growing user base. SQL was on its way to standardisation. CODD’s 1970 paper had won.

And then the marketing departments got involved.

Software vendors who had built non-relational systems — or partially relational systems — began advertising them as “relational databases” because the term had acquired commercial cachet. A system that supported SQL queries but stored data in hierarchical structures. A system that had foreign keys but no support for joins. A system that was relational in spirit but navigational in implementation. The word “relational” was being diluted to the point of meaninglessness.

In 1985, CODD published a two-part article in Computerworld magazine under the headline “Is your DBMS really relational?” He laid out 12 rules — numbered 0 through 12 — that define what a system must do to legitimately call itself a relational database. The rules are not aspirational guidelines. They are a minimum specification. A system that violates any of them, CODD argued, is selling something that isn’t what it claims to be.

Reading the rules today is illuminating not because every database satisfies all of them — none do — but because the ones that get broken reveal exactly where the relational model turns out to be genuinely hard to implement, and what the industry decided to sacrifice.

Rule 0: The foundation rule

A system advertised as a relational database management system must manage data through its relational capabilities exclusively.

Rule 0 is the meta-rule. It says that if a system calls itself relational, it must use the relational model for everything — not just for some operations while using a different model for others. A system that supports SQL queries but also exposes record IDs as physical pointers, or that allows navigational access through proprietary APIs, fails Rule 0.

This rule was aimed directly at the hybrid systems of the early 1980s: products that had bolted SQL on top of existing hierarchical or network databases, offering relational queries as a layer while the underlying data management remained navigational. If you could get better performance by bypassing the SQL layer and using the underlying navigational interface directly — and many of these systems let you do exactly that — you didn’t have a relational database. You had a navigational database with a SQL interface.

Why it matters today: Some NoSQL databases offer SQL-like query languages as a thin layer over non-relational data models. Apache Cassandra’s CQL looks like SQL but doesn’t support joins or transactions in the relational sense. Using these systems’ SQL syntax doesn’t give you the guarantees of the relational model. Rule 0 says: the model is what matters, not the syntax.

Rules 1 and 2: Information and guaranteed access

Rule 1 — All information in a relational database is represented at the logical level in exactly one and only one way: as values in tables.

Rule 2 — Each and every datum (atomic value) in a relational database is guaranteed to be logically accessible by resorting to a combination of table name, primary key value, and column name.

These two rules together establish the fundamental data representation contract. All data is in tables. All data is accessible by known logical names. There are no hidden identifiers, no physical row pointers visible to the user, no secondary representations that the programmer must know about.

Where this gets interesting with ORMs: Object-relational mapping libraries often expose internal identifiers — auto-increment integers, UUIDs — as primary keys that are visible to application code. This is fine and consistent with Rules 1 and 2. The problem arises when the ORM begins relying on implementation details: assuming that row IDs are sequential, or using physical offsets, or bypassing the SQL layer for “performance.” When an ORM’s behaviour depends on the physical storage layout rather than the logical schema, it’s violating the spirit of Rule 1.

The ctid issue in PostgreSQL: Postgres has a system column called ctid — the physical location of a row on disk, expressed as (page_number, tuple_offset). You can query it with SELECT ctid, * FROM table. This is a violation of Rule 1’s spirit — it exposes physical storage information as a queryable value. Postgres doesn’t advertise ctid as a user-facing feature for exactly this reason. It exists for internal use and should never appear in application code.

Rule 3: Systematic NULL treatment

Null values (distinct from the empty character string or a string of blank characters and distinct from zero or any other number) are supported in fully relational DBMS for representing missing information and inapplicable information in a systematic way, independent of data type.

This rule acknowledges something genuinely uncomfortable: sometimes information is unknown or inapplicable, and the database must represent this in a consistent, type-independent way. NULL is not zero. NULL is not an empty string. NULL is the absence of a known value, and it must propagate correctly through all operations.

Why NULL is so strange — and why CODD’s rule is mathematically correct: NULL introduces three-valued logic into SQL. A comparison like salary > 50000 can evaluate to TRUE, FALSE, or UNKNOWN — unknown when salary is NULL. This propagates through boolean expressions: TRUE AND UNKNOWN = UNKNOWN, FALSE AND UNKNOWN = FALSE, TRUE OR UNKNOWN = TRUE.

This is internally consistent but deeply counterintuitive. Consider:

sql

SELECT * FROM employees WHERE salary > 50000 OR salary <= 50000;

This looks like it should return all rows — every salary is either above or below 50,000. But employees with NULL salary are returned by neither condition, because both comparisons evaluate to UNKNOWN. The query misses them.

Or consider the NOT IN trap:

sql

SELECT name FROM employees
WHERE dept_id NOT IN (SELECT dept_id FROM departments WHERE budget > 1000000);

If any row in the subquery returns NULL for dept_id, the entire NOT IN clause evaluates to UNKNOWN for every employee, and the query returns zero rows — silently. This is a genuine bug that has affected real production systems.

Postgres handles this correctly, mostly: Postgres implements three-valued logic faithfully and provides IS NULL, IS NOT NULL, IS DISTINCT FROM, and NULLIF to handle NULL explicitly. The behaviour is correct by CODD’s specification, even when it surprises developers.

Rule 4: Active online catalog

The database description is represented at the logical level in the same way as ordinary data, so that authorised users can apply the same relational language to its interrogation as they apply to the regular data.

The database schema — the table definitions, column types, constraints, indexes, views — is itself stored as data in tables and queryable with the same SQL used to query regular data. There’s no separate metadata API; there’s no special administrative interface required to understand the schema. The schema is just more tables.

In Postgres: The information_schema views and pg_catalog system tables implement Rule 4. You can query the schema exactly as you query your data:

sql

-- What tables exist in my database?
SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public';

-- What columns does the orders table have?
SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'orders';

-- What indexes exist on a table?
SELECT indexname, indexdef
FROM pg_indexes
WHERE tablename = 'orders';

This is elegant and powerful. It means introspection tools, schema migration frameworks, and database administration interfaces can all be built using standard SQL rather than proprietary metadata APIs. ORM frameworks use exactly this capability to understand your database schema at runtime without you having to tell them what tables exist.

Rule 5: Comprehensive data sublanguage

A relational system may support several languages and various modes of terminal use. However, there must be at least one language whose statements are expressible, per some well-defined syntax, as character strings and that is comprehensive in supporting all of the following: data definition, view definition, data manipulation (interactive and by program), integrity constraints, authorization, and transaction boundaries.

SQL satisfies Rule 5. A single language handles CREATE TABLE, CREATE VIEW, SELECT, INSERT, UPDATE, DELETE, CHECK constraints, GRANT/REVOKE, BEGIN/COMMIT/ROLLBACK. You don’t need separate tools for schema management, data manipulation, and transaction control.

Where vendors broke this in the 1980s: Early database products often had separate languages for DML (data manipulation) and DDL (data definition), or required transactions to be managed through an external library rather than within the query language itself. Separating these concerns was a Rule 5 violation — it meant you couldn’t express a complete database operation in a single language.

Rule 6: View updating

All views that are theoretically updatable must be updatable by the system.

This is the rule most databases break most visibly, and it’s worth spending real time on.

A view is a saved query — a named SELECT that you can treat as a table. The question Rule 6 asks is: if a view is a simple, unambiguous projection of a base table, should you be able to UPDATE through it?

CODD says yes — the database must determine which views are theoretically updatable and honour that.

What “theoretically updatable” means: A view is theoretically updatable if there is an unambiguous mapping between every row in the view and exactly one row in an underlying base table, and if every update to the view translates unambiguously into an update to the base table.

A view that selects a subset of rows and columns from a single table is theoretically updatable — every view row maps to exactly one base table row, and updating the view means updating that row.

sql

-- Theoretically updatable: simple projection
CREATE VIEW eng_staff AS
SELECT emp_id, name, salary FROM employee WHERE dept = 'Engineering';

-- This SHOULD work, and in Postgres it does:
UPDATE eng_staff SET salary = 95000 WHERE emp_id = 1;

A view that joins multiple tables, aggregates rows, or uses window functions is theoretically non-updatable — there’s no unambiguous base table row to modify.

sql

-- Not updatable: aggregate across rows
CREATE VIEW dept_stats AS
SELECT dept, AVG(salary) as avg_sal, COUNT(*) as headcount
FROM employee GROUP BY dept;

-- What would this mean? Which rows to change?
UPDATE dept_stats SET avg_sal = 90000 WHERE dept = 'Engineering';
-- ERROR in all databases — correctly rejected

Where modern databases partially break Rule 6: Postgres supports updating simple views automatically. But it doesn’t automatically support updating views that join two tables, even when the update is theoretically unambiguous (changing a column that maps clearly to one of the joined tables). You must create explicit INSTEAD OF triggers to enable this. MySQL has similar limitations. CODD would say this is a partial violation: the system could determine theoretical updatability for join views in many cases, but chooses not to.

Rule 7: High-level insert, update, and delete

The capability of handling a base relation or a derived relation as a single operand applies not only to the retrieval of data but also to the insertion, update, and deletion of data.

SQL satisfies this: UPDATE orders SET status = 'shipped' WHERE order_date < '2024-01-01' updates an entire set of rows in one statement, not one row at a time. This is a fundamental departure from navigational databases where you position a cursor on a single record and modify it.

Set-based operations are not just syntactic convenience — they enable the query optimiser to choose the most efficient execution strategy. A set-based UPDATE on indexed data can use the index; a row-by-row cursor update must access every row individually.

The RBAR anti-pattern: RBAR (Row By Agonizing Row) is the name given to the common mistake of writing SQL that processes one row at a time — typically via a cursor in a stored procedure — when a set-based operation would be more efficient. It’s a violation of Rule 7’s spirit: you’re bypassing the relational engine’s set-processing capabilities to implement navigational logic in SQL.

sql

-- RBAR: processing one row at a time
DECLARE cur CURSOR FOR SELECT id FROM orders WHERE status = 'pending';
OPEN cur;
FETCH NEXT FROM cur INTO @id;
WHILE @@FETCH_STATUS = 0 BEGIN
    UPDATE orders SET status = 'processing' WHERE id = @id;
    FETCH NEXT FROM cur INTO @id;
END;

-- Rule 7: set-based, as CODD intended
UPDATE orders SET status = 'processing' WHERE status = 'pending';

The second version can leverage indexes, run in parallel, and be optimised by the query planner. The first version cannot.

Rules 8 and 9: Physical and logical data independence

Rule 8 — Application programs and terminal activities remain logically unimpaired whenever any changes are made to either storage representations or access methods.

Rule 9 — Application programs and terminal activities remain logically unimpaired when information-preserving changes of any kind that theoretically permit unimpairment are made to the base tables.

These are the two data independence rules from CODD’s 1970 paper, now formalised.

Rule 8 (physical independence) is generally well-honoured: you can add indexes, change storage parameters, partition tables, move data between tablespaces, and application SQL stays unchanged. This is the rule that lets DBAs tune performance without requiring application code changes.

Rule 9 (logical independence) is where things get interesting — and where most databases only partially comply. Adding a new column to a table (an information-preserving change) should not break existing queries. In Postgres and MySQL, it doesn’t — queries that select specific columns continue to work. But queries that use SELECT * will now return the new column, which may surprise applications that expected a fixed column count.

The harder cases: Splitting one table into two (decomposing a wide table with mixed concerns) or merging two tables are information-preserving changes — the data hasn’t changed, just its organisation. Rule 9 says these changes shouldn’t require application modifications. In practice, they usually do — applications that named the original table must be updated to use the new table names or a compatibility view.

True Rule 9 compliance would mean the database automatically maintains compatibility views that preserve the old interface after structural reorganisation. No major database does this automatically.

Rules 10, 11, and 12: Integrity, distribution, and non-subversion

Rule 10 — Integrity constraints specific to a particular relational database must be definable in the relational data sublanguage and storable in the catalog, not in the application programs.

Constraints belong in the database, not in the application. CHECK (price > 0), FOREIGN KEY (customer_id) REFERENCES customers(id), UNIQUE (email) — these are Rule 10. When constraints live in application code, a different application that bypasses the ORM (a data migration script, an admin tool, a reporting query) can violate them silently.

Rule 11 — A relational DBMS has distribution independence.

A database should appear identical whether it is a single-server system or a distributed system. Queries written for a non-distributed database should work unchanged on a distributed one. This is the hardest rule to fully satisfy — distributed systems introduce consistency trade-offs (CAP theorem, which we’ll cover in issue #059) that make true location transparency impossible without performance costs. Every distributed database makes some compromise here.

Rule 12 — If a relational system has a low-level (single-record-at-a-time) language, that low-level language cannot be used to subvert or bypass the integrity rules and constraints expressed in the higher-level relational language.

Stored procedures, triggers, and embedded SQL must respect the constraints defined at the relational level. You cannot write a stored procedure that sets a foreign key to an invalid value by operating on raw storage structures. The relational constraints are non-negotiable, regardless of the interface used to access the data.

Where MySQL historically broke Rule 12: For many years, MySQL’s MyISAM storage engine did not enforce foreign key constraints — even when they were defined in the schema. Application code could insert orphaned records (orders without valid customer IDs) through the normal SQL interface, and the engine would silently accept them. InnoDB enforced foreign keys correctly; MyISAM did not. This was a clear Rule 12 violation — the constraint was defined in the relational schema but not enforced.

The compliance scorecard

The scorecard shows how four systems — PostgreSQL, MySQL, SQLite, and MongoDB (as a NoSQL comparison point) — score against all 13 rules. The results are instructive:

PostgreSQL is the most compliant major database, passing most rules and partially complying with the genuinely difficult ones (Rule 6 view updating, Rule 9 logical independence, Rule 11 distribution independence for single-node deployments).

MySQL has historically been more permissive — silent truncation of out-of-range values, optional foreign key enforcement, non-standard NULL handling in some edge cases. Modern MySQL (8.x) is significantly more correct than older versions, but its compliance history is checkered.

SQLite is intentionally a single-user embedded database, not a server. It makes explicit compromises: loose typing (”type affinity” rather than strict types), no stored procedures, limited view updating. It’s not trying to be a fully relational system in the way Postgres is — it’s optimising for embeddability.

MongoDB fails nearly every rule — not because it’s poorly built, but because it is deliberately not a relational database. It has a different model with different trade-offs. Measuring it against CODD’s rules is useful precisely to show what you give up when you leave the relational model.

Why these rules still matter

CODD wrote these rules in 1985 to protect a word — “relational” — from being diluted. Forty years later, the relevant question isn’t really “does your database pass CODD’s rules?” The relevant question is: for each rule your database violates, do you know what you’re giving up?

A database that partially violates Rule 9 (logical independence) means that some schema changes will require application code changes — you should plan for that. A database that violates Rule 11 (distribution independence) means that moving from a single-server to a distributed setup will require query modifications — you should know that before committing to a distributed architecture. A database that violates Rule 12 means that bypassing the ORM with raw SQL can silently corrupt your data — you should enforce discipline around who can write migration scripts.

The rules are not a test to pass. They’re a map of where the relational model’s guarantees end and your application’s responsibilities begin.

The year is 1966. NASA is three years away from putting a man on the Moon. The Apollo programme is the largest engineering project in human history, involving 400,000 people, 20,000 contractors, and roughly 10 million individual parts. IBM wins the contract to build the information system that tracks all of it — what part goes where, what’s been ordered, what’s in stock, what’s been installed on which spacecraft.

The system they build is called IMS: the Information Management System. It stores data in tree structures — hierarchies of parent and child records — and processes it with transaction management that prevents the data corruption problems we covered in issue #004. IMS works. It helps put humans on the Moon. Versions of it are still running in banks and airlines today, half a century later, processing billions of transactions annually.

IMS is also, by the measure of what came after it, a design with a fundamental flaw. Not in its engineering — the engineering is superb. The flaw is in its data model. And understanding exactly what that flaw is, and why it took until 1970 for someone to articulate the fix, is one of the most illuminating stories in the history of computing.

IMS: the hierarchical model

IMS organises data as a forest of trees. Every record type is either a root or a child of exactly one parent type. A company has departments. A department has employees. An employee has projects. The physical storage on disk mirrors this hierarchy — children are stored near their parent, making navigation along the hierarchy fast.

The animated diagram builds the tree structure piece by piece: the company root, then department records branching from it, then employee records hanging from each department, then project assignment records below each employee. Each level is a different record type with its own fields. Navigation moves along the branches — to find an employee’s projects, you start at the root, navigate to the right department, navigate to the right employee, and then access their project children.

What IMS does well

Concurrent access and recovery. IMS had proper locking, transaction management, and crash recovery from the beginning. The flat file problems from issue #004 — concurrent write corruption, atomicity failures, no crash recovery — are all handled by IMS. It is a real database engine in the most important sense.

Hierarchical access is fast. If your access pattern follows the hierarchy — “give me all projects for this specific employee in this specific department” — IMS is very fast. Children are stored physically near their parents. Navigation along a known path is efficient.

Consistent, structured records. IMS enforces the schema. Records have defined fields with defined types. You can’t accidentally write a string into an integer field.

What IMS cannot do well

Cross-branch queries. The animated diagram shows this clearly. The query “find all employees currently working on the Apollo project” requires traversing every branch of the tree — every department, every employee within each department — and checking whether that employee has an Apollo project child. There is no way to go directly from a project name to the employees on it, because the hierarchy runs in one direction only: from company down to project. Going the other way means a full scan.

Multiple parent relationships. In reality, an employee might belong to multiple departments (a matrix organisation). A project might be assigned to employees across different departments. IMS trees cannot represent this. Each record has exactly one parent. Relationships that aren’t hierarchical require either data duplication (store the employee record in multiple departments) or a workaround that degrades the query performance IMS was designed to provide.

Schema changes. Adding a new field to the employee record type requires restructuring the physical storage of every employee record on disk. In a large IMS installation with millions of records, this is a multi-day offline operation. The application code that navigates the hierarchy is tightly coupled to the physical layout — change the layout, and you must find and update every program that touches those records.

The deeper problem, which IMS shares with all navigational databases, is this: the application code knows the physical structure of the data. Every program that reads or writes IMS records contains explicit navigation logic: go to this record type, follow this parent-child link, find this field at this byte offset. When the physical structure changes, every program that depends on it must change too.

CODASYL: the network model

In 1969, the Conference on Data Systems Languages — CODASYL — published a specification for a more general database model. Rather than limiting relationships to parent-child trees, the CODASYL model allowed any record to participate in multiple named sets — directed relationships between record types that could form arbitrary graphs rather than trees only.

The animated diagram shows a CODASYL schema with DEPARTMENT, EMPLOYEE, PROJECT, and ASSIGNMENT record types. An EMPLOYEE belongs to the EMP-IN-DEPT set (owned by DEPARTMENT) and also participates in the EMP-ASSIGNS set (owned by ASSIGNMENT). An ASSIGNMENT participates in both EMP-ASSIGNS and PROJ-ASSIGNS. The graph is more expressive than a tree — an employee can have relationships to multiple departments, multiple assignments, multiple projects — all without data duplication.

What CODASYL improved over IMS

Many-to-many relationships. CODASYL could represent relationships that IMS required data duplication for. An employee on multiple projects, a project with employees from multiple departments — these are natural in the CODASYL model.

Richer query paths. Because relationships are bidirectional and an entity can participate in multiple sets, you can navigate from any direction. To find employees on the Apollo project, you start at the Apollo PROJECT record, follow the PROJ-ASSIGNS set, and reach all ASSIGNMENT records, from which you follow EMP-ASSIGNS backwards to reach the employees. No full-tree scan required.

What CODASYL didn’t fix

Look at the navigation code in the animated diagram:

FIND FIRST DEPARTMENT WHERE dept_id = 10
FIND FIRST EMPLOYEE WITHIN EMP-IN-DEPT
WHILE FOUND:
    FIND FIRST ASSIGN WITHIN EMP-ASSIGNS
    WHILE FOUND:
        GET PROJECT VIA PROJ-ASSIGNS
        FIND NEXT ASSIGN WITHIN EMP-ASSIGNS
    FIND NEXT EMPLOYEE WITHIN EMP-IN-DEPT

Eight lines of code to answer what we would today express as a two-table JOIN. And this is the simple version — real CODASYL programs navigating complex schemas ran to hundreds of lines of cursor manipulation code.

The fundamental problem is unchanged from IMS: the application must know the physical structure of the data and navigate it explicitly. The programmer must know which set to follow, which direction to traverse, which record type sits at each position in the graph. The query specifies not what you want but how to physically retrieve it.

This creates what database historian C.J. Date called the navigation problem. When you want different data, you write different navigation code. When the schema changes, you rewrite all the navigation code that touched the changed portion. The application logic and the data structure are deeply entangled. There is no separation between the question being asked and the physical process of answering it.

The cursor-based interface

CODASYL uses a cursor — a pointer to a current record — as the primary interface. You move the cursor around the graph: FIND FIRST, FIND NEXT, GET CURRENT. The programmer explicitly manages cursor position throughout the program. This is not an abstraction — it exposes the physical navigation mechanism directly.

For a programmer who knows the schema and has written CODASYL code before, this is workable. For a business analyst who wants to ask an ad hoc question — “how many employees across all departments are assigned to Apollo?” — it is completely inaccessible. Only someone who knows the physical schema intimately can write the code to answer the question.

This was the state of the art in 1969. Then CODD wrote his paper.

CODD’s insight: separate the logical from the physical

Edgar F. CODD was a British mathematician working at IBM’s San Jose Research Lab when he published “A Relational Model of Data for Large Shared Data Banks” in the Communications of the ACM in June 1970. He was 39 years old. The paper is 11 pages. It changed everything.

CODD’s core insight sounds simple in retrospect: the logical structure of data should be independent of its physical storage.

IMS and CODASYL forced the programmer to know how data was physically laid out on disk — which record type was the parent, which set to follow, which pointer to dereference. This created an application dependency on physical structure that made schema changes expensive and ad hoc queries impossible for non-programmers.

CODD proposed that the database present data as a collection of simple, flat relations — what we now call tables. Each relation has named attributes (columns) and a set of tuples (rows). Relationships between tables are expressed by shared values (keys), not by physical pointers. The programmer expresses what data they want using operations on these relations. The database engine figures out the physical retrieval.

This separation is called data independence: the logical data model (tables and their relationships) is independent of the physical storage model (how those tables are laid out on disk, what indexes exist, how they’re partitioned). Change the physical storage — add an index, change the file layout, move data to a different disk — and the logical queries still work unchanged. The application code doesn’t know or care about the physical structure, because it never navigates it directly.

The animated diagram shows the same data from the IMS and CODASYL examples — departments, employees, projects, assignments — reorganised into flat tables. EMPLOYEE has a dept_id foreign key referencing DEPARTMENT. PROJECT_ASSIGN connects employees to projects by value, not by pointer. To query across the tables, you write a JOIN expressed as equality on shared key values.

Relational algebra: the mathematical foundation

CODD didn’t just propose a data model — he grounded it in mathematics. The relational model is based on set theory and first-order predicate logic. A relation is a set of tuples, and queries are expressed as operations on sets: select (filter rows), project (choose columns), join (combine tables on a condition), union, intersection, difference.

This mathematical foundation has two important consequences.

First, queries have well-defined semantics. A SQL query has a precise mathematical meaning that is independent of how the database executes it. Two queries that express the same set-theoretic operation are guaranteed to produce the same result, regardless of which execution path the database chooses. In CODASYL, different navigation paths to the same data could — under certain concurrency conditions — produce different results.

Second, the query optimiser is possible. Because the logical query is separated from the physical execution, the database engine can choose different physical execution strategies for the same logical query — use an index or not, choose a hash join or a merge join, execute in parallel or sequentially — based on the actual data distribution. The programmer never has to specify the physical strategy. The engine picks it.

This is the crucial enabler of everything we’ll study in issues #030–#035: how a query planner transforms a logical query into an efficient physical execution plan. None of that is possible without data independence — the clean separation between what you ask for and how the database retrieves it.

What CODD’s paper actually said

The paper opens with a precise diagnosis of the problem:

“Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation).”

This single sentence explains IMS, CODASYL, and every navigational database. Users — including application programmers — shouldn’t need to know the physical layout. They should express queries in terms of the data’s logical meaning.

CODD then introduces the relational model: relations as mathematical sets, normal forms to eliminate redundancy, and relational algebra as the query language. The paper defines the first, second, and third normal forms — the foundations of relational schema design we’ll cover in depth in issues #050 and #051.

The paper also introduces the concept that became the PRIMARY KEY: the notion that every tuple in a relation is uniquely identified by some minimal subset of its attributes, called the primary key. Duplicate rows aren’t allowed — a relation is a set, and sets don’t have duplicates.

And it introduces foreign keys: the mechanism by which one relation references another by value. An EMPLOYEE tuple’s dept_id value must match the dept_id of some DEPARTMENT tuple. This is not a pointer — it’s a constraint on values. The constraint can be checked without knowing where either tuple is physically stored.

Why IMS and CODASYL people resisted

CODD’s paper met substantial resistance, particularly from within IBM, where IMS was a commercial product with paying customers and an enormous installed base.

The main technical objection was performance. In 1970, computers were slow enough that the query optimisation overhead seemed prohibitive. Navigating a known path through an IMS hierarchy was faster than letting a general-purpose optimiser figure out a query plan. The IMS performance argument was legitimate — for the access patterns IMS was designed for, it was genuinely faster than early relational implementations.

The deeper resistance was organisational. Large organisations had invested heavily in IMS and CODASYL systems. The navigation code was written. The schemas were designed. The database administrators knew the systems intimately. The idea that all of it should be replaced by a new model — however elegant — was threatening.

CODD’s relational model took roughly a decade to move from paper to production. The first commercial relational databases — Oracle (then called Relational Software), IBM’s own DB2, and Ingres — arrived in the late 1970s and early 1980s. SQL, the language that implemented relational algebra in a syntax accessible to non-mathematicians, was standardised in 1986. By the early 1990s, relational databases dominated enterprise computing. IMS and CODASYL systems were legacy infrastructure.

The performance objection was resolved not by making relational systems slower — but by making hardware faster. By the mid-1980s, hardware had improved enough that a well-optimised relational query could outperform hand-written CODASYL navigation, because the optimiser could choose strategies that a human programmer wouldn’t have thought of.

What IMS and CODASYL got right that is still true

It’s easy to read this history as “IMS and CODASYL were bad and the relational model was good.” That’s too simple.

Transaction management. IMS had proper ACID transactions before the relational model was even proposed. The need for atomicity, consistency, isolation, and durability was understood and implemented in hierarchical and network databases. The relational model inherited this; it didn’t invent it.

Physical data organisation still matters. The relational model achieves data independence at the logical level — your SQL query doesn’t specify the physical access path. But the physical storage still exists, and it still matters enormously for performance. When we study B-trees in issue #018, we’ll see that the physical organisation of data on disk — how related rows cluster together, how indexes are structured — determines query performance just as much as it did in IMS. The difference is that modern databases let the programmer express the logical query without specifying the physical path, while still allowing the physical structure to be tuned separately.

The navigational access pattern is sometimes right. There are workloads — graph traversals, recursive hierarchies, certain time-series access patterns — where navigating explicit relationships is genuinely more efficient than joining flat tables. This is part of why graph databases exist today. The relational model is not always the right model. But it is the right default, for most workloads, most of the time.

CODD’s insight wasn’t that navigation was wrong — it was that navigation shouldn’t be the programmer’s responsibility. The database should choose the navigation path. The programmer should express what they want. That separation of concerns is what made relational databases the foundation of modern data infrastructure.

That model is accurate enough for writing a log file or saving a configuration. For a database engine managing hundreds of gigabytes under concurrent write load, it breaks down in three specific places — and each place is a source of bugs, data loss, and performance degradation that has burned real systems in production.

This issue is about those three leaks. Not because database engineers need to fix the OS — they don’t — but because understanding where the abstraction breaks explains every unusual thing databases do when writing and reading files. The WAL, the fsync dance, the pre-allocation of large extents, the O_DIRECT flag, the careful tuning of checkpoint_completion_target — all of it makes immediate sense once you see what the OS abstraction doesn’t actually guarantee.

From filename to bytes: the three-layer translation

When Postgres opens its main data file, the OS doesn’t just hand it a pointer to raw bytes on disk. It translates the filename through three distinct layers before a single byte flows.

Layer 1: directory entry → inode

The filename pg_data is stored in a directory entry — a simple mapping from a human-readable name to an inode number. An inode is a fixed-size data structure stored at a known location on the filesystem. It contains the file’s metadata (size, permissions, timestamps, ownership) and, critically, the list of block addresses where the file’s actual data is stored.

This separation — name from inode, inode from data — is deliberate. It means a file can have multiple names (hard links) pointing to the same inode. It means renaming a file is cheap (you just update the directory entry) rather than requiring moving all the data. It means file metadata and data can be updated independently.

For databases, the inode layer creates a subtle constraint: renaming a file is atomic at the OS level, but creating a new file, writing data to it, and then renaming it to replace the old one is the standard pattern for safe file replacement — and whether the rename survives a crash depends on whether the directory entry write was fsynced. Databases doing crash-safe configuration file updates must fsync both the file and the directory containing it.

Layer 2: inode → block addresses

The inode contains a list of block addresses: the physical locations on disk where the file’s 4KB blocks live. In older filesystems like ext2, this was a literal array of block numbers, with indirect pointer blocks for large files. In modern filesystems like ext4, this uses an extent tree: instead of listing individual block numbers, the inode lists contiguous ranges (extents) like “blocks 4812 through 4847.” This is far more efficient for large files — a 1GB file might be described by a handful of extents instead of 262,144 individual block numbers.

The block addresses in the inode are what get handed to the block device layer for actual I/O. The filesystem translates “give me bytes 0–8191 of this file” into “read blocks 4812 and 4813 from the block device.”

Layer 3: block device → hardware

The block device layer presents a uniform interface: a linear array of numbered 4KB blocks. Whether the underlying hardware is an HDD, SSD, NVMe, or a RAID array, the filesystem sees the same interface. The device driver translates block numbers into hardware-specific commands (SCSI for HDDs, NVMe commands for NVMe drives).

This uniformity is what lets the same Postgres binary run on an HDD server, an NVMe-backed cloud instance, and a RAM disk with no changes. The database operates entirely in terms of file offsets; the OS translates everything below.

Abstraction leak #1: the page cache lie

This one we covered in issue #002, but it’s worth stating precisely in the context of the full abstraction stack.

The OS page cache sits between the filesystem and the hardware. When your database calls write(), data lands in the page cache — kernel-managed RAM — and the OS returns success immediately. The kernel schedules the actual disk write for later. From the application’s perspective, the write completed. From the hardware’s perspective, nothing happened yet.

The animated diagram shows the full lifecycle: a write() call fills pages in the kernel’s page cache (shown in amber as “dirty” pages), the OS returns success, the database celebrates a completed write — and then a power failure hits, and every dirty page in the cache is simply gone.

Why the OS does this: Batching writes is genuinely faster. If your application writes a 100-byte record every millisecond, and the OS flushed each write to disk immediately, you’d be doing 1,000 fsync() operations per second, each costing 500µs–30ms depending on hardware. Instead, the OS accumulates dirty pages and flushes them in larger, more efficient batches. For most applications — web servers, log writers, configuration tools — this is the right trade-off. Durability is less important than throughput.

For a database that has promised “I won’t lose your data if the power goes out,” it’s the wrong trade-off. Databases solve this by calling fsync() on specific files at specific times — specifically, after writing to the WAL and before acknowledging a commit. The WAL is append-only and sequential, so fsync() on the WAL file is as fast as fsyncing gets on a given piece of hardware.

The O_DIRECT alternative: Some databases bypass the page cache entirely using the O_DIRECT file flag. Writes with O_DIRECT go directly from the database’s buffer pool to the storage hardware, skipping the OS page cache entirely. This eliminates double-buffering (the same data in both the database buffer pool and the OS page cache), gives the database precise control over what’s in memory, and removes the “dirty page accumulation and background flush” unpredictability.

The trade-off: with O_DIRECT, the database is solely responsible for durability. There’s no OS page cache copy to fall back on. The WAL and recovery mechanisms must be airtight. This is fine for a production-grade database engine, but it means there’s no safety net. MySQL’s InnoDB uses O_DIRECT by default. Postgres defaults to using the OS page cache but can be configured otherwise.

Abstraction leak #2: block alignment

The OS block size and the database page size are not always the same number.

Linux uses 4KB blocks by default. Postgres uses 8KB pages. MySQL InnoDB uses 16KB pages. When a Postgres page is written, it occupies exactly two OS blocks. When an InnoDB page is written, it occupies four. As long as both layers agree — a database page is always written as a complete set of OS blocks — this is fine.

The problem is what happens if the write is interrupted halfway. If a system crash occurs while Postgres is writing an 8KB page that spans OS blocks 4812 and 4813, the crash might leave block 4812 with new data and block 4813 with old data. The two halves of the same database page are now from different points in time. This is called a torn write, and it is a form of corruption that the database’s recovery system must detect and handle.

How ext4 and XFS handle this: Modern filesystems running in their default journaling modes provide atomicity guarantees at the OS block level — a single 4KB block write is atomic. But database pages that span multiple blocks don’t get atomicity for free. The two-block write of a Postgres 8KB page is not guaranteed to be all-or-nothing.

How Postgres handles this: InnoDB addresses torn writes with the doublewrite buffer — a technique where pages are written to a known-safe region of the disk before being written to their final location, so a torn write in the final location can always be repaired from the doublewrite buffer copy. We’ll cover this in detail in issue #042.

Postgres relies on the WAL instead. Even if a torn write leaves a page half-corrupted, the WAL contains a full record of what the page should look like after recovery. On restart, Postgres replays the WAL and reconstructs any pages damaged by torn writes. The WAL record takes priority over whatever is on disk.

The practical implication for schema design: This is why some PostgreSQL configurations write full-page images to the WAL after checkpoints. The setting full_page_writes = on (default: on) tells Postgres to write the entire 8KB page to the WAL the first time that page is modified after a checkpoint. This ensures that even if the page is torn during a crash, the full original page image is available in the WAL for recovery. It increases WAL volume by potentially a lot, but it’s the correct default for production.

Abstraction leak #3: fragmentation

The filesystem abstraction promises that a file is a sequence of bytes. It doesn’t promise that those bytes are stored in physically contiguous locations on the storage device. They usually aren’t, and the implications for database performance are severe.

What fragmentation actually means

When the OS allocates blocks for a file, it takes whatever free blocks are available — not necessarily contiguous ones. On a filesystem with lots of activity (other processes creating and deleting files, the database itself allocating and freeing space), a file being extended block by block ends up with its blocks scattered across the disk. This is fragmentation.

For SSDs, fragmentation matters much less. There’s no physical head movement — the device can access any block in ~50–150µs regardless of its physical location. The difference between accessing adjacent blocks and blocks at opposite ends of the device is in the noise.

For HDDs, fragmentation is catastrophic for performance. A sequential scan of a fragmented file is not actually sequential — the disk head must seek between each fragmented extent, paying the full 5–15ms seek penalty each time. A file with 50 fragments requires 50 seeks. What should be a 1-second sequential scan becomes a 50-second seek-dominated crawl.

What databases do about it: pre-allocation

Rather than letting the OS allocate blocks on demand as the file grows, databases pre-allocate large contiguous extents upfront. When Postgres creates a new table segment or extends an existing one, it doesn’t ask for 8KB — it asks for 1MB or more at once, claiming a contiguous block of disk space before writing any data to it. The OS, finding a large free contiguous region, allocates it as a single extent.

This means the file’s blocks are physically adjacent on disk. When Postgres later does a sequential scan of the table, it’s a true sequential read — one seek, then continuous streaming data at full disk throughput.

How Postgres pre-allocates: Postgres extends relation files in chunks defined by the BLCKSZ compile-time constant (8KB by default) but writes them in segments up to 1GB (RELSEG_SIZE). The pg_prewarm extension and the OS’s posix_fallocate() system call can pre-allocate disk space without writing data, reserving contiguous blocks immediately.

WAL files: the most important case: The WAL is perhaps the most fragmentation-sensitive file in the entire database. Every commit that requires fsync() writes to the WAL, and WAL write latency is directly in the commit latency path. Postgres pre-allocates WAL segment files (16MB by default, configurable with wal_segment_size) and keeps a small number of pre-allocated segments ready at all times. This ensures that WAL writes are always sequential into pre-allocated space — never waiting for the OS to find and allocate new blocks mid-write.

The wal_keep_size parameter controls how many WAL segments to retain for replication standbys, and max_wal_size controls when checkpoints are forced. Getting these right for your workload is one of the first things to tune on a busy write-heavy database.

How the three leaks combine: a production scenario

Consider a Postgres server under heavy write load. A bug in the application causes a large transaction to hold an open cursor for two hours. During those two hours:

Leak #1 (page cache): The page cache has been accumulating dirty pages for the table being modified. Because the long transaction holds back the oldest snapshot, VACUUM can’t clean the table. Dirty pages build up. Checkpoint is delayed. When the checkpoint eventually runs, it has to flush a large backlog of dirty pages to disk all at once — a write burst that saturates the storage device and causes a spike in commit latency for all other transactions.

Leak #2 (block alignment): During the checkpoint write burst, the server crashes mid-checkpoint. Some pages were written torn. On restart, Postgres replays the WAL from the last completed checkpoint. Because full_page_writes is on, it has the full pre-crash page images and can reconstruct everything correctly. Data loss: zero.

Leak #3 (fragmentation): The autovacuum process has been unable to reclaim space on the bloated table due to the long transaction holding back the snapshot. When the transaction finally ends, autovacuum runs aggressively, reclaiming and reusing free space in the table. Some of this reclaimed space is non-contiguous — the table is now fragmented. Future sequential scans pay a fragmentation penalty until a VACUUM FULL or CLUSTER operation is run to rebuild the table with contiguous storage.

None of these is a catastrophic failure. All three are the normal, expected operation of the database and filesystem working within their known constraints. But understanding the constraints is what lets you recognise each symptom, trace it to its source, and make the correct intervention.

The three tuning knobs these leaks explain

full_page_writes = on — Addresses leak #2. Writes full page images to WAL after checkpoints to enable recovery from torn writes. Leave this on unless you have hardware-level write atomicity guarantees (some enterprise storage controllers provide this).

checkpoint_completion_target = 0.9 — Addresses leak #1. Spreads dirty page flushing across 90% of the checkpoint interval rather than concentrating it at the end. Prevents write bursts. Set this to 0.7–0.9 on busy write workloads.

wal_segment_size — Addresses leak #3. Larger WAL segments mean fewer file allocations, less fragmentation, and more contiguous sequential writes. The default of 16MB is sensible; 64MB or 256MB is reasonable for write-heavy workloads on modern NVMe storage.

What’s actually behind pg_test_fsync

Postgres ships with a diagnostic tool, pg_test_fsync, that benchmarks fsync performance on your specific hardware. It tests different wal_sync_method options — fsync, fdatasync, open_sync, open_datasync — and reports the number of WAL writes per second achievable with each. The method that achieves the highest rate is the right choice for your hardware.

Run this before deploying a new database server. The right answer varies enormously by hardware. On some SSD configurations, fdatasync is 3–5× faster than fsync because it skips fsyncing file metadata changes. On others the difference is negligible. The only way to know is to measure.

bash

# Run as the postgres user on your actual server hardware:
pg_test_fsync -s 2

# Output example (your numbers will vary by hardware):
# Compare file sync methods using one 8kB write:
# (in wal_sync_method preference order, fastest first)
# open_datasync                 3429.5/sec    291 usec/op
# fdatasync                     3152.8/sec    317 usec/op
# fsync                         2847.3/sec    351 usec/op
# fsync_writethrough               n/a
# open_sync                     2103.6/sec    476 usec/op

The winning method becomes your wal_sync_method in postgresql.conf.

The practical takeaway

The OS file abstraction is clean and useful — and it has three specific gaps that every serious database must work around:

Gap 1 — write() doesn’t mean “on disk.” fsync() does. Databases use the WAL pattern: write a compact sequential record, fsync it, then acknowledge the commit. The data pages follow asynchronously.

Gap 2 — Multi-block writes aren’t atomic. A crash mid-write leaves torn pages. Databases address this with either a doublewrite buffer (InnoDB) or full-page WAL writes (Postgres), so recovery always has a clean copy.

Gap 3 — The OS allocates blocks for throughput, not locality. Databases pre-allocate contiguous extents upfront — especially for WAL files — so that writes stay sequential and reads stay fast.

When you see any database configuration knob related to fsync methods, WAL sizing, checkpoint spreading, or pre-allocation — it’s addressing one of these three gaps. There’s no mystery. There’s just the physics of storage and the engineering required to work within it reliably.

Last week we talked about the memory hierarchy — the stack from CPU registers down to spinning metal, and the enormous latency gaps between each layer. This week we go one level deeper into the bottom two layers of that stack: hard disk drives and solid-state drives. Not because database engineers need to know how to manufacture storage hardware, but because the physical geometry of these devices creates constraints that travel all the way up through the OS, through the database engine, and into every schema and index design decision you’ll ever make.

There’s a number that shapes nearly every database storage decision ever made: 7 to 23 milliseconds. That’s the time it takes a hard disk drive to perform a random read — the combined cost of moving a mechanical arm to the right track and waiting for the platter to rotate the right sector underneath it. During those 20 milliseconds, a 3GHz CPU executes roughly 60 million instructions. An entire sorting algorithm can run to completion. A round-trip TCP packet can cross a continent and come back. And yet, for decades, every database application had to wait this long to fetch a single row that wasn’t cached in memory.

That constraint — mechanical movement as the bottleneck for data access — is the origin story of almost every database structure you’ll encounter in this newsletter.

Inside a hard disk drive

A hard disk drive is, at its core, a precision optical instrument wearing the clothes of a storage device.

The data is stored on one or more platters — circular discs coated in magnetic material, stacked on a central spindle, spinning at 5,400 to 15,000 rotations per minute. Each platter has two surfaces, and each surface has a dedicated read/write head that floats on a cushion of air just a few nanometers above the surface. The head never touches the platter — if it does, that’s a head crash, and it destroys the data beneath it permanently.

The magnetic coating is divided into concentric circles called tracks. Each track is further divided into sectors — arc-shaped segments that are the smallest unit the drive can independently read or write. On modern drives, a sector is 4KB.

The heads are mounted on an actuator arm that pivots on a fixed point at the edge of the platters. To read a specific sector, the arm swings the head to the right track (seek), then waits for the platter’s rotation to bring the right sector under the head (rotational latency), and then reads continuously as the sector passes.

This is the physical geometry that every database storage decision must work around.

Seek time: the most expensive thing a database can do

Seek time is the time it takes for the actuator arm to swing to the correct track. This depends entirely on how far the head has to travel — from its current position to the target track. The average seek time on a modern HDD is 5 to 15 milliseconds for a random access. The minimum (adjacent tracks) is around 1ms. The maximum (full platter sweep) is around 20ms.

Rotational latency: waiting for the platter

Once the head is over the correct track, the platter still has to rotate until the right sector is underneath. This is rotational latency. At 7,200 RPM — a common server disk speed — the platter completes one full revolution in 8.3 milliseconds. Average rotational latency is half that: about 4ms. In the worst case (the sector just passed the head as it arrived), you wait almost a full revolution.

Transfer time: the fast part

Once the head is positioned correctly, data flows continuously at the drive’s full throughput — typically 100 to 200 MB/s for a modern HDD. Reading a single 8KB database page takes well under 0.1ms once the head is there. The data transfer is not the problem. The positioning is the problem.

What this means for software

The total random read cost — seek + rotational latency + transfer — is 7 to 23ms. But sequential reads pay the positioning cost only once, at the start. After that, data flows continuously. If you read 1,000 sequential 8KB pages, you pay one seek and then read at full throughput. If you read 1,000 random 8KB pages scattered across the disk, you pay the full 7–23ms positioning cost 1,000 times.

That’s the inequality every database storage engine is designed around:

A factor of 100–300x difference. Not for a different workload. For the exact same 8MB of data, in different access patterns.

This is why database engineers obsess over access patterns. This is why B-trees cluster related keys on adjacent pages. This is why write-ahead logs append sequentially before touching scattered data pages. The entire structure of every storage engine you’ll study is a response to this single inequality.

Solid-state drives — what changed, what didn’t

SSDs replaced mechanical components with NAND flash memory. No platters. No spindle. No actuator arm. No seek time. No rotational latency. The time to initiate a random read dropped from 7–23ms to 50–150 microseconds — a 100x improvement. Every database got meaningfully faster without any code changes just by replacing spinning disks with SSDs.

But “no moving parts” doesn’t mean “no constraints.” SSDs have their own geometry — different from HDDs, but equally consequential.

NAND flash structure: pages and blocks

NAND flash is organised in a hierarchy. At the bottom: individual cells, each storing 1 to 4 bits depending on the type of flash (SLC, MLC, TLC, QLC). Cells are grouped into pages, typically 4KB to 16KB — the smallest unit that can be read or written. Pages are grouped into blocks, typically 128 to 512 pages — typically 256KB to 4MB of storage per block.

Here is the constraint that matters: you can only write to a page that has been erased. And you can only erase at the block level, not the page level.

This means you cannot update a single page in place. To overwrite a 4KB page, the SSD must:

1. Read the entire block containing that page into a buffer (~256KB)

2. Erase the entire block (a relatively slow operation, 1–5ms)

3. Modify the target page in the buffer

4. Write the entire block back to flash

The SSD firmware handles this transparently — to your operating system, it looks like any other random write. But under the hood, updating 4KB of data causes the SSD to write 256KB to 512KB of data. This ratio — bytes actually written to flash divided by bytes the application intended to write — is called write amplification. On a busy database with many small random writes, write amplification factors of 10x to 30x are common.

Wear levelling: why SSDs have a write budget

Every erase cycle degrades the oxide layer in each flash cell. Consumer NAND flash cells are rated for roughly 3,000 erase cycles; enterprise multi-level cell (MLC) flash for around 100,000. Once a cell exceeds its cycle limit, it can no longer reliably hold charge — it starts flipping bits silently.

SSD firmware addresses this with wear levelling: it tracks the erase cycle count of every block and deliberately spreads writes across all blocks rather than hammering the same ones repeatedly. This is why an SSD that receives concentrated sequential writes in one region doesn’t wear out that region faster than the rest — the firmware is constantly moving data around in the background.

This background management means SSD write performance isn’t constant. Under heavy sustained write load, the firmware is simultaneously handling new writes, garbage-collecting blocks that are partially valid, and doing wear levelling relocations. This is what causes SSD write performance to degrade under sustained pressure — write cliff — a phenomenon that matters a lot for database write workloads.

NVMe vs SATA: the interface matters too

Early SSDs plugged into the SATA interface — the same connector used for spinning disks. SATA’s maximum throughput is ~550MB/s sequential, and its command queue depth is limited to 32 commands. An NVMe SSD connected directly to the PCIe bus has no such limitations: current NVMe drives deliver 3–7GB/s sequential and support command queue depths of 65,535. For a database receiving thousands of concurrent I/O requests, that queue depth difference matters enormously.

The practical takeaway: “SSD” is not one thing. A SATA SSD in a database server is meaningfully different from an NVMe SSD, which is meaningfully different from enterprise-grade NVMe with power-loss protection capacitors. The device class determines the performance envelope your database operates within.

The OS I/O stack — what happens between your code and the hardware

When your database calls write() to save a WAL record, that data does not immediately go to the SSD or HDD. It goes through a stack of OS layers, each adding behaviour and each potentially delaying or buffering the actual hardware write.

Layer 1: Virtual File System (VFS)

The VFS is a kernel abstraction that presents a unified file system interface regardless of the underlying file system type. Whether Postgres is storing files on ext4, XFS, or a network file system, the database makes the same read(), write(), and fsync() calls. The VFS routes them to the appropriate file system driver.

Layer 2: OS Page Cache — the silent middleman

This is the most important layer for database engineers to understand. The OS page cache is a region of kernel-managed RAM that transparently caches file data. When your application calls write(), the data lands in the page cache first. The kernel acknowledges the write as complete immediately, then schedules the actual disk write for later — in batches, at its own convenience, optimised for throughput.

From your application’s perspective: the write succeeded. From the hardware’s perspective: nothing has happened yet.

This is not a bug. For most applications, this behaviour is desirable — it makes writes fast and amortises disk I/O. But for a database that has promised durability, it is a serious problem. If the machine loses power after a write() but before the kernel flushes the page cache to disk, the data is gone. The write returned success. The OS confirmed it. And the data still doesn’t exist on any non-volatile medium.

Layer 3: fsync() — the bypass

fsync() is the system call that forces the OS to flush all buffered writes for a file descriptor to the storage device and wait until the device confirms. It bypasses the “defer to later” behaviour of the page cache. When Postgres calls fsync() after writing a WAL record, it blocks until that record is confirmed on physical storage — not just in the OS page cache.

This is expensive. On an HDD: 10–30ms. On an SSD: 500µs to 5ms. On a busy system processing many commits per second, fsync() cost is often the single largest contributor to write latency.

This is why the WAL exists. Instead of fsyncing every modified table page on every commit — which could be gigabytes of data — Postgres writes a compact log record (typically a few hundred bytes), fsyncs that, and returns the commit. The actual table pages are written to disk asynchronously later. The WAL is the minimum durable record needed to reconstruct whatever the async write hasn’t done yet.

Layer 4: O_DIRECT — going around the page cache entirely

Some databases bypass the OS page cache entirely using the O_DIRECT flag when opening files. With O_DIRECT, all reads and writes go directly to the storage hardware, bypassing the kernel’s caching. The database manages its own in-memory page cache (Postgres’s shared_buffers), which it can tune precisely.

O_DIRECT avoids double-buffering (the same data being cached both in the database’s buffer pool and the OS page cache), and it gives the database full control over what’s in memory. MySQL’s InnoDB uses O_DIRECT by default. Postgres does not by default, though it has an option for it.

The trade-off: with O_DIRECT, if the database crashes between a write() and an fsync(), the data is gone — there’s no OS page cache copy to fall back on. The database must handle this entirely with its own WAL and recovery mechanisms. This is fine for a well-designed database engine, but it means the durability guarantees live entirely in database code, not partially in the OS.

Layer 5: I/O scheduler

The block layer sits between the file system and the device driver. It maintains a queue of pending I/O requests and reorders them for throughput — for example, merging adjacent reads into a single larger request, or reordering requests to minimise seek distance on an HDD.

For SSDs, aggressive scheduling is counterproductive — there’s no seek cost to optimise for, and the reordering just adds latency. Modern Linux kernels use mq-deadline or none for NVMe devices, and configure the scheduler accordingly.

For HDDs on database workloads, the scheduler can meaningfully improve throughput by converting random access patterns into more sequential ones. But this is increasingly irrelevant as HDDs disappear from primary storage use cases.

What this means for database configuration

Understanding the I/O stack makes several Postgres configuration knobs make sense immediately:

shared_buffers — Postgres’s in-memory page cache. The larger this is, the more frequently reads are served from RAM rather than triggering OS page cache or disk reads. The general recommendation is 25% of total RAM, but the right value is “as much as you can give it while leaving room for OS and other processes.”

wal_sync_method — controls how Postgres calls fsync. Options include fsync (the default, uses fsync()), fdatasync (slightly more efficient, skips fsyncing file metadata), open_sync, and open_datasync (use O_SYNC/O_DSYNC flags). The right choice depends on your OS and storage hardware.

synchronous_commit — when set to off, Postgres returns commit confirmation to the application before fsyncing the WAL. This dramatically increases write throughput at the cost of potentially losing the last few transactions in a crash. Acceptable for some workloads (logging, analytics). Never acceptable for financial transactions.

checkpoint_completion_target — controls how aggressively Postgres writes dirty (modified) pages from shared_buffers to disk between checkpoints. A lower value means more I/O concentrated at checkpoint time; a higher value spreads it out. The goal is keeping I/O smooth so the storage device doesn’t get hammered with write bursts.

The practical mental model

Two facts to carry forward from this issue into every future discussion of database storage:

1. Random I/O is fundamentally different from sequential I/O. On HDDs, random reads cost 100–300x more than sequential reads for the same data volume. On SSDs, the gap is ~10x. This inequality never goes away — it just gets smaller. Every database storage structure is an attempt to turn random access patterns into sequential ones.

2. The OS page cache lies. A successful write() does not mean data is on disk. A successful fsync() does. The distinction is the foundation of database durability. When your database guarantees that a committed transaction will survive a power failure, it means there was a successful fsync() somewhere in the commit path.

When you see a WAL, a write-ahead log, a redo log, or a journal in any storage system, you are looking at the same idea: a compact sequential append that gets fsynced before the commit is acknowledged, so that the database can recover the full state from just the log if everything else is lost.

Bytes & B-trees | Pre-launch #3 of 3 | Free | Bookmark recommended

Most online courses and newsletters launch with a vague promise: “we’ll cover everything you need to know.” This is not that.

What follows is the exact plan for every issue of Bytes & B-trees — the 10 content tracks, 172 individual issues mapped by quarter, the three Build It Yourself implementation series, the 12 annotated research papers, and the company case studies. Read this before you subscribe. If what you see here doesn’t match what you want to learn, this isn’t the newsletter for you. That’s fine — we’d rather you know now.

The 10 content tracks

Every issue belongs to one or two of ten tracks. These describe the primary lens through which the concept is examined.

Total: 172 issues · 3 years · ~6,000 words each · 52 issues/year

Year 1 — Issues #001 to #056

Foundations: from storage physics to the full database taxonomy

Q1 (Issues #001–014): Why databases exist + Build It Yourself #1

#001 Data, storage & memory — first principles
#002 Disk geometry & the I/O stack — seek time, rotational latency, and what SSDs changed
#003 Files, block devices & the OS abstraction — where databases actually write
#004 Why flat files fail — the six problems that forced databases into existence
#005 IMS, CODASYL & the hierarchical model — what came before relational
#006 CODD’s relational model — the idea that changed everything
#007 Atomicity at the byte level — what “all or nothing” actually means in hardware
#008 Durability — what fsync guarantees, and what it doesn’t
#009 Isolation levels visualised — dirty reads, phantom reads, serialisable anomalies
#010 SQL and relational algebra — the formal foundation behind the syntax
#011 NULLs, three-valued logic & the SQL traps — why NULL is not zero or empty
#012 🔨 Build It Yourself #1 — Part 1: a key-value store: crash-safe writes & file pointers
#013 🔨 Build It Yourself #1 — Part 2: adding a hash index
#014 🔨 Build It Yourself #1 — Part 3: adding a write-ahead log & crash recovery

By the end of Q1: You understand why every ACID property exists at the hardware level, and you have a working key-value store that demonstrates all of it.

Q2 (Issues #015–029): Storage engines & indexing

#015 Pages, heaps & row storage anatomy — how Postgres lays out rows on disk
#016 TOAST & variable-length columns — how Postgres stores text and JSON that doesn’t fit
#017 Row vs columnar storage — when each wins and why the gap is enormous
#018 B-trees — the balanced tree on disk: why this structure, why this shape
#019 B+ trees — the leaf-linked difference that makes range scans fast
#020 Index-only scans, covering indexes & fill factor — the performance levers
#021 Index bloat, splits & the cost of maintenance — what indexes cost to keep
#022 ✨ GIN — generalised inverted indexes: full-text search & JSONB under the hood
#023 ✨ GiST & BRIN — the specialist indexes most engineers never encounter
#024 Hash indexes & bloom filters — when equality beats range lookup
#025 Bitmaps, skip lists & low-cardinality index structures
#026 LSM trees — why writes are cheap: the write path from memtable to SSTables
#027 SSTables, compaction & write amplification — the cost the write path defers
#028 LSM vs B-tree — the fundamental trade-off that shapes engine selection
#029 RocksDB internals — the LSM tree in production at Meta, Cassandra, and TiKV

By the end of Q2: You understand every major index structure — why it exists, how it works physically, and when to choose it.

Q3 (Issues #030–042): Query execution & concurrency

#030 Parse → plan → execute — the full lifecycle of a SQL query
#031 Cost-based optimisers & table statistics — how Postgres decides what to do
#032 Reading EXPLAIN like an expert — every node type, every cost field
#033 Nested loop, hash join & merge join — when each algorithm fires and why
#034 Join order & why it matters exponentially — the combinatorial explosion
#035 Parallel query execution — when Postgres uses multiple workers, when it doesn’t
#036 Two-phase locking in depth — how row-level locking actually works
#037 MVCC — why it’s the hardest thing in databases
#038 Vacuum, dead tuples & table bloat — the cost of MVCC deferred
#039 Optimistic concurrency & serialisable snapshot isolation in practice
#040 📄 Annotated paper: The ARIES algorithm — the paper that defined crash recovery
#041 WAL, redo logs & the crash recovery sequence — ARIES in Postgres
#042 InnoDB’s doublewrite buffer — why it exists and what it costs

By the end of Q3: You understand how a query goes from text to result, how concurrent transactions are isolated, and how databases recover from crashes.

Q4 (Issues #043–056): Database taxonomy + Build It Yourself #2

#043 NoSQL — what it genuinely gave up and what it genuinely gained
#044 Document stores & the BSON choice — why MongoDB stores data the way it does
#045 Key-value stores — simplicity as a deliberate design choice
#046 Column-family stores & the wide-row model — Cassandra’s data model explained
#047 Time-series databases & hypertable partitioning — TimescaleDB’s approach
#048 ClickHouse’s MergeTree — columnar storage for analytical workloads
#049 Graph databases — when property graphs beat recursive SQL
#050 The database selection framework — a decision tool for any workload
#051 OLTP vs OLAP — the architectural divide and why it matters
#052 Polyglot persistence — when one database genuinely isn’t enough
#053 Year 1 synthesis — your complete mental model, connected
#054 🔨 Build It Yourself #2 — Part 1: parsing SQL into an AST
#055 🔨 Build It Yourself #2 — Part 2: building a logical plan
#056 🔨 Build It Yourself #2 — Part 3: executing filter, projection & hash join

By the end of Year 1: You understand the full database landscape from first principles. You have built a key-value store and a working query engine. You can explain any database’s core behaviour from the bottom of the storage stack.

Year 2 — Issues #057 to #113

Internals & Scale: real engines, distributed systems, and performance mathematics

Q1 (Issues #057–069): Distributed foundations

#057 The 8 fallacies of distributed computing — each one a class of bugs
#058 Network partitions & partial failures — why distributed systems are fundamentally harder
#059 CAP theorem — what it actually claims, what it doesn’t, and how it’s misused
#060 Primary-replica replication anatomy — the standard topology dissected
#061 Statement vs row vs logical replication — the trade-offs in each format
#062 Replication lag, read-your-writes & monotonic reads — consistency guarantees
#063 Multi-master & leaderless topologies — when and why
#064 FLP impossibility — why consensus is provably hard in asynchronous systems
#065 📄 Annotated paper: Paxos Made Simple — Lamport’s algorithm, actually explained
#066 📄 Annotated paper: Raft — leader election with pictures
#067 Raft — log replication, safety guarantees, and membership changes
#068 etcd, CockroachDB & TiKV’s Raft implementations — theory in production code
#069 Why wall-clock time is dangerous in distributed systems

Q2 (Issues #070–084): Real engine dissections

#070 Lamport timestamps & vector clocks — establishing order without a shared clock
#071 📄 Annotated paper: Google TrueTime — atomic clocks, GPS, and Spanner’s clock API
#072 Hybrid logical clocks in CockroachDB — the practical compromise
#073 Postgres buffer manager & shared_buffers — the most important tuning knob
#074 Postgres query planner in depth — join reordering, partition pruning, statistics
#075 Postgres MVCC snapshots & visibility rules — which row version does a query see?
#076 Postgres extensions architecture — how PostGIS and pgvector plug into the engine
#077 MySQL InnoDB — clustered indexes, gap locks & next-key locks
#078 InnoDB binlog and the replication story
#079 ✨ Apache Arrow — the universal columnar memory format
#080 ✨ DuckDB — the in-process analytics revolution and why it’s architecturally novel
#081 Cassandra — consistent hashing ring, vnodes, quorum reads & anti-entropy
#082 📄 Annotated paper: Dynamo — Amazon’s highly available key-value store, paragraph by paragraph
#083 Redis — eviction policies, RDB vs AOF, cluster slot-based sharding
#084 Redis Streams as a persistent event log

Q3 (Issues #085–100): Sharding, cloud databases & OLAP at scale

#085 Sharding strategies — range, hash & directory: what each costs
#086 Hot keys, vnodes & resharding without downtime
#087 Vitess — MySQL sharding at YouTube scale
#088 CockroachDB — distributed SQL from the ground up
#089 📄 Annotated papers: Spanner & F1 — the two papers that changed distributed databases
#090 TiDB & YugabyteDB — the NewSQL generation
#091 ✨ Benchmark methodology — how to run honest performance comparisons
#092 ✨ Benchmarking SurrealDB, EdgeDB & Turso — reproducible, methodology-first
#093 📄 Annotated paper: Amazon Aurora — why “the log is the database”
#094 Neon — copy-on-write branching for Postgres
#095 PlanetScale — online schema changes at scale
#096 Serverless databases — economics, cold starts & architectural implications
#097 Snowflake — compute/storage separation dissected
#098 BigQuery’s Dremel engine — nested columnar storage
#099 ClickHouse vectorised execution internals
#100 The modern data stack — where the seams are

Q4 (Issues #101–113): Performance engineering + Build It Yourself #3

#101 Query optimisation — a systematic methodology beyond “add an index”
#102 Index design methodology — choosing the right index for the right query pattern
#103 Statistics staleness & its cascading effect on query plans
#104 Materialised views, lateral joins & window functions
#105 Schema design for write performance — normalisation and its limits
#106 Schema design for read performance — denormalisation and its costs
#107 PgBouncer modes, ProxySQL & the connection pool design space
#108 ✨ Queuing theory — Little’s Law, M/M/1 queues & database capacity planning mathematics
#109 The Linux page cache & how databases fight it
#110 io_uring — what it means for database I/O in practice
#111 NVMe topology, NUMA effects & huge pages
#112 OS & kernel tuning for database workloads — the full checklist
#113 🔨 Build It Yourself #3: a minimal replication log — primary, replica & failover

By the end of Year 2: You can dissect real database source code, design distributed architectures, size a connection pool using queuing mathematics, and explain every major cloud database’s architectural trade-offs.

Year 3 — Issues #114 to #172

Frontier, ultra-scale & synthesis: what’s next and how it all connects

Q1 (Issues #114–128): The emerging landscape

#114 Approximate nearest neighbour — the ANN problem and why exact search doesn’t scale
#115 HNSW — how hierarchical navigable small worlds work
#116 IVF, FAISS & product quantisation
#117 pgvector vs dedicated vector databases — an honest comparison
#118 ✨ RAG Part 1 — chunking strategies and their effect on recall quality
#119 ✨ RAG Part 2 — embedding pipeline design, staleness & update latency
#120 ✨ RAG Part 3 — hybrid search: BM25 + vector + re-ranking models
#121 ✨ RAG Part 4 — the full data architecture behind a production RAG system
#122 NL-to-SQL — where it works, where it fails, and why
#123 LLM-assisted query optimisation — research vs production reality
#124 Edge databases — SQLite’s architecture, Turso & libSQL
#125 CRDTs — offline-first sync without a coordination server
#126 ElectricSQL & local-first architecture
#127 Materialize & differential dataflow
#128 RisingWave — streaming SQL in production

Q2 (Issues #129–143): Research frontiers + company case studies

#129 Exactly-once semantics across stream and store
#130 📄 Annotated paper: The Case for Learned Index Structures — Kraska et al
#131 Learned cardinality estimation — replacing histograms with models
#132 Bao — learning to optimise queries from execution feedback
#133 🏢 Company case: Netflix — Cassandra, EVCache & the viewing history data model
#134 🏢 Company case: Discord — MongoDB → Cassandra → ScyllaDB, the full story
#135 🏢 Company case: Stripe — sharding Postgres when one instance isn’t enough
#136 🏢 Company case: Uber — Schemaless/Docstore for trip data at scale
#137 🏢 Company case: Cloudflare D1 — SQLite at the edge, globally
#138 🏢 Company case: Figma — scaling Postgres from one machine to many
#139 📄 Annotated paper: Napa — powering scalable data warehousing at Google
#140 CXL memory pooling & what it means for buffer management
#141 Persistent memory — what Optane taught the database industry
#142 TLA+ for distributed database protocol verification
#143 Jepsen — how Kyle Kingsbury breaks databases

Q3–Q4 (Issues #144–172): Ultra-scale design & 3-year synthesis

#144 Access pattern analysis for billion-row systems
#145 Partition key design — avoiding the hot key disaster
#146 Index topology for tables at extreme scale
#147 Geo-replication — designing for global latency
#148 Multi-region consistency patterns in production
#149 CRDTs in production — the failure modes nobody warns you about
#150 Event sourcing — the immutable log as a database
#151 Event sourcing pitfalls nobody warns you about
#152 CQRS — when and why to separate reads from writes
#153 Zero-downtime schema migrations at scale
#154 Database chaos engineering — breaking things safely
#155 Row-level security in depth
#156 GDPR right-to-erasure in an append-only system
#157 Audit logging without killing performance
#158 The people: Edgar F. CODD — the relational model and its reception
#159 The people: Michael Stonebraker — Ingres, Postgres, and 40 years of database design
#160 The people: Jim Gray — transactions, fault tolerance, and the 1998 Turing Award
#161 The people: Leslie Lamport — clocks, consensus, and distributed systems theory
#162 50 years of databases — the full economic and technical arc
#163 Where storage hardware is going — NVMe, CXL, photonic interconnects
#164 Post-SQL query models — what comes after the relational interface
#165 The AI query interface endgame — where NL-to-SQL leads
#166 The 3-year concept map — Part 1: Years 1 & 2 connected
#167 The 3-year concept map — Part 2: Year 3 and the full picture
#168 The complete trade-off matrix — every design decision, compiled
#169 The database selection framework — final version
#170 The performance engineering playbook — the full checklist
#171 The distributed systems decision guide
#172 Issue #172 — the final synthesis: 172 concepts, one mental model

By the end of Year 3: You can design systems for billions of records, understand and critique database research papers, explain the full 50-year history of database engineering, and synthesise every concept from all three years into a coherent, connected mental model.

The 12 annotated research papers

Each paper gets a full dedicated issue: we read it together, paragraph by paragraph, translate the academic language into plain engineering, explain the context, and trace what the industry actually adopted.

How to use this guide

Bookmark this page. Each time a new issue arrives, return here to see how it connects to what came before and what comes next. The issue references in each week’s diagram panels all point to entries in this map.

The curriculum is designed so that every concept you encounter in Year 3 was seeded in Year 1 — this is the document that shows you where. The B-tree you learn in issue #019 reappears in issue #073 (Postgres buffer manager), and again in issue #146 (index topology at scale). Each time with more context, more depth, and more connection to the surrounding concepts.

One last thing

Databases are one of those rare engineering topics where genuine understanding compounds dramatically. The engineer who knows why a B-tree organises data the way it does doesn’t just understand B-trees — they immediately have better intuition about LSM trees, write amplification, columnar storage, and why certain Cassandra data models are catastrophically slow while others are fast.

Understanding is recursive in this domain. The more you have, the faster new concepts click into place.

That compounding is what the three-year structure is designed to produce. Not 172 isolated facts. One interconnected mental model, built one piece at a time, week by week, until databases stop being black boxes and start being something you genuinely understand.

Issue #001 ships this Sunday. See you then.

Subscribe — free tier always available. Issue #001: “Data, storage & memory — first principles.” Free for all subscribers.

© Bytes & B-trees · Substack · Weekly · bytesbtrees.substack.com
Publish order: ① Promotional → ② Course Introduction → ③ This curriculum guide → Issue #001 (Sunday)

Bytes & B-trees | Pre-launch #2 of 3 | Free to read

There’s a particular kind of database problem that trips up even experienced engineers. Not the easy ones — slow queries, missing indexes, bad schema decisions. The harder kind: a transaction that committed but whose data was partially lost after a failover. A replication lag that spiked mysteriously during a schema migration. A Cassandra cluster that returned stale reads even though you configured quorum. A Postgres table that autovacuum couldn’t keep up with despite running constantly.

These problems share a pattern. They’re invisible to engineers who learned databases from the interface down — from SQL syntax, to ORM usage, to “add an index when slow.” They’re immediately understandable to engineers who know what’s happening inside the engine. The difference between the two isn’t years of experience. It’s a specific kind of knowledge: the internals.

That knowledge is what this newsletter builds, systematically, over three years.

The learning philosophy

Every concept in this newsletter follows the same sequence. We start with the problem — the specific constraint or failure mode that motivated someone to invent the mechanism in the first place. We ask: what would happen without it? What breaks? What becomes impossible?

Only after that question is fully answered do we introduce the solution. This order matters. The mechanism only makes sense in light of the problem it solves. Presented the other way — mechanism first, motivation second — it’s just memorisation.

Then we go into the implementation: how the mechanism actually works, with real code, real configurations, and real data structures. Not pseudocode. Not simplifications that obscure how it actually behaves. The real thing — or close enough that the difference doesn’t matter until you’re reading source code.

Then we anchor it in a real system — Postgres, RocksDB, Cassandra, DynamoDB, whichever shows the concept most clearly in production code.

Then we examine the trade-offs: what did this design gain, and what did it give up? What workloads does it help? What workloads does it hurt? When would you choose differently?

The goal is not that you can recite how MVCC works. The goal is that when your Postgres tables start bloating, you know immediately that it’s a vacuum problem, and you know exactly why — because you understand what MVCC is doing under the hood and what vacuum’s job is. That’s the difference between knowledge and understanding.

The three-year arc

Year 1 — Foundations (Issues #001–056)

What you’ll be able to do by the end:

By the end of Year 1, you understand why databases exist, how data is physically stored and retrieved, what ACID actually guarantees at the hardware level, how indexes are structured internally, how queries are parsed and executed, and how crash recovery works. You’ll have built a working key-value store and a minimal query engine from scratch. You’ll be able to look at any database’s core behaviour and understand the first-principles reason it works the way it does.

Topics covered:

• Storage & I/O physics — bits, bytes, pages, disk geometry

• B-trees & LSM trees — the two index structures that run the world

• ACID from first principles — atomicity, durability, isolation at the byte level

• MVCC & WAL internals — how Postgres actually manages concurrent transactions

• SQL execution engine — parse, plan, execute, optimise

• Database taxonomy — relational, document, key-value, columnar, graph, time-series

• Build It Yourself #1: a key-value store (crash-safe writes → hash index → WAL)

• Build It Yourself #2: a minimal SQL query engine (parser → logical plan → executor)

Year 2 — Internals & Scale (Issues #057–113)

What you’ll be able to do by the end:

You can dissect real database source code, design distributed architectures, choose between Raft and Paxos for a given use case, configure connection pools using queuing mathematics, and explain exactly why CockroachDB and Spanner make the trade-offs they do.

Topics covered:

• Postgres buffer manager, query planner, and MVCC snapshot mechanism — at the code level

• Distributed consensus — Raft, Paxos, FLP impossibility, the real implementations

• Replication topologies — primary-replica, multi-master, leaderless, and when each breaks

• Sharding strategies — range, hash, directory, hot keys, and resharding without downtime

• NewSQL — CockroachDB, Spanner, TiDB — the ACID + horizontal scale problem

• DuckDB & Apache Arrow — the unbundled analytics revolution

• Cloud databases — Aurora, Neon, PlanetScale, Snowflake, ClickHouse

• Queuing theory — Little’s Law, M/M/1 queues, and database capacity planning mathematics

• Build It Yourself #3: a minimal replication log with primary and replica

Year 3 — Frontier & Ultra-scale (Issues #114–172)

What you’ll be able to do by the end:

You can design systems for billions of records, explain how vector search works at the index level, architect a production RAG pipeline correctly, read and critique database research papers independently, and synthesise three years of concepts into a coherent mental model you’ll use for the rest of your career.

Topics covered:

• Vector search — HNSW, IVF, FAISS, approximate nearest neighbour

• RAG as a database architecture problem — chunking, embeddings, hybrid search, re-ranking

• AI-native query interfaces — NL-to-SQL, LLM query optimisation

• Edge databases & local-first — SQLite, Turso, CRDTs, ElectricSQL

• Streaming databases — Materialize, RisingWave, exactly-once semantics

• Research frontiers — learned indexes, learned cardinality estimation, Bao

• Company case studies — Netflix, Discord, Stripe, Uber, Cloudflare, Figma

• Ultra-scale system design — 10B-record architectures, multi-region consistency

• Event sourcing, CQRS, zero-downtime migrations, chaos engineering

• The 3-year synthesis — every concept map and decision framework, compiled

How each week works

Every Sunday morning, a new issue arrives. The main piece — free for everyone — is 4,000–5,000 words covering the week’s concept from problem to mechanism to trade-off. The paid section includes: hands-on experiments you can run on your own machine, the Query Lab puzzle and its solution from the previous week, and (when applicable) the Build It Yourself implementation code.

The weekly production cycle

Day Activity Monday Research — primary sources, paper re-reads, code exploration, angle confirmed Tuesday Outline — section structure, diagrams listed, code examples scoped Wednesday–Thursday Write — full draft written, all code verified and run, diagrams built Friday Edit — technical accuracy check, clarity pass, Query Lab puzzle set Saturday Format — Substack layout, social thread drafted, GitHub repo pushed Sunday Ship — issue at 8am, thread at 9am, next issue outlined

The three quality tests every issue passes

Before any issue ships, it passes three internal tests:

1. The code test — every code sample must run without modification on a standard installation of the relevant system. No “you might need to adjust this.” It runs, or it doesn’t ship.

2. The newcomer test — could someone encounter this concept for the very first time in this issue and come away with a complete, correct mental model? Not a simplified one — a correct one.

3. The expert test — does an engineer who already knows this topic learn at least one non-obvious thing? If the answer to any of these three is no, the issue gets rewritten.

What the Query Lab is

Every issue ends with a Query Lab puzzle — an extreme SQL, Cypher, or reasoning challenge that forces you to apply the week’s concept directly. Not “write a SELECT statement.” More like: given this table size, page size, and row count, calculate exactly how many 8KB pages Postgres read before and after an index was added — and explain the I/O difference in terms of what the storage engine actually did.

Solutions — with multiple valid approaches and annotations on each — are published in the following issue’s paid section.

What to do before issue #001

Read the curriculum guide — Pre-launch #3, the next article in this series. It maps every one of the 172 issues with enough detail that you’ll know exactly what you’re signing up for before you subscribe. Then subscribe — free is fine to start.

If issue #001 isn’t what we described, unsubscribe immediately. We’d rather have readers who are genuinely here than subscribers who are being polite.

Issue #001 ships this Sunday.
“Data, storage & memory — first principles.” Free for all subscribers. Paid section includes four hands-on experiments and the first Query Lab puzzle.

Next: Pre-launch #3 — the complete curriculum guide. Every issue mapped, every track explained, every connection drawn. Long by design. Bookmark it.

© Bytes & B-trees · Substack · Weekly · bytesbtrees.substack.com

Thanks for reading Bytes & B-Tree's! Subscribe for free to receive new posts and support my work.