I break down real-world database outages, PostgreSQL C-engine internals, distributed databases and production triage runbooks.

• Free subscribers get our architecture deep dives and database engineering challenges.

• Premium members 5$/month unlock full high-severity RCAs, deep DBA diagnostic toolkits, optimization cheatsheets etc.

• New Platform Launched

If you find this breakdown valuable, consider subscribing or sharing it with your team:

Subscribe now

Share The Sev-1 Database

Table of Contents

• 1. Scenario

• 2. Challenge

• 3. Engine Mechanics: Shell Tables vs. Physical Shards

• 4. Inside the C Engine

• 5. Why Fail to Prevent the OOM

• 6. 10-Second Diagnostic SQL Checklist

• 7. 4 Hidden Architectural Failures of Schema-Based Sharding

• 8. Architectural Decision Framework: Row-Based vs. Schema-Based

• 9. Upstream Evolution & Operational Guardrails

• 10. Level Up on The Sev-1 Database

1. Incident Scenario

You run a multi-tenant SaaS application on a distributed PostgreSQL cluster using Citus with schema-based sharding (each customer tenant has their own dedicated schema, holding ~20 tables, custom types, sequences, and distributed functions).

Your business is growing fast: you now have 2,500 active tenants, resulting in over 50,000 distributed relations and dependent objects across the coordinator.

To handle increased traffic, you provision a brand-new worker node (worker_4) and execute the standard activation command on the coordinator:

SQL

SELECT citus_add_node('worker_4', 5432);
-- Or the metadata activation UDF:
-- SELECT start_metadata_sync_to_node('worker_4', 5432);

What happens next:

1. Coordinator starts churning CPU for several minutes.

2. Suddenly, the command crashes with:

ERROR: out of memory
DETAIL: Failed on request of size 8192 in memory context "metadata_sync_context".

3. A few minutes later, distributed queries routed to worker_4 start throwing:
   Plain text

ERROR: relation "tenant_1402.orders" does not exist

Your colleague suggests: “Let’s increase work_mem to 1GB, bump maintenance_work_mem”

2. Challenge

Before scrolling down to the breakdown, ask this to yourself:

1. Why did worker_4 throw relation does not exist even though shard placements were registered in the coordinator catalog?

2. Why does increasing work_mem completely fail to prevent this OOM?

3. What is start_metadata_sync_to_node() actually doing under the hood in C?

4. What are the downstream architectural traps of schema-based sharding as your tenant count continues to scale?

3. Engine Mechanics: Shell Tables vs. Physical Shards

If you dont know what is shell tables vs physical shards

Shell Tables vs. Physical Shards

• Shell Table: A metadata-only representation of a distributed table on a worker node. It contains the table definition (schema, columns, indexes, etc.) but does not store actual data. Citus uses shell tables so DDL operations and metadata stay consistent across the cluster.

• Physical Shard: Actual data-containing partition of a distributed table stored on a worker node. Queries on distributed tables ultimately read from and write to these physical shards.
  
  In one line:
  Shell Table = table structure and metadata.
  Physical Shard = actual distributed data storage.
  

In schema-based sharding, a distributed table has two distinct identities:

[ Coordinator Node ]
  ├── Logical Shell:  tenant_a.orders (in pg_class & pg_dist_partition)
  └── Routing Map:    Placement points to Worker 1

[ Worker 1 (Existing) ]
  ├── Logical Shell:  tenant_a.orders (Used for parsing, planning, locking)
  └── Physical Shard: tenant_a.orders_102034 (Actual data on disk)

[ Worker 4 (New) ]
  ├── Logical Shell:  MISSING (Sync aborted mid-flight!)
  └── Physical Shard: (Not yet placed)

• Physical Shard Table (orders_102034): Stores the actual tuples for that tenant. It only exists on the specific worker hosting that shard placement.

• Logical Shell Table (tenant_a.orders): Contains column types, constraints, indexes, and schema definitions. It holds zero rows on workers that don’t own the shard, but it is mandatory for PostgreSQL to parse SQL, verify foreign keys, and execute distributed functions.

If a worker is missing the logical shell table, running a query or distributed function against it fails immediately with relation does not exist even if placement rows exist in the coordinator catalog.

4. Inside the C Engine: metadata_sync.c

When you add a metadata-capable worker, the coordinator executes start_metadata_sync_to_node() (src/backend/distributed/metadata/metadata_sync.c).

Here is the call graph:

start_metadata_sync_to_node('worker_4', 5432)
  │
  ├── 1. Creates dedicated MetadataSyncContext
  ├── 2. Calls ActivateNodeList()
  └── 3. Runs SyncNodeMetadataSnapshotToNode()
           ├── Drops old metadata snapshot on the worker
           ├── Re-creates distributed catalogs (pg_dist_*)
           └── Calls SyncCitusTableMetadata()
                 └── Dispatches thousands of CREATE SCHEMA, 
                     CREATE TABLE (shell), CREATE SEQUENCE, 
                     and CREATE FUNCTION statements over libpq

5. Why Fail to Prevent the OOM

PostgreSQL memory is strictly compartmentalized:

• Executor Memory (work_mem): Used for sorting, hash joins, and aggregates. When exceeded, the engine spills tuples to temporary files on disk.

• Internal Memory (MemoryContext / palloc): Used for query trees, cache-invalidation messages, and metadata DDL command buffers. MemoryContext allocations live in backend heap RAM and can NEVER spill to disk.

When synchronizing 50,000 objects in a single long transaction, the coordinator builds a massive dependency graph in memory, accumulating DDL command strings and cache-invalidation queues inside MetadataSyncContext.

Eventually, a memory allocation exceeds the process ceiling, and PostgreSQL raises out of memory. The size 8192 in the log detail is simply the size of the final allocation request that failed not the total memory consumed.

💡 Enjoying this breakdown? Share it with a teammate or database engineer who manages PostgreSQL at scale.

Share The Sev-1 Database

6. 10-Second Diagnostic SQL Checklist

When managing or scaling a Citus cluster, always verify metadata synchronization before routing traffic:

-- 1. Check metadata sync status across all workers
SELECT nodeid, nodename, nodeport, hasmetadata, metadatasynced, shouldhaveshards
FROM   pg_dist_node;

What to look for:

 nodename | hasmetadata | metadatasynced 
----------+-------------+----------------
 worker_1 | t           | t
 worker_2 | t           | t
 worker_3 | t           | t
 worker_4 | t           | f   <--- ⚠️ SYNC INCOMPLETE OR FAILED

SQL

-- 2. Verify worker-side catalog state
SELECT nodename, success, result
FROM   run_command_on_workers('SELECT count(*) FROM pg_dist_partition');

(If result = 0 with success = true, the worker is reachable, but its distributed metadata is empty).

7. 4 Hidden Architectural Failures of Schema-Based Sharding

Metadata synchronization OOM is only the first failure mode. When you scale PostgreSQL past thousands of schemas, four subtle architectural bottlenecks emerge:

1. Per-Connection relcache Memory Bloat:

Each backend process caches metadata for touched tables in its private relcache. With 50,000 tables, if connections touch many schemas, backend memory balloons to 200MB–500MB per connection just storing catalog cache. 200 connections can silently consume 60GB of RAM on catalog cache alone.

2. DDL Lock Storms:

Adding a single column in schema-based sharding requires executing 2,500 separate ALTER TABLE statements, acquiring 2,500 ACCESS EXCLUSIVE locks. If one tenant has a running query, the entire migration blocks and stalls incoming application reads.

3. Autovacuum Scheduler Starvation:

Autovacuum workers spend immense time scanning 50,000 catalog entries, causing vacuum lag on the active tables that actually need cleanup.

4. Cross-Tenant Analytics Wall:

Cross-customer reporting requires dynamically unioning 2,500 distinct tables (SELECT * FROM t1.orders UNION ALL SELECT * FROM t2.orders...), blowing past query planning memory limits.

8. Architectural Decision Framework: Row-Based vs. Schema-Based

Before choosing schema-based sharding, use this decision framework:

Core Rule of Thumb:

• If tenants share the same schema structure and you plan to scale beyond 1,000 tenants: Use Row-Based Sharding (tenant_id distribution column).

• If you must use Schema-Based Sharding (e.g. strict regulatory schema isolation): Set a hard limit of ~1,000 tenants per cluster, then shard across separate database clusters.

9. Upstream Evolution & Operational Guardrails

1. Upstream Stabilization (PR #6728 / Citus 11.3+):

Citus team introduced major memory stabilization by splitting metadata synchronization into nontransactional phases and aggressively resetting child memory contexts during batch execution. Ensure your cluster runs on modern builds.

2. Schema & Object Hygiene:

In multi-tenant architectures, obsolete tenant schemas, unused views, and abandoned index definitions aren’t just wasted disk they multiply the metadata synchronization dependency graph. Regularly drop decommissioned tenant objects.

3. Safe Worker Recovery:

If a worker’s metadata sync is corrupted or failed, stop propagation before retrying:
SQL

SELECT stop_metadata_sync_to_node('worker_4', 5432, true);

10. Level Up on The Sev-1 Database

This challenge illustrates the core philosophy of The Sev-1 Database: understanding the exact C-source call paths, memory structures, and catalog invariants so you can diagnose production outages with precision.

On thesev1database.com, we’ve built:

• 71 deep engine lessons with verified C source citations.

• 105 production runbooks with literal psql outputs.

• 102 lab-verified SQLSTATE error references.

👉 Start exploring with our 7-Day Free Pro Trial

Did you get the answers right? How does your team manage schema scale across distributed clusters? Let me know in the comments below!

Share

Leave a comment

Subscribe now

When I talk to database engineers about why high-scale PostgreSQL clusters fail, 80% of the time it comes down to a misunderstanding of how the engine handles retention.

Today, I’m sharing a complete preview of one of our core Pro engine lessons from The Sev-1 Database: Replication Slots.

If you’ve ever had a primary run out of disk space with no huge queries running, this is what was happening under the hood.

Subscribe now

Share The Sev-1 Database

Problem Slots Were Invented to Solve

Before replication slots existed (introduced in PostgreSQL 9.4), standbys relied on wal_keep_size (or the older wal_keep_segments).

Logic was simple: “Keep the last 64 GB of WAL on the primary and pray the replica doesn’t lag further than that.”

If the replica went down for maintenance or network latency spiked and the primary generated 65 GB of WAL, the primary recycled the older files. When the replica woke up, the WAL it needed was gone. Standby was hopelessly stranded (requested WAL segment has already been removed) and had to be rebuilt from scratch using pg_basebackup.

Replication slots solved this by turning “hope we keep enough” into a guarantee.

How a Slot Works in the C Engine

A replication slot (src/backend/replication/slot.c) is a named, durable data structure written to $PGDATA/pg_replslot/<slot_name>/state.

Instead of guessing a fixed buffer size, a slot records the exact Log Sequence Number (restart_lsn) that a specific consumer still needs.

When your primary runs a checkpoint and looks to recycle old 16 MB WAL segments, it calls ReplicationSlotReserveWal():

[ Primary writes WAL ] ───> Current LSN: 0/2A000000
                                  │
Live Standby Slot ────────> restart_lsn: 0/29F80000 (Lag: ~500 KB) ──> OK to recycle before this!
                                  │
Stalled CDC Slot  ────────> restart_lsn: 0/14000060 (3 days old)   ──> LOCKED. No recycling!

Primary finds the minimum restart_lsn across all slots. It is physically forbidden from deleting any WAL segment beyond that point.

Slots survive server restarts. If a standby drops offline for 6 hours and comes back, it picks up right where it stopped. Zero manual intervention.

Trap: Logical Slots and Catalog Bloat

Read more

You don’t want another blog post.

You want to know:

What broke?
What should I run?
What did the output actually look like?
And how do I know I’ve fixed it?

That’s what I’m building with The Sev-1 Database.

Every PostgreSQL error explanation is free.

And now, you can try Pro for 7 days without paying.

No watered-down demo. No sample chapter.

For 7 days, you get the same Pro content engineers use:

• 100+ production runbooks diagnosis → exact SQL → pitfall → verification

• Deep PostgreSQL lessons including the actual psql commands and output

• Interview concepts + model answers for engineers preparing for senior/staff roles

• Lab-backed troubleshooting lock graphs, blocking PIDs, audit output, and other evidence from real scenarios

Idea is simple:

Don’t read about what you could do.
Run it. See the evidence. Understand why it worked.

If it becomes useful enough to keep in your incident toolkit, Pro is $24.99/month.

If it doesn’t, cancel before day 7 and you pay nothing.

Free content isn’t going away.

But if you’re the person people call when PostgreSQL is on fire, I’d try Pro for a week.

→ Start the 7-day Pro trial: https://thesev1database.com/pricing

Haider

Link: https://thesev1database.com

Table of contents (What’s inside)

• One thing to understand first

• Two purpose‑built storage services

• How a commit actually happens

• Why “stateless compute” is the real win

• Where it sits among the cloud-native family (Aurora · AlloyDB · Neon)

• Other bet: AI built into the engine (filtered DiskANN · Foundry models)

• What people miss

• Key takeaway

📬 I break down one Postgres / cloud‑database architecture like this.

Share The Sev-1 Database

One thing to understand first

Azure HorizonDB is Microsoft’s cloud-native, PostgreSQL-compatible database. It went to preview in the 2025–2026 wave as the tier that sits above Flexible Server, and Microsoft built it out of what it learned running some of the largest PostgreSQL fleets in Azure, including the databases behind ChatGPT.

Two ideas hold the whole thing up, and Microsoft states both of them plainly. Compute and storage are pulled apart. And the write-ahead log is the database. Get those two, and the rest of HorizonDB falls out of them. One caveat before we start: this is new, and it is in preview. Treat the specifics as a moving target, and check them against Microsoft’s current docs.

A picture helps. Every database does two jobs: cooking, which is running your queries, and keeping the food, which is holding your data. Where those two jobs live is the whole story. Ordinary PostgreSQL is a food truck. Both jobs ride in one vehicle, so the cook (the query engine) works right next to the pantry (your data files on local disk). That is simple, and it works. But the two can never scale apart. Want a second server to take more readers? You build a whole second truck and copy the entire pantry into it.

HorizonDB puts those jobs in two separate buildings: a kitchen (compute) and a storehouse (storage). Then it does something odd with the storehouse. It never shelves a single finished dish.

Your data pages are not kept there at all. What it keeps instead is one ever-growing order book: an exact, append-only list of every change ever made. Add this row. Change that value.

That order book is the WAL, PostgreSQL’s write-ahead log. Kitchen’s only job toward storage is to write the next line in it. It never ships finished plates back. And when a dish is finally needed, the storehouse replays the order book and rebuilds that exact dish on the spot.

That is what the log is the database really means. The true copy of your data is the running list of changes, not a stack of data files, and the pages get reconstructed from that list on demand. It is the family trait HorizonDB shares with Aurora, AlloyDB and Neon.

Two purpose-built storage services

Microsoft splits the storehouse into two fleets. Each one is tuned for a single job.

• WAL service takes the log stream from the primary. It is built for exactly one thing: appending WAL to the tail of a sequential stream, fast. And it is durable and zone-resilient by default.

• Data (page) storage fleet holds the actual pages. They are sharded across many storage nodes, and every shard is copied into more than one availability zone.

  
  As WAL lands, these nodes replay it to rebuild the 8 KB data pages and hand them back to compute. Spread a database’s pages across the whole fleet, with each node reading from a bank of high-performance NVMe disks, and Microsoft says the page tier can serve hundreds of thousands of IOPS per instance. Both fleets lean on Azure Blob storage underneath for durability.

Here is the consequence, and it is the whole point. Compute never writes data pages to storage. It writes only the log. Pages are just derived state that storage rebuilds from WAL. That is exactly why checkpoints and full-page writes, the biggest sources of write amplification in classic PostgreSQL, stop being the compute node’s job at all.

How a commit actually happens

Follow a single UPDATE on the primary.

• The change is written as WAL and appended to the durable WAL service, which acknowledges it back to the client.
  
  Microsoft’s framing here is unusually precise: the WAL service durably commits a transaction across three availability zones in a single network hop. A classic PostgreSQL primary plus hot standby needs four hops to make that same commit durable, and every extra hop is one more thing that can fail or add latency. On this path Microsoft reports sub-millisecond multi-zone commit latencies.

• After that, filtered WAL is applied asynchronously to the data shards that own those pages, and streamed out to the read replicas so they can refresh whatever pages they already hold in memory.

Compare that to community PostgreSQL. There, the same commit writes WAL to the server’s own disk, and sooner or later the server has to checkpoint its dirty pages down to its own data files.

A hot standby has to receive that WAL and replay it onto its own separate copy of the data. HorizonDB acknowledges the moment the shared, zone-resilient log service has the record. And there is only ever one copy of the data, shared by every replica.

Why stateless compute is the real win

Durability, checkpointing, full-page writes, WAL archiving, backups: all of it moves into the storage layer. So the compute node is left doing almost nothing except running your queries.

Microsoft publishes a whole table of traditional PostgreSQL background jobs it hands off from compute to storage. The takeaway for a practitioner is simple. The CPU, disk and network that used to go on keeping the database durable now go on your workload instead.

Each compute replica gets 8 GB of memory per core, and a local NVMe cache for hot pages on top of that.

Two operational superpowers fall out of a compute tier that owns no data.

• Read replicas come up almost instantly. A new replica just attaches to the same shared storage. There is no volume to copy and replay first.
  
  A read-only endpoint then load-balances across them. And since the storage layer feeds them WAL, not the primary, adding replicas puts basically no extra load on the primary. Each one scales on its own, so you get real workload isolation.

• Failover is fast and clean. A standby already shares the same durable WAL and storage as the primary, so it only has to replay a little WAL to take over.
  
  Because storage is the single source of truth, the old primary cannot diverge after a failover. That kills the pg_rewind-style “which timeline won?” reconciliation that self-managed failovers always have to wrestle with.

Where it sits among the cloud-native family

Every product in this pathway makes the same core move. Ship the log, and let a distributed storage service turn it into pages.

What differs is where each one spends that architecture. Lining HorizonDB up against its peers is the fastest way to see what it actually is.

• vs AWS Aurora. Aurora slices data into 10 GB segments, replicates them six ways across three AZs, and commits on a 4-of-6 quorum, with up to 15 shared-storage replicas. HorizonDB gets to the same “log is the database” goal by another route: two separate services (a WAL service and a page-server fleet) instead of one unified redo fleet. And it frames durability as a three-AZ commit in a single hop rather than a published quorum count. Same family, different plumbing.

• vs Google AlloyDB. AlloyDB is disaggregated Postgres too, but its headline act is a built-in columnar engine that speeds up analytical (HTAP) queries on the same data. HorizonDB’s headline is a different one: AI in the engine plus raw OLTP scale, not a columnar analytics accelerator.

• vs Neon. Neon takes disaggregation to a serverless extreme, with scale-to-zero and instant copy-on-write branching. HorizonDB is not serverless, it is provisioned. It is tuned for large, always-on, mission-critical OLTP, not ephemeral branch-per-PR development.

• vs Azure Flexible Server (the managed tier right below it in Azure). Flexible Server is community PostgreSQL on a managed disk: a single primary that owns its data files. HorizonDB is the disaggregated tier above it. Microsoft cites up to 3,072 vCores across primary and replicas, storage that auto-grows to 128 TB, and up to 3x the throughput of open-source Postgres on transactional workloads.

The one-line mental model: Aurora buys you AZ-resilient durability, Neon buys elasticity and branching, AlloyDB buys analytics. HorizonDB buys OLTP scale plus an AI stack, on Azure.

Other bet: AI built into the engine

HorizonDB’s second design bet is as big as the storage one. Instead of bolting a separate vector database and a layer of app-side glue onto Postgres, Microsoft pulls the AI work inside the engine, right next to your operational rows.

The announcement leads with two concrete engine features, and one launch customer frames the service as being for “Vector DB, RAG, and Agentic AI.” Read each of the two as feature, then value, then impact, rather than as a name to memorise.

1. A filtered DiskANN vector index.

• Feature. HorizonDB adds advanced filtering to the DiskANN vector index, so an ordinary SQL predicate, your WHERE clause, gets pushed down into the vector similarity search itself. Microsoft says this gives significant performance and scalability gains over pgvector’s HNSW index while holding accuracy, and that it is built for similarity search over transactional data already living in Postgres.

• Value. Filtered vector search is the classic trap. Pre-filter, and you fight the index. Post-filter, and you wreck recall. Ask for the 10 nearest rows “for tenant 42” and a naive index hands you the 10 globally-nearest, most of which you then throw away, sometimes leaving you two real matches instead of ten. Push the predicate into the graph traversal, and the search only ever walks candidates that already satisfy the filter.

• Impact. “Find similar rows for this customer, in this category, since this date“ stays fast and accurate as the table grows. And because the vectors sit right beside the rows they describe, you are not running a separate vector store or keeping two copies of the truth in sync. One database, one backup, one security boundary.

2. Built-in AI model management.

• Feature. Generative, embedding and reranking models from Microsoft Foundry are wired into the database with zero configuration, callable from inside the engine.

• Value. Normally, turning a row into an embedding, or calling an LLM, means standing up app-tier code: an API client, secret management, retries, a round-trip out of the data tier. Make the model a first-class thing the database itself can call, and all of that plumbing collapses.

• Impact. You can embed a row as you write it, or rerank a result set, without the data ever leaving the database’s trust boundary, and without a separate service to run. Reranking earns its spot here. It is the step that turns a raw nearest-neighbour list into genuinely relevant RAG context, and now it is a built-in model call instead of one more dependency to operate.

Why this is even possible, and where to go deeper. Both features lean straight on the architecture from the first half of this post. A stateless compute tier with spare CPU, plus a shared page store, is exactly the headroom that in-engine model calls and big vector scans need to run without starving your transactions.

That is the through-line to hold: the storage design is what pays for the AI design. The broader stack, surface by surface, is a deep topic of its own: durable embedding pipelines, BM25 keyword ranking, hybrid search, graph queries, agent protocols, all with runnable SQL.

Reproduced on PostgreSQL 17.10. Failure-mode signal: pgmoodring.com —105 reports till 24 July, deadlocks #1. No numbers in this post are fabricated.

What people miss

• PostgreSQL-compatible still means proprietary storage. Same as Aurora and AlloyDB: your SQL ports over, but the storage internals and some operational behaviours are Microsoft-specific. Factor in the lock-in.

• It is single-writer. HorizonDB scales reads by adding replicas onto shared storage, but writes still funnel through one primary. It is not multi-master.

• Preview means real gaps today. At launch, Microsoft documents fixed seven-day backup retention, no cross-region read replicas, service-managed keys only (no customer-managed keys), and no built-in connection pooling yet. If your workload needs any one of those, it is a blocker. The full preview reality check lists them with their impact.

• Match the tool to the need. If a single Flexible Server primary carries your load comfortably, the disaggregated tier may be more capability, and more cost, than you actually need.

🚀 The PostgreSQL Health Report bundles the lock-chain and contention checks I run during a Sev-1 into one SQL file — $29.

Key takeaway

Do not memorise HorizonDB as a feature list. Hold the mental model. Compute ships the log; a two-fleet storage service turns that log into the database.

From that one idea you can derive the rest, and place it next to its neighbours: instant shared-storage replicas and clean, no-rewind failover (the family trait), a three-AZ single-hop commit (its own twist), and the headroom for an in-engine AI stack (its bet).

Aurora, AlloyDB and Neon start from the same idea and spend it differently. Once the model clicks, you can reason about any of them, which is exactly the muscle a real design discussion, or an interview, asks for.

References Azure HorizonDB: Architecture, AI Functions. Azure HorizonDB Architecture Details, Revealed from Microsoft.

Premium edition. This is a reproduction RCA not a customer incident. Every log line, lock row, error and timing below is real captured output from a local PostgreSQL 17.10 run. Nothing is hand-written for the article; it is generated from the artifacts my reproduction lab captured.

Contents

1. Setup

2. What actually happened

3. Real deadlock report

4. Victim’s client error

5. Root cause: ShareLock on transaction

6. The lock graph (pg_locks)

7. RCA timeline

8. Right way to fix it

9. Reproduce it yourself

1. Setup

A single local PostgreSQL 17.10 instance. One table, seeded with 50,000 rows an ordinary accounts table, the kind every payments, ledger or inventory system has:

CREATE TABLE wallet.accounts (
    id      bigint PRIMARY KEY,
    owner   text    NOT NULL,
    balance numeric NOT NULL DEFAULT 0
);
-- seeded with 50,000 rows

One setting matters for this write-up: I set deadlock_timeout = '2s' in the lab (production default is 1s) so I had a clean window to snapshot the blocked state before the detector fired. That timeout is exactly how long a session waits on a lock before PostgreSQL bothers to check for a cycle it is not a performance knob, it is the deadlock detector’s patience.

2. What actually happened

Read more

Share The Sev-1 Database

𝗪𝗵𝗮𝘁 𝗱𝗶𝗱 𝗜 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝘄𝗼𝗿𝗸 𝗼𝗻 𝘁𝗼𝗱𝗮𝘆?

Not for timesheets.

Not for tickets.

Just to understand what production engineers are really spending their time on.

Then I had another thought...

𝗪𝗵𝗮𝘁 𝗶𝗳 𝘄𝗲 𝗮𝗹𝗹 𝗹𝗼𝗴𝗴𝗲𝗱 𝗷𝘂𝘀𝘁 𝗼𝗻𝗲 𝗣𝗼𝘀𝘁𝗴𝗿𝗲𝗦𝗤𝗟 𝗶𝗻𝗰𝗶𝗱𝗲𝗻𝘁 𝗮 𝗱𝗮𝘆?

Imagine opening one page and seeing what engineers around the world are reporting today.

Deadlocks.

Replication lag.

Connection storms.

Checkpoint issues.

Not opinions.

Just real production problems engineers are choosing to share.

So I vibe-coded a small experiment called 𝗣𝗚 𝗠𝗼𝗼𝗱 𝗥𝗶𝗻𝗴.

The idea is simple:

• Log the PostgreSQL incident you worked on today.

• See what other PostgreSQL engineers are reporting.

• Build your own month-by-month production history.

For me, it also becomes a signal for 𝗧𝗵𝗲 𝗦𝗲𝘃-𝟭 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲.

If I consistently see the community struggling with deadlocks or replication, that’s where I know I should invest more time creating guides, labs, and troubleshooting content.

It’s an early experiment, so I’d genuinely love feedback from people running PostgreSQL in production.

🌐 𝗽𝗴𝗺𝗼𝗼𝗱𝗿𝗶𝗻𝗴.𝗰𝗼𝗺

SQL Server → Postgres gets almost no airtime. Which is strange, because it’s not a small migration path plenty of shops are quietly doing it. They just aren’t writing about it.

I have done one. 9TB, production, ORM-heavy application, real users, real SLAs. And the thing that nearly broke us wasn’t on anyone’s migration checklist.

Share The Sev-1 Database

Setup

Schema conversion: done. Datatype mapping: done. T-SQL stored procedures rewritten into PL/pgSQL: done, painfully, but done. Indexes recreated with equivalent definitions on the Postgres side: done.

By every standard pre-cutover checklist, we were green across the board.

Then the application went live on Postgres for stress testing and the ORM-generated queries exact same queries, same predicates, same expected index usage were slow. Not slightly slower, needs tuning slow. Materially, user-facing-latency slow.

Part that wasted our time

First instinct: missing indexes. Checked the indexes existed. Matched the predicates. The planner was even picking them. EXPLAIN ANALYZE showed index scans, not sequential scans. On paper, everything looked correct.

So if the right index is being used, why is it this slow?

We burned real hours chasing planner statistics, work_mem, connection pool settings the usual suspects when Postgres should be fast and isn’t. None of it moved the needle meaningfully.

Actual answer: the indexes were fragmented. Badly.

Why this happens on this specific migration path, and not others

If you’ve spent your career on SQL Server, fragmentation is a background concern. You’ve got scheduled maintenance plans doing index rebuilds/reorganizes, fill factor tuned per workload, and the app almost never feels it directly because the DBA tooling around SQL Server has been dealing with this problem for decades in a very visible, very manageable way.

Postgres doesn’t fragment in the same way, and it doesn’t get maintained the same way by default either.

Postgres uses MVCC. Every UPDATE doesn’t modify a row in place it writes a new row version and marks the old one dead. Those dead tuples sit there until VACUUM reclaims them. Indexes point at physical locations (tuple pointers), so when rows move around under update-heavy workloads, indexes accumulate dead entries and lose their tight, sequential layout.

That’s bloat, and bloat in an index behaves exactly like fragmentation the index is technically valid and technically used, but it’s now doing far more I/O per lookup than it should.

Now add the ORM into the picture. ORMs are notorious for generating update patterns that are far from bloat-friendly: wide UPDATE statements that touch indexed columns unnecessarily, no batching, no awareness of Postgres’s HOT (Heap-Only Tuple) update optimization, which only kicks in when the update doesn’t touch any indexed column.

Touch an indexed column on every update which most ORM-generated save() calls do without you asking them to and you lose HOT updates entirely. Every single update now forces an index update too, and bloat accumulates far faster than a lean, hand-written update path would produce.

So: correct schema, correct indexes, correct query plans on paper and still 5x the I/O it should be doing, because nobody told Postgres this index needs attention the way SQL Server’s maintenance plans quietly had been doing for years on the old system.

What actually fixed it

- Autovacuum tuning per table, not just database-wide defaults high-churn tables needed far more aggressive settings than Postgres ships with out of the box

- Bloat monitoring added to standard health checks, so this gets caught in staging next time, not three months post-cutover

- A longer-term conversation with the app team about batching updates and avoiding unnecessary writes to indexed columns

- Clean up index bloat where heap size was around 100GB but index size of table was 1TB+.

Currently we are reviewing each index def, validating if it requires it took us hours to reach this point.

Once Postgres migration is completed with expected results, will share complete series of challenges we went through.

None of this is exotic. It’s all standard Postgres operational knowledge.

The problem was never technical difficulty it was that this wasn’t on the migration checklist, because migration checklists are almost all written from an Oracle → Postgres lens, where this specific failure mode doesn’t show up the same way.

Actual lesson

If you’re migrating from SQL Server to Postgres, don’t just migrate the schema and the indexes. Migrate the maintenance mindset.

SQL Server has spent decades normalizing scheduled index maintenance as an invisible background job. Postgres expects you to understand MVCC, vacuum, and bloat as first-class operational concerns from day one not something you reach for after users start complaining.

The query plan being correct tells you the optimizer picked the right index. It tells you nothing about whether that index is healthy.

1. Constraint That Shaped Everything

2. 64-Bit ID: Time + Shard + Sequence

3. Routing: Two Lookups, One Python Dict

4. Logical Shards via PostgreSQL Schemas

5. Moving Shards With Streaming Replication

6. Why Not Redis, HBase, or Composite Keys?

7. What Any PostgreSQL Team Can Apply Today

📨 If this was useful, share it with one engineer on your team who manages production databases.

Share The Sev-1 Database

1. Constraint That Shaped Everything

Instagram launched in October 2010 on a single PostgreSQL server on EC2. By September 2011, they had over 10 million users. Team? Two-and-a-half engineers. No ops team. No DBA.

From co-founder Mike Krieger on Hacker News:

We’re on EC2, which has its set of limitations but means we can run a 10 million+ user system with two-and-a-half engineers (and no ops team / overhead).

Photos go to S3. PostgreSQL stores metadata, social graph, likes, comments, feed relationships. Almost every query is keyed by user. If you can guarantee a query touches exactly one user, you can guarantee it touches exactly one shard.

That’s the insight they exploited.

2. 64-Bit ID: Time + Shard + Sequence

Instagram designed a custom 64-bit ID generated entirely inside PostgreSQL via a PL/pgSQL function:

• 41 bits — milliseconds since a custom epoch (~69 years of runway)

• 13 bits — logical shard ID (0–8191)

• 10 bits — auto-increment sequence mod 1024

Function fires on DEFAULT nextval(). No application code generates IDs. No external service. Database itself produces globally-unique, time-sortable identifiers.

Why time-sortable? “Newest first” feeds become a simple index scan no secondary sort on a timestamp column.

Why 64-bit single integer instead of a composite key? Krieger’s answer:

“We often have to store keys in other systems like Redis, where having a single 64-bit integer makes it more portable and stored compactly (Redis, for example, has an optimization when storing integer values vs string values in its lists).”

One integer. Portable across every cache layer. No serialization overhead.

3. Routing: Two Lookups, One Python Dict

Routing is surprisingly simple. From Krieger’s HN response:

“Each lookup is: user_id % 1000 → schema ID, schema ID → database ID, then SELECT FROM schemaID.tablename on that particular database.”

That mapping lives as a Python dictionary inside their Django application. Not a service. Not a database call. A dict.

# Conceptual representation of their routing
SHARD_MAP = {
    0: 'db-01',
    1: 'db-01',
    2: 'db-03',
    # ... through 999
}

def get_db(user_id):
    schema_id = user_id % NUM_SHARDS
    return SHARD_MAP[schema_id]

When asked about routing tables vs algorithmic hashing, Krieger said:

“Right now, it’s a lookup dict in our Django app which involves brief downtime just to update the shard map when moving the data.”

O(1). Zero network cost. Tradeoff: updating the dict requires a brief deploy.

4. Logical Shards via PostgreSQL Schemas

Instagram pre-creates all logical shards as PostgreSQL schemas. Each physical server hosts many schemas:

-- Each schema is a complete logical shard
CREATE SCHEMA insta5;
CREATE TABLE insta5.photos (
  id bigint PRIMARY KEY DEFAULT insta5.next_id(),
  user_id bigint NOT NULL,
  ...
);

CREATE SCHEMA insta6;
CREATE TABLE insta6.photos (
  id bigint PRIMARY KEY DEFAULT insta6.next_id(),
  user_id bigint NOT NULL,
  ...
);

Why schemas instead of separate databases or table-name prefixes? Krieger explained the PostgreSQL advantage:

“One huge benefit to PostgreSQL is the schema/tablespace feature, since it means all our logical shards all live inside one database.”

Each schema has its own sequence. Database’s own nextval() guarantees no duplicate IDs within a shard. 13-bit shard ID in the key guarantees no duplicates across shards.

When a physical server gets hot, you move entire schemas to a new machine. The application dict update is the only code change. No data re-keying. No ID rewrites.

This is the detail most sharding projects get wrong. They shard physically first and cannot rebalance without rewriting keys.

🚀 PostgreSQL Health Report — One SQL file. 60+ diagnostic checks. Detects vacuum lag, bloat, lock chains, replication slot danger, unindexed FK columns, and more.

Get it here → $29

5. Moving Shards With Streaming Replication

How do you move a logical shard from one physical server to another without extended downtime?

Krieger described their process:

“The way we move shards is to use PostgreSQL’s built-in streaming replication to create an exact, in-sync copy of a set of tablespaces, then ‘fail over’ to a new machine and start reading/writing to a subset of those tablespaces.”

The process:

1. Set up streaming replication from source to destination server

2. Wait for the replica to be fully in sync

3. Briefly stop writes (update the Django dict to point to the new server)

4. Start reading/writing to the new machine

Downtime window is the dict update seconds, not minutes. Data copy happens in the background via PostgreSQL’s native replication. No custom migration tooling.

Clock Drift and Duplicate Keys

41-bit timestamp relies on clocks being roughly correct. Instagram uses ntpd in gradual-adjustment mode (no sudden jumps). But what if a clock is slightly behind after a shard move?

Krieger’s response:

“At worst, you’d have a duplicate key when trying to insert, and can re-try without the risk of having a duplicate ID floating around your system.”

Failure mode is a caught exception on insert, not silent data corruption. 10 bits of sequence give 1,024 IDs per millisecond per shard a shard would need to be creating more than 128 entries per second with clocks 8+ seconds off to trigger this.

6. Why Not Redis, HBase, or Composite Keys?

Instagram uses Redis heavily for other systems. They explicitly considered it for ID generation and rejected it:

“[Redis] would have introduced a single point of failure, unless we split the load between several Redis instances, at which point it would be hard to make the IDs time-sortable. Also, most of our Redis systems are durable within a minute (we write to disk on a slave every minute), but if we were to lose the master and slave simultaneously (imagine an EC2 network issue), then it would be hard to know what the last known ‘good’ ID was.”

On HBase and NoSQL solutions:

“The data we shard out is more suited to an RDBMS, and since we’re way more familiar with that world than with HBase and similar, it was the choice that let us make the most progress in a short time with a small team.”

This is the underrated lesson. They didn’t pick PostgreSQL because it was technically optimal. They picked it because 2.5 engineers can’t afford the learning curve of a system they don’t deeply understand. Familiarity is a scaling strategy when your team is small.

7. What Any PostgreSQL Team Can Apply Today

Encode shard identity into your primary keys. Single highest-leverage decision in a sharded system. It eliminates routing services, lookup tables, and entire categories of cross-shard bugs.

Shard logically, not physically. Pre-create thousands of schemas on a handful of machines. When you need to rebalance, move schemas don’t re-key data.

Use stock database features. PL/pgSQL functions, schemas, sequences, streaming replication Instagram’s entire sharding infrastructure used features that ship with PostgreSQL. No custom extensions. No external ID services.

Choose what your team already knows deeply. Instagram explicitly rejected technologies they were less familiar with. At 2.5 engineers, the cost of learning a new system is measured in months of shipping velocity.

Keep the routing layer dumb. A Python dict that maps shard→server is less elegant than consistent hashing. It’s also something any engineer can debug in 30 seconds at 3am during an incident.

📧 Want the complete PostgreSQL Health Report? 60+ diagnostic queries — statistics freshness, extended stats coverage, bloat, vacuum health. Ready-to-run fixes.

Get it here → $29

📌 Upgrade to Premium ($8/month) — full SQL diagnostic packs, incident playbooks, and the queries I actually run during a Sev-1.

Upgrade to Paid

Share

Subscribe now

Leave a comment

Sources

• Instagram Engineering — Sharding & IDs at Instagram (September 2011) — Original post (now dead)

• Hacker News discussion with author responses from mikeyk (Mike Krieger) — news.ycombinator.com/item?id=3058327

• Instagram Engineering — Storing hundreds of millions of simple key-value pairs in Redis — instagram-engineering.com

Table of Contents

1. Running vs Winning Distinction

2. First Wrong Diagnosis: Bump autovacuum_max_workers

3. Second Wrong Diagnosis: Run VACUUM FULL

4. What Autovacuum Actually Does (PG 18)

5. Query 1 — Dead Tuple Accumulation Rate

6. Query 2 — Current Autovacuum Activity

7. Query 3 — Worker Saturation

8. Query 4 — Tables That Never Autovacuum

9. Query 5 — Long-Running Transactions Blocking Cleanup

10. Query 6 — Scale Factor Audit on Large Tables

11. Query 7 — Cost-Limit Throttling

12. Query 8 — Aggressive vs Normal Autovacuum

13. Query 9 — Wraparound Worker Status

14. Query 10 — Per-Table Autovacuum Settings Override

15. Emergency Decision: When to Run Manual VACUUM

16. Prevention Checklist

17. Key Takeaway

📨 If this saved you time, share it with one engineer who manages production databases.

Share The Sev-1 Database

1. Running vs Winning Distinction

I’ve seen instances where autovacuum was technically active in pg_stat_activity while dead tuples climbed past 30% on the highest-write tables.

Running means process exists. Winning means it’s keeping pace with your write rate.

Confusing two is the most expensive autovacuum mistake. Queries below distinguish them.

2. First Wrong Diagnosis: Bump autovacuum_max_workers

Instinct is: autovacuum isn’t keeping up, give it more workers.

Problem: autovacuum throughput is gated by autovacuum_vacuum_cost_limit (shared across all workers by default). Adding workers without raising the cost limit just splits the same I/O budget more ways. Each worker goes slower. Net progress identical.

Raise cost limit first. Then add workers if the limit isn’t the bottleneck.

3. Second Wrong Diagnosis: Run VACUUM FULL

VACUUM FULL takes an ACCESS EXCLUSIVE lock and rewrites the entire table. On a 500GB table in production, this means hours of downtime.

99% of bloat problems can be fixed with plain VACUUM, pg_repack (online), or tuning autovacuum to actually win. VACUUM FULL is almost always the wrong answer.

4. What Autovacuum Actually Does (PG 18)

From the PostgreSQL 18 manual, Chapter 25.1.6:

The autovacuum daemon consists of multiple processes. There is a persistent daemon process, called the autovacuum launcher, which is in charge of starting autovacuum worker processes for all databases. The launcher will distribute the work across time.

Two triggers fire autovacuum on a table:

• autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor × n_live_tup dead tuples normal vacuum

• autovacuum_freeze_max_age reached aggressive vacuum for wraparound prevention

Default scale factor is 0.2 meaning autovacuum triggers at 20% dead tuples. On a 100M row table, that’s 20M dead tuples before autovacuum even starts. That’s not a setting designed for large tables.

5. Query 1 Dead Tuple Accumulation Rate

SELECT
  schemaname,
  relname AS table_name,
  n_live_tup,
  n_dead_tup,
  round(n_dead_tup::numeric / NULLIF(n_live_tup + n_dead_tup, 0) * 100, 2) AS dead_pct,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||relname)) AS total_size,
  last_autovacuum,
  last_autoanalyze,
  CASE
    WHEN n_dead_tup::float / NULLIF(n_live_tup, 0) > 0.2
    THEN 'Autovacuum losing — manual VACUUM needed'
    WHEN n_dead_tup::float / NULLIF(n_live_tup, 0) > 0.1
    THEN 'Bloat elevated — watch closely'
    WHEN last_autovacuum < now() - interval '24 hours'
      AND n_live_tup > 1000000
    THEN 'Large table not vacuumed in 24h'
    ELSE 'OK'
  END AS status
FROM pg_stat_user_tables
WHERE n_live_tup + n_dead_tup > 10000
ORDER BY n_dead_tup DESC
LIMIT 20;

Any row with Autovacuum losing is a table where autovacuum is running but not winning.

Read more

I wanted to do something practical to help.

So I decided to reduce the premium membership price for The Sev-1 Database from $8/month to $5/month.

Subscribe to The Sev-1 Database

For existing premium members, the discounted price has already been applied automatically.

Premium content currently includes:

• 20+ deep PostgreSQL troubleshooting guides

• 120+ diagnostic and debugging queries

• Production incident workflows

• Query tuning and planner analysis

• Locking, WAL, replication, vacuum, and performance troubleshooting

Sev 1 Premium Content

I also recently launched:
PostgreSQL DBA Interview Help Kit – Part 1

Originally valued at $12.99, it’s now available from starting price $1 for those preparing for interviews.

If you are already a premium member, drop me an email and I’ll send you a free copy.

Get it here:
PostgreSQL DBA Interview Help Kit

My goal is simple:
Help more engineers prepare better and get back into strong roles faster.

Newsletter:
The Sev-1 Database

Share The Sev-1 Database

Table of Contents

1. What We’re Working With

2. Paper

3. Four Ideas That Became Every SQL Optimizer

4. What PostgreSQL Kept From 1979

5. What PostgreSQL Had to Change

6. Where Selinger’s Assumptions Still Bite You in Production

7. 5 Things You Can Do About It

8. Key Takeaway

1. What We’re Working With

Every time you run EXPLAIN ANALYZE and see a join order, a hash join chosen over a nested loop, or a cost estimate in arbitrary units you are looking at output from an algorithm designed by Patricia Selinger at IBM Research in 1979.

Not a distant ancestor of the algorithm. Algorithm. Paper’s pseudocode is close enough to PostgreSQL’s planner source code that reading one helps you read the other.

46 years later, SQL Server, Oracle, DB2, MySQL, and PostgreSQL all run variations of same design. Understanding what Selinger got right and what she didn’t tells you why your database behaves the way it does.

2. Paper

“Access Path Selection in a Relational Database Management System”
Patricia G. Selinger, Morton M. Astrahan, Donald D. Chamberlin, Raymond A. Lorie, Thomas G. Price.
ACM SIGMOD International Conference on Management of Data, 1979.

Paper describes the optimizer for System R IBM’s research prototype that became DB2. It solves a problem that didn’t exist before 1979: given a SQL query with multiple joins, how does the database decide which order to join the tables, which access method to use, and which algorithm to apply at each step?

Before Selinger, databases picked join orders heuristically the order the programmer wrote them. Selinger’s insight was simple and has never been improved on: enumerate all plans, cost each one, pick the cheapest.

Paper Link

📨 If this was useful, share it with one engineer on your team who manages production databases.

Share The Sev-1 Database

3. Four Ideas That Became Every SQL Optimizer

Idea 1: Cost as a Currency

Selinger introduced concept of cost units an abstract currency combining I/O pages read and CPU work, blended into one number so the optimizer can compare any two plans.

In PostgreSQL today, cost 1.0 is the price of fetching one 8KB page sequentially. Every parameter seq_page_cost, random_page_cost, cpu_tuple_cost is a descendant of Selinger’s original units.

Idea 2: Dynamic Programming for Join Order

Rather than trying every permutation of N tables (which is N factorial untenable beyond 8 tables), Selinger applied dynamic programming. Build up optimal plans for 2-table joins, then 3-table joins, reusing the results.

PostgreSQL uses this exact approach. Source file PostgreSQL: joinrels.c is named after the concept. Query plans you see in EXPLAIN are the winners of a dynamic-programming search.

Idea 3: Interesting Orders

An ordered intermediate result is worth more than an unordered one it can feed a merge join, avoid a sort, or support ORDER BY for free. Selinger called these “interesting orders” and proved the optimizer must track them separately.

PostgreSQL’s pathkeys are the direct descendant. When you see a plan that skips a sort because an index returned data in the right order.

Idea 4: Selectivity Estimation via Statistics

Selinger introduced the idea that the optimizer must estimate how many rows pass each filter. To do this, database maintains statistics about column distributions.

Her 1979 formulas used simple assumptions: uniform distribution, column independence, constant join selectivity factors. Every one of those assumptions is wrong on real data.

Which brings us to the second half of this post.

4. What PostgreSQL Kept From 1979

Read src/backend/optimizer/README in the PostgreSQL source. It reads like a modernized version of Selinger’s paper.

Preserved almost unchanged:

• Dynamic programming for join enumeration (up to geqo_threshold, default 12 tables)

• Cost units as an abstract currency

• Interesting-order tracking via pathkeys

• Per-node cost estimation composed from child node costs

• Overall pattern: enumerate plans, cost them, pick the cheapest

5. What PostgreSQL Had to Change

The 1979 paper assumed:

• Columns are independent (wrong)

• Data is uniformly distributed (wrong)

• Join selectivity can be estimated from single-table stats (wrong past 2 joins)

• The database is small enough to cost every plan (wrong past ~12 tables)

Every major PostgreSQL statistics feature is a patch on one of these assumptions:

• Histograms (for non-uniform distributions)

• Most-common-value lists (for skewed data)

• GEQO genetic query optimization (for queries with too many tables for dynamic programming)

• Extended statistics via CREATE STATISTICS (for correlated columns)

• Planner hooks and custom scan nodes (for when the default costing fails)

The VLDB 2015 paper I wrote about last week measured what happens when these patches aren’t enough. Answer: plans that are 1,000x to 100,000,000x off on large join queries.

6. Where Selinger’s Assumptions Still Bite You in Production

Three patterns I see repeatedly:

Pattern 1: Correlated predicates at the leaf

WHERE region = 'US' AND currency = 'USD' the planner multiplies selectivities as if they’re independent. They aren’t. CREATE STATISTICS is the direct fix, and the PostgreSQL 18 manual says so in Chapter 14.2.2.

Pattern 2: Join cardinality explosion

Selinger’s join selectivity formula was a constant. PostgreSQL refined it but the error compounds with join depth. By the fifth join the estimate is meaningless. This is why big analytical queries against real schemas go wildly wrong.

Pattern 3: GEQO kicks in at 12 joins and plan becomes non-deterministic

When your query has more than geqo_threshold tables, PostgreSQL switches from Selinger’s dynamic programming to a genetic algorithm. Same query can produce different plans between executions. On Sev-1 calls this is what “the query was fine yesterday” usually means.

7. 5 Things You Can Do About It

Fix 1 — Read your EXPLAIN plans with Selinger’s framework in mind

Every node has a cost. Every cost came from a row estimate. Every row estimate came from statistics. When a plan is bad, walk backward through that chain.

Fix 2 — Raise geqo_threshold if your queries routinely touch 12+ tables

Set geqo_threshold to 14 or 16 if you have CPU budget. Deterministic dynamic-programming plans beat genetic-algorithm plans almost every time.

Fix 3 — CREATE STATISTICS on the column pairs Selinger’s math can’t handle

This is direct fix for Pattern 1. Eight years old, still the biggest planner win you can give yourself in one DDL statement.

Fix 4 — Use the EXPLAIN ANALYZE, BUFFERS output to validate cost model inputs

If shared_blks_read is high and your random_page_cost is 4.0, the planner thinks random reads are slow when they aren’t. Fix random_page_cost on SSDs.

Fix 5 — Stop fighting the planner. Fix its inputs.

Selinger’s algorithm is correct given accurate statistics. The VLDB paper proved this empirically in 2015. Hints, query rewrites, and enable_seqscan = OFF are not the answer. Fresh ANALYZE, correct statistics target, extended statistics, and correct cost parameters are.

8. Key Takeaway

Your PostgreSQL optimizer is a 46-year-old algorithm with 30 years of patches on top.

Algorithm is almost always right. Patches cover the places where Selinger’s 1979 assumptions fail on real data.

If you know which assumption is failing, you know which patch to apply.

🚀 PostgreSQL Health Report — 60+ diagnostic checks in one SQL file. Statistics freshness, extended stats coverage, cost parameter audit, bloat, vacuum health.

Get it here → $29

📌 Upgrade to Premium ($8/month) — Premium subscribers get the full SQL diagnostic packs, step-by-step playbooks, and the queries I actually run during a Sev-1.

Upgrade to Paid

Share

Leave a comment

References

• Selinger, P. G., Astrahan, M. M., Chamberlin, D. D., Lorie, R. A., Price, T. G. (1979). Access Path Selection in a Relational Database Management System. ACM SIGMOD. — https://dl.acm.org/doi/10.1145/582095.582099

• PostgreSQL 18 Source — src/backend/optimizer/README — GitHub

• PostgreSQL 18 Documentation — Genetic Query Optimizer (GEQO) — https://www.postgresql.org/docs/18/geqo.html

Engineers actively interviewing for DBA or SRE roles. They all say the same thing:

I know this stuff. I’ve run Postgres in production. But in the interview I freeze.

Premium subscribers who’ve gone deep into the incident write-ups, asking:

How do I turn this into something I can actually perform at a senior level?

This kit is built for both.

Value isn’t the questions.

It’s the language clarifying questions, reasoning patterns, way senior engineers communicate under pressure that signals: this person has handled a Sev-1 in production.

That’s the gap most candidates underestimate.

Part 1 covers what eliminates most candidates before the offer stage:

incident diagnosis, migration safety, and query plan reading.

Part 2 is where offers are won or lost. Work in progress.

→ PostgreSQL DBA Interview Prep Kit Part 1

Price: €12.99

Haider

Table of Contents

1. Why This Audit Matters

2. First Wrong Response: CREATE STATISTICS On Every Table

3. Second Wrong Response: Trust Autoanalyze to Catch It

4. How PostgreSQL Uses Extended Statistics Internally

5. Query 1 — Existing Extended Statistics Inventory

6. Query 2 — Top Slow Queries with Multi-Column WHERE Clauses

7. Query 3 — Detect Functional Dependencies Automatically

8. Query 4 — Find Estimate Divergence in pg_stat_statements

9. Query 5 — Column Correlation Via Cross-Tabulation

10. Query 6 — Extended Stats Coverage on Partitioned Tables

11. Query 7 — Statistics Target Audit

12. Query 8 — Before/After Plan Validation

13. Diagnostic Sequence to Follow

14. Key Takeaway

📨 If this saved you time, share it with one engineer who manages production databases.

Share The Sev-1 Database

1. Why This Audit Matters

If you read last week’s post, you already know the finding: PostgreSQL multiplies selectivities as if every column is independent. On real data, columns are correlated. Planner underestimates. Nested loops get chosen where hash joins should win. Queries that ran in seconds start taking hours.

Fix CREATE STATISTICS has been in PostgreSQL since version 10. Most production databases have zero extended statistics objects. This audit finds where you need them and where you don’t.

2. First Wrong Response: CREATE STATISTICS On Every Table

Don’t. Extended statistics have a real cost. ANALYZE time grows. Planner work per query grows. Defining mcv across five columns on a 500M-row table can add seconds to every ANALYZE run and measurable overhead to every query plan.

Target only where the planner is getting it wrong. Query 4 tells you where.

3. Second Wrong Response: Trust Autoanalyze to Catch It

Autoanalyze updates single-column statistics. It does not create extended statistics objects. An object only exists after you run CREATE STATISTICS manually. Until then, autoanalyze can refresh its values but it cannot invent the object itself.

4. How PostgreSQL Uses Extended Statistics Internally

From the PostgreSQL 18 manual, Chapter 14.2.2:

Extended statistics objects track data about the specified table, foreign table, or materialized view. The object does not generate statistics until ANALYZE is run on the table.

Three kinds of extended statistics:

• dependencies — functional dependency between columns. Cheapest to compute, smallest storage.

• ndistinct — number of distinct combinations. Critical for GROUP BY on multiple columns.

• mcv — most common value combinations. Largest storage, highest accuracy on skewed data.

Planner reads these from pg_statistic_ext_data at plan time.

5. Query 1 — Existing Extended Statistics Inventory

Start here. You need to know what already exists before creating more.

SELECT
  n.nspname AS schema_name,
  c.relname AS table_name,
  s.stxname AS stats_name,
  pg_catalog.array_to_string(
    ARRAY(
      SELECT a.attname
      FROM unnest(s.stxkeys) AS k
      JOIN pg_attribute a ON a.attrelid = s.stxrelid AND a.attnum = k
    ),
    ', '
  ) AS columns,
  s.stxkind AS kinds,
  CASE
    WHEN 'd' = ANY(s.stxkind) THEN 'dependencies '
    ELSE ''
  END ||
  CASE
    WHEN 'f' = ANY(s.stxkind) THEN 'ndistinct '
    ELSE ''
  END ||
  CASE
    WHEN 'm' = ANY(s.stxkind) THEN 'mcv'
    ELSE ''
  END AS stat_types
FROM pg_statistic_ext s
JOIN pg_class c ON c.oid = s.stxrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
ORDER BY n.nspname, c.relname;

If this returns zero rows, your database is running with the exact limitation the VLDB paper measured in 2015.

6. Query 2 — Top Slow Queries with Multi-Column WHERE Clauses

The queries most likely to benefit from extended statistics are the ones with two or more columns in the WHERE clause on the same table.

Read more

Table of Contents

1. What We’re Working With

2. Paper

3. Four Findings Matter

4. What PostgreSQL 18 Chapter 70 Actually Promises

5. Gap Between Manual and Reality

6. 6 Things You Can Fix (Paper Says Will Actually Help)

7. Checklist What to Verify Before You Blame Planner

8. Skip the Guesswork → PostgreSQL Health Check

📨 If this was useful, share it with one engineer on your team who manages production databases.

Last week’s post walked through an EXPLAIN ANALYZE plan line by line. The question I got back most often came in the same form every time:

“My estimate says 1,000 rows. Actual says 8 million. Why does PostgreSQL keep doing this?”

Short answer: PostgreSQL’s planner is rarely wrong about math. It’s wrong about your data. And that’s not a bug anyone is going to fix in PostgreSQL 18, 19, or 20 because the problem is deeper than a code path.

Four researchers proved this in 2015. Paper is public. Findings still apply to PostgreSQL 18 today. Every DBA should read it once. This post is that reading.

Share The Sev-1 Database

1. What We’re Working With

Before we get to the paper, here’s the thing it tried to measure.

Every query plan in PostgreSQL is a tree. Each node has two numbers the planner cares about:

• Rows — how many rows this node will produce

• Cost — how expensive this node is to execute

Cost is calculated from rows. Rows is calculated from statistics. Statistics are gathered by ANALYZE. If the row estimate is wrong, every cost decision downstream is wrong. Wrong index choice. Wrong join algorithm. Wrong memory allocation.

Question nobody had measured at scale until 2015: how wrong are these estimates on real workloads?

2. Paper

“How Good Are Query Optimizers, Really?”
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, Thomas Neumann.
Proceedings of the VLDB Endowment, Vol. 9, No. 3 (2015).

Paper

They built a benchmark called JOB the Join Order Benchmark. 113 real queries over the public IMDB dataset. Not TPC-H. Not synthetic. Real schemas with real skew and real correlations.

They ran JOB against PostgreSQL, two commercial databases (anonymized as DBMS A and DBMS B), and HyPer (a research system). At every operator in every plan, they recorded the estimated cardinality and the actual cardinality.

Then they asked four questions:

1. How accurate are cardinality estimates in practice?

2. If estimates are wrong, how much does it actually cost?

3. Is PostgreSQL’s cost model the problem, or are the inputs?

4. Do better join-enumeration algorithms help?

The answers changed how I diagnose slow queries.

3. Four Findings Matter

Finding 1: Estimates get wrong fast, and then exponentially wrong

Single-table filters were within a factor of 10 most of the time. Reasonable.

Two-way joins were already off by factors up to 100.

By the time you reach six-way joins, estimate-to-actual ratios hit 10⁶ to 10⁸. That’s six to eight orders of magnitude.

From the paper:

Cardinality estimates are routinely wrong by factors of 1,000 or more. The errors grow super-linearly with the number of joins.

If you’ve ever seen a query that runs fine on a 3-table join and collapses when you add a fourth, this is why. Error isn’t linear. Adding one more join doesn’t add 10% more risk. It multiplies the existing error.

Finding 2: PostgreSQL’s cost model is not the problem

This is the finding that changed my mental model completely.

Researchers ran every query with actual cardinalities substituted in, then let each database pick a plan using that correct information.

When given accurate cardinalities, PostgreSQL’s cost model chooses near-optimal plans.

Cost math is fine. Inputs are not.

If you’re reaching for hints, query rewrites, or SET enable_seqscan = OFF, stop. Paper says PostgreSQL already knows how to pick the right plan. You just need to fix what it believes about your data.

Finding 3: Independence assumption is where it breaks

PostgreSQL assumes that predicates on different columns are statistically independent.

WHERE country = 'US' AND state = 'California'

It computes:

selectivity(country='US') × selectivity(state='California')

Which is only correct if knowing the country tells you nothing about the state. California exists only in the US. Knowing one determines the other. Multiplication is wrong, and planner has no way to know.

Paper’s finding is that this single assumption drives most of the underestimate errors in JOB. It’s also exactly the problem PostgreSQL manual documents. Which brings us to section 4.

Finding 4: Join enumeration is a smaller problem than people think

Most engineers, when a query is slow, blame the join order. Paper tested this directly. Given correct cardinalities, even a basic dynamic-programming join enumerator produced near-optimal plans.

Join order matters. But it matters much less than your estimates being right in the first place. Fix cardinality first. Then worry about whether the planner is picking the wrong algorithm.

🚀 PostgreSQL Health Report — One SQL file. 60+ diagnostic checks. Detects stale statistics, bloat, lock chains, replication slot danger, unindexed FK columns, and more. Used by DBAs managing production at scale.

Get it here → $29

4. What PostgreSQL 18 Chapter 69 Actually Promises

Open the PostgreSQL 18 manual. Go to Chapter 70 How the Planner Uses Statistics. Read the first section carefully.

By default, the planner assumes that values of different columns are independent.

That is the manual telling you, in the current documentation for the current version, that default behavior is the problem Leis et al measured ten years ago.

Then manual tells you the fix exists:

Since PostgreSQL 10, it has been possible to tell the planner about multi-column correlations by creating extended statistics objects with the CREATE STATISTICS command.

And in Chapter 14.2.2 — Extended Statistics:

It is common to see slow queries running bad execution plans because multiple columns used in the query clauses are correlated. The planner normally assumes that multiple conditions are independent of each other, an assumption that does not hold when column values are correlated.

PostgreSQL 18 did not remove the independence assumption. It still ships. Fix is a feature you have to turn on, per table, per column set.

5. Gap Between Manual and Reality

Here’s the part that frustrates me. Problem is documented. Fix is documented. Eight years after CREATE STATISTICS shipped, I still walk into production PostgreSQL instances where it has never been used on a single table.

Run this against your database right now:

SELECT count(*) FROM pg_statistic_ext;

If the answer is 0 and for most PostgreSQL instances I audit, it is you are running with the exact limitation the VLDB paper measured in 2015. You haven’t opted out of it. You just never opted in to the fix.

This isn’t a PostgreSQL-is-bad story. It’s a tooling gap. Default is safe. Optimization is opt-in. And opt-in features in databases are the ones that almost never get turned on.

6. 6 Things You Can Fix (Paper Says Will Actually Help)

In order of impact, based directly on the paper’s findings and PostgreSQL 18 manual’s recommendations.

Fix 1 — Raise statistics target on high-cardinality filter columns

Default is 100. Meaning PostgreSQL samples 100 values for the most-common-values list and 100 histogram buckets. On a 100M row table with skewed data, that is not enough.

ALTER TABLE orders ALTER COLUMN customer_id SET STATISTICS 1000;
ANALYZE orders;

Raise it to 1000 on columns that appear in the WHERE clause of slow queries. 10,000 is the maximum and rarely needed.

Fix 2 — CREATE STATISTICS on correlated columns

This is direct fix for Finding 3.

CREATE STATISTICS stats_orders_region_status (dependencies, ndistinct, mcv)
ON region, status
FROM orders;

ANALYZE orders;

Three kinds of extended statistics:

• dependencies — for columns where one determines another (zip → city)

• ndistinct — for correct row estimates on GROUP BY multiple columns

• mcv — for skewed distributions where certain combinations dominate

Combine all three unless storage is a concern.

Fix 3 — ANALYZE after bulk changes

Autovacuum’s autoanalyze trigger is default 10% of the table. On a 1 billion row table, that’s 100 million changes before autoanalyze fires. By then your statistics have been wrong for hours or days.

After any large migration, backfill, or DELETE run ANALYZE manually. Don’t wait.

Fix 4 — Fix function-wrapped WHERE clauses

Paper found function wrapping disables index use silently. WHERE UPPER(email) = 'USER@X.COM' cannot use a B-tree index on email.

Two fixes:

-- Option 1: Functional index
CREATE INDEX idx_users_email_upper ON users (UPPER(email));

-- Option 2: Rewrite to use the existing index
SELECT * FROM users WHERE email = lower('USER@X.COM');

Fix 5 — Set random_page_cost correctly for SSD

Default is 4.0. That assumes spinning disk. On SSD or cloud block storage, random reads are close to sequential reads.

ALTER SYSTEM SET random_page_cost = 1.1;
SELECT pg_reload_conf();

This single change flips the planner’s cost math for thousands of queries. On cloud-hosted PostgreSQL it is the highest-impact one-line change you can make.

Fix 6 — Use EXPLAIN (ANALYZE, BUFFERS) not EXPLAIN ANALYZE

BUFFERS tells you whether the query hit the page cache or read from disk. Without BUFFERS, you’re debugging with half the information.

EXPLAIN (ANALYZE, BUFFERS) SELECT ...;

If shared_blks_read is much higher than shared_blks_hit, your working set doesn’t fit in shared_buffers, and no amount of query tuning fixes that.

7. Checklist What to Verify Before You Blame Planner

Run through these every time a plan looks wrong. Paper’s findings map directly.

• Estimated rows vs actual rows if off by more than 10×, statistics are the problem

• Last ANALYZE on the table if stale, fix that before anything else

• WHERE clause has multiple columns from the same table candidate for CREATE STATISTICS

• Any column wrapped in a function index bypass guaranteed

• random_page_cost set for your storage type

• shared_buffers sized for your working set

• pg_statistic_ext does it even have entries? If not, you’re running with the paper’s limitation

If every item checks out and the plan is still wrong, then and only thene planner might genuinely be wrong. That’s rare.

8. Skip the Guesswork → PostgreSQL Health Check

Reading one EXPLAIN plan teaches you. Reading one paper changes how you think. But your database has hundreds, maybe thousands, of active queries. Every one of them has a row-estimate number. Most of them you’ve never looked at.

PostgreSQL Health Check runs 60+ diagnostic queries against your database and surfaces:

• Every table with stale statistics

• Extended statistics coverage

• The top time-consuming queries

• Missing and unused indexes

• Table bloat, vacuum health, and statistics freshness

• Everything ranked by severity with specific fix recommendations

One report. 60+ queries. Every blind spot uncovered.

👉 Get Your PostgreSQL Health Check →

Key Takeaway

PostgreSQL’s cost model is not the problem.

Assumptions about your data are.

VLDB proved it in 2015. PostgreSQL 18 manual confirms it. Fix CREATE STATISTICS has existed for eight years. Most databases have never used it.

Now yours doesn’t have to be one of them.

📧 Want the deep-dive playbook?

Upgrade to Paid

Enjoyed reading? Restack, share with your audience.

Share

Leave a comment

References

• Leis, V., Gubichev, A., Mirchev, A., Boncz, P., Kemper, A., Neumann, T. (2015). How Good Are Query Optimizers, Really? Proceedings of the VLDB Endowment, Vol. 9, No. 3. — http://www.vldb.org/pvldb/vol9/p204-leis.pdf

• PostgreSQL 18 Documentation, Chapter 69 — How the Planner Uses Statistics — https://www.postgresql.org/docs/18/planner-stats-details.html

• PostgreSQL 18 Documentation, Chapter 14.2 — Statistics Used by the Planner — https://www.postgresql.org/docs/18/planner-stats.html

• PostgreSQL 18 Documentation, Chapter 14.2.2 — Extended Statistics — https://www.postgresql.org/docs/18/planner-stats.html#PLANNER-STATS-EXTENDED

• Join Order Benchmark (JOB) — https://github.com/gregrahn/join-order-benchmark

Table of Contents

1. Why Engineers Get This Wrong

2. First Wrong Response: Force Index

3. Second Wrong Response: Add Another Index

4. How PostgreSQL Actually Decides Between Index Scan and Sequential Scan

5. Eight Causes of Index Bypass In Order of Frequency

6. Query 1 Index Usage by Table: Which Indexes Are Never Used?

7. Query 2 Table Statistics Freshness: Is Planner Working From Stale Data?

8. Query 3 Column Statistics Detail: What Planner Believes About Your Data

9. Query 4 Function-Wrapped Columns: Index Bypass Nobody Expects

10. Query 5 Table Bloat Distorting Cost Estimates

11. Query 6 Storage Cost Parameters: Is random_page_cost Wrong for Your Storage?

12. Query 7 Actual vs Estimated Rows: Finding Planner’s Worst Mistakes

13. Query 8 Force Index Usage for Testing (NOT for Production)

14. Query 9 Full Query Plan Audit for Your Slowest Queries

15. Diagnostic Sequence to Follow

16. Key Takeaway

📨 If this saved you time, share it with one engineer who manages production databases.

Share The Sev-1 Database

This is a post I’ve been asked about more than almost any other PostgreSQL topic. Question always comes in the same form: Index is there. Query uses the column. Why is PostgreSQL doing a sequential scan?

Frustrating answer: PostgreSQL is almost always right. It’s choosing the sequential scan because its math says it’s cheaper. Problem is usually that the math is based on wrong inputs stale statistics, wrong cost parameters, or a query structure that prevents index use.

Before the queries, two responses that make the situation worse.

Why Engineers Get This Wrong

Instinct is to fight the planner. Add a hint. Force the index. Rewrite the query to trick PostgreSQL into using it. These responses treat the planner as an obstacle rather than a tool that’s working with incorrect information.

PostgreSQL doesn’t have index hints in the traditional sense.

SET enable_seqscan = OFF exists, but using it in production is almost always wrong it forces index use even when it’s genuinely more expensive. You’ll speed up one query and slow down ten others.

Right response is to fix what the planner is getting wrong, then let it choose. Query 7 is designed exactly for this: it shows you where the planner’s estimates diverge most from reality. Fix the estimate; the plan fixes itself.

First Wrong Response: Force the Index

SET enable_seqscan = OFF forces PostgreSQL to use indexes even when they’re more expensive. This works temporarily on the query you’re debugging. It breaks other queries in the same session that legitimately benefit from sequential scans. Never set this globally. Never set it in application connection strings. Use it only in a debugging session to confirm the index would actually be used if the planner chose it.

Second Wrong Response: Add Another Index

If PostgreSQL isn’t using the existing index, adding another index on the same or related columns is almost never the fix. PostgreSQL has one index to choose from and isn’t using it adding a second gives it two to ignore. Problem is why it’s not using the first one, not how many indexes are available.

How PostgreSQL Actually Decides Between Index Scan and Sequential Scan

PostgreSQL’s cost-based optimizer calculates a cost estimate for every possible execution plan. It chooses the plan with the lowest estimated cost. Cost is measured in arbitrary units, where 1.0 = time to fetch one 8KB page sequentially from disk.

An index scan costs:
(pages traversed in index) × random_page_cost + (rows fetched from heap) × random_page_cost.

A sequential scan costs:
(total table pages) × seq_page_cost.

If the planner estimates 90% of rows will match the WHERE clause even with a correct index a sequential scan is genuinely cheaper.

If the planner’s statistics are stale and it thinks 90% match when only 0.1% actually match it chooses a sequential scan incorrectly. That’s the case you’re diagnosing.

🚀 PostgreSQL Health Report 60+ diagnostic checks. Unused indexes, vacuum lag, replication slot danger, and more.

Get it here → $29

Eight Causes of Index Bypass In Order of Frequency

1. Stale statistics planner underestimates selectivity, thinks too many rows match

2. Low selectivity column actually has few distinct values; sequential scan is correct

3. Function wrap on the column WHERE UPPER(email) = ? breaks index on email

4. Wrong random_page_cost default 4.0 assumes spinning disk; SSDs need 1.1

5. Table too small PostgreSQL always seq scans tables under ~8 pages

6. Implicit type cast WHERE id = '123' when id is integer forces a cast, index may not apply

7. Table bloat dead tuples inflate estimated row count, making seq scan look relatively cheaper

8. LIKE with leading wildcard WHERE name LIKE '%smith' cannot use B-tree index

Query 1 Index Usage by Table: Which Indexes Are Never Used?

Read more

Table of Contents

1. Core Challenge: Multiplayer at Database Speed

2. Decision 1: Separate Real-Time Collaboration from Persistence

3. Decision 2: Read Replicas Routed by Staleness Tolerance

4. Decision 3: Aggressive Connection Pooling With PgBouncer

5. Decision 4: Partitioning Long-Lived Tables

6. What’s Universally Applicable

📨 If this was useful, share it with one engineer on your team who manages production databases.

Share The Sev-1 Database

The Sev-1 Database is a reader-supported publication. Each week I share practical takeaways from real incidents no theory, just what actually breaks and how to fix it.

Subscribe now

Figma is unusual among design tools in that its core value proposition is fundamentally a database problem. Real-time multiplayer editing where multiple users see each other’s changes instantly requires a data architecture that can handle concurrent writes, deliver fast reads, and maintain consistency across sessions.

Here’s how they’ve built and evolved that architecture over time.

Based on Figma’s engineering blogs and talks by Evan Wallace, here’s a grounded breakdown of how their PostgreSQL architecture evolves at scale.

Core Challenge: Multiplayer Collaboration as a PostgreSQL Performance Problem

Most web applications are read-heavy. Figma’s workload is different. In an active design session, every cursor movement, every property change, every layer modification generates a database write.

Multiply that by hundreds of simultaneous users in a shared file and you have a write workload that few relational databases are designed to absorb directly.

Figma’s answer wasn’t to move away from PostgreSQL it was to be thoughtful about what hits PostgreSQL and what doesn’t.

Decision 1: Separate Real-Time Collaboration from Persistence

Figma separates the live, in-session document state from the durable version stored in PostgreSQL.

Real-time collaboration is handled by a custom in-memory engine, where updates from multiple users are processed and synchronized.

PostgreSQL is used to persist the canonical document state rather than storing every individual user event.

Principle: the database should store a consistent, durable representation of state, not every intermediate change.

Decision 2: PostgreSQL Read Replica Routing by Staleness Tolerance

Figma routes queries based on whether they need fresh data or can tolerate a small replication lag.

Queries for file metadata, user settings, and non-real-time data go to read replicas. Queries that require the most current state of an active document go to the primary.

They built logic to detect when a replica’s lag exceeds a threshold and automatically fail that query to the primary. Replica lag threshold isn’t fixed it adapts based on the type of operation and the user’s context.

🚀 PostgreSQL Health Report One SQL file. 60+ diagnostic checks. Detects vacuum lag, bloat, lock chains, replication slot danger, and more. Used by DBAs managing production at scale.

Get it here → $29

Decision 3: PgBouncer Transaction-Mode Connection Pooling at Figma Scale

At Figma’s scale, the number of concurrent active users far exceeds what PostgreSQL can handle as individual connections. PostgreSQL allocates ~10MB per connection; with tens of thousands of concurrent users, a direct connection-per-user model is impossible.

Figma runs PgBouncer in transaction-pooling mode. Rather than each user holding a dedicated PostgreSQL connection, connections are shared a user gets a connection for the duration of a transaction, then it’s returned to the pool.

Transaction-pooling requires careful application design. Prepared statements, advisory locks, and session-level features don’t work reliably in this mode. Figma’s application layer was designed with this constraint from relatively early on.

Decision 4: PostgreSQL Range Partitioning for Long-Lived Tables

As Figma’s document history grew years of changes across millions of files some tables became large enough that even indexed queries started slowing down simply due to the volume of data being scanned.

Figma adopted range partitioning on time-based tables (document versions, event history, audit logs). Queries filtering by time “show me changes from the last 30 days” only scan the relevant partitions.

Operational benefit beyond query speed: older partitions can be archived or moved to cheaper storage without touching the hot partition that current queries hit.

What Any PostgreSQL Engineering Team Can Apply From Figma’s Architecture

Coalesce writes at the application layer. If your application generates events faster than you want to write to the database, batching is simpler to implement than it sounds and pays off immediately.

Not all reads need to be fresh. Categorizing your read queries by staleness tolerance is low-effort design work that pays off in reduced primary load.

Design for your connection pooler from the start. Retrofitting PgBouncer onto an application that uses session-level features is painful. If you’re not running a pooler yet, plan for transaction-pooling mode constraints before you need them.

📧 Want the complete PostgreSQL Health Report?

Includes MultiXact checks, vacuum monitoring, checkpoint diagnostics, WAL slot health, and 60+ diagnostic queries — all in one SQL file.

Get it here → $29

📌 Upgrade to Premium ($8/month) — Premium subscribers get the full SQL diagnostic packs, step-by-step playbooks, and the queries I actually run during a Sev-1.

Upgrade to Paid

References:

How Figma's Databases Team Lived to Tell the Scale | Figma Blog

How Figma’s multiplayer technology works - Made by Evan

The growing pains of database architecture | Figma Blog

Share

Subscribe now

Leave a comment

Table of Contents

1. Phase 1 Replication Slot Health (5 min)

2. Phase 2 Transaction ID Wraparound Risk (3 min)

3. Phase 3 Autovacuum Winning vs Running (10 min)

4. Phase 4 Lock Risk Audit (5 min)

5. Phase 5 Connection Layer Health (5 min)

6. Phase 6 WAL and Checkpoint Configuration (5 min)

7. Phase 7 Index Health and Bloat (10 min)

8. Phase 8 Schema Risk Unindexed Foreign Keys (5 min)

9. Phase 9 Security and Access Audit (5 min)

10. Phase 10 Monitoring Coverage Gaps (5 min)

Share this with one engineer on your team who owns a production PostgreSQL instance.

Share The Sev-1 Database

The Sev-1 Database Real incidents. Root causes. The queries I actually run. No theory.

Subscribe now

Performance troubleshooting checklist I published in February tells you what is broken right now. This one is different.

This is the checklist I run on a PostgreSQL instance before it breaks. The checks that catch replication slots hours before the disk fills. The checks that catch wraparound risk weeks before PostgreSQL stops accepting writes.

Most of these take under 60 seconds each. All of them have found real problems on production instances I’ve inherited or audited.

Run every phase.

Phase 1 Replication Slot Health (5 minutes)

Replication slots are the most common source of surprise disk-full incidents I’ve handled. An inactive slot retains WAL indefinitely with no cap by default. The disk fills over days, not hours. By the time the alert fires you have minutes.

Check 1.1 Slot inventory and retained WAL size

SELECT
  slot_name,
  slot_type,
  database,
  active,
  active_pid,
  pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
  ) AS retained_wal,
  pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
  ) AS unconfirmed_wal,
  CASE
    WHEN active = false
    THEN 'INACTIVE — retaining WAL indefinitely'
    WHEN pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) > 10737418240
    THEN 'CRITICAL — >10GB retained'
    WHEN pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) > 1073741824
    THEN 'WARNING — >1GB retained'
    ELSE 'OK'
  END AS status
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;

Flag: Any slot where active = false investigate immediately. Find the slot owner. Either restart the consumer or drop the slot.

Flag: retained_wal in gigabytes on any slot disk fill is a function of your write rate and time, not a fixed threshold.

All slots active and retained WAL under 500MB healthy.

Check 1.2 WAL retention cap

SELECT name, setting, unit
FROM pg_settings
WHERE name = 'max_slot_wal_keep_size';

Flag: setting = -1 no cap. A single inactive slot can fill your entire disk.

Set this immediately:

ALTER SYSTEM SET max_slot_wal_keep_size = '10GB'; then SELECT pg_reload_conf();

Specify the maximum size of WAL files that replication slots are allowed to retain in the pg_wal directory at checkpoint time. If max_slot_wal_keep_size is -1 (the default), replication slots may retain an unlimited amount of WAL files.

Source: PostgreSQL: Documentation: 18: 19.6. Replication

Check 1.3 Physical standby lag

SELECT
  application_name,
  client_addr,
  state,
  pg_size_pretty(
    pg_wal_lsn_diff(sent_lsn, replay_lsn)
  ) AS total_lag,
  pg_size_pretty(
    pg_wal_lsn_diff(write_lsn, flush_lsn)
  ) AS disk_write_lag,
  pg_size_pretty(
    pg_wal_lsn_diff(flush_lsn, replay_lsn)
  ) AS apply_lag,
  sync_state
FROM pg_stat_replication
ORDER BY pg_wal_lsn_diff(sent_lsn, replay_lsn) DESC;

Flag: total_lag above 100MB on a synchronous standby your primary commits are waiting for a slow standby.

Flag: No rows returned when you expect standbys standby has disconnected.

Read more

Table of Contents

1. Why Financial Writes Are a Different Problem

2. What DocDB Is In Stripe’s Own Words

3. The Core Architecture: Shards, Proxies, and Chunk Metadata

4. The Data Movement Platform: Zero-Downtime Migrations at Scale

5. Idempotency Keys: Exactly-Once Semantics at the API Layer

6. What the Idempotency Pattern Looks Like at the Database Layer

7. What Database Engineers Can Take From This

📨 If this was useful, share it with one engineer on your team who manages production databases.

Share The Sev-1 Database

The Sev-1 Database is a reader-supported publication. Each week I share practical takeaways from real incidents no theory, just what actually breaks and how to fix it.

Subscribe now

In 2023, Stripe processed $1 trillion in total payments volume.
Their database infrastructure a system they built themselves maintained 99.999% uptime throughout. This post covers what that infrastructure actually looks like, drawn directly from their engineering blog and API documentation.

Why Financial Writes Are a Different Problem

Most high-scale systems accept some degree of eventual consistency. Stripe cannot. A payment either happened or it didn’t. Duplicate charges are a catastrophic failure mode, not an edge case.

Two hard requirements shaped every architectural decision:

• Every write must be durable. A payment that appears to succeed but isn’t committed is a financial integrity failure.

• Every write must be idempotent. At scale, network retries are inevitable. The same request will arrive more than once. The system must handle this without creating duplicates.

Standard databases handle the first constraint. The second required a deliberate design decision at both the API and storage layers.

What DocDB Is In Stripe’s Own Words

From Stripe’s engineering blog (June 2024):

Stripe’s DocDB is an extension of MongoDB Community a popular open-source database and consists of a set of services that we built in-house. It serves over five million queries per second from Stripe’s product applications.

And on why they built it rather than using an existing service:

We chose to build DocDB on top of MongoDB Community because of the flexibility of its document model and its ability to handle massive volumes of real-time data at scale. MongoDB Atlas didn’t exist in 2011, so we built a self-managed cluster of MongoDB instances running in the cloud.

The key numbers from the same post:

• 5 million queries per second

• 10,000+ distinct query shapes

• Petabytes of financial data

• 5,000+ collections across 2,000+ database shards

This is not MySQL. It is not PostgreSQL. It is not MongoDB itself. It is a custom database-as-a-service layer built on top of MongoDB Community’s storage engine, with Stripe’s own proxy layer, replication system, and migration infrastructure on top.

🚀 PostgreSQL Health Report — One SQL file. 60+ diagnostic checks. Detects vacuum lag, bloat, lock chains, replication slot danger, unindexed FK columns, and more.

Get it here → $29

Core Architecture: Shards, Proxies, and Chunk Metadata

Stripe describes their architecture this way in the blog post:

Thousands of database shards, each housing a small chunk of the cumulative data, now underlie all of Stripe’s products. When an application sends a query to a database proxy server, it parses the query, routes it to one or more shards, combines the results from the shards, and returns them back to the application.

The routing layer works through a chunk metadata service:

They rely on a chunk metadata service that maps chunks to database shards, making it easy to look up the relevant shards for a given query.

Each physical database shard is deployed as a replica set primary node plus multiple secondary nodes with replication and automated failover. Product applications at Stripe never connect directly to database shards. They go through a fleet of database proxy servers that Stripe built internally in Go.

The proxy layer handles: reliability, scalability, admission control, and access control. It enforces query shape restrictions what Stripe calls a minimal set of database functions to avert self-inflicted issues due to suboptimal queries from client applications.

That last point is worth pausing on. Stripe’s proxy actively rejects queries that violate their access patterns. They do not give application teams open access to the database. This is an explicit architectural constraint, not just a convention.

Data Movement Platform: Zero-Downtime Migrations at Scale

The most technically detailed part of Stripe’s blog post covers their Data Movement Platform the system that allows them to split, merge, and migrate database shards with zero downtime.

From the post:

The Data Movement Platform enabled our transition from running a small number of database shards (each with tens of terabytes of data) to thousands of database shards (each with a fraction of the original data). It also provides client-transparent migrations with zero downtime, which makes it possible to build a highly elastic DBaaS offering.

In 2023 alone they used this to bin-pack underutilized databases:

From the post:

We bin-packed thousands of underutilized databases by migrating 1.5 petabytes of data transparent to product applications, and reduced the total number of underlying DocDB shards by approximately three quarters.

The traffic switch at the core of a shard migration takes under two seconds:

The entire traffic switch protocol takes less than two seconds to execute, and all failed reads and writes directed to the source shard succeed on retries.

Migration sequence has six steps: chunk registration, bulk data import, async replication from source to target, correctness check via point-in-time snapshots, traffic switch, and deregistration.

Correctness check compares snapshots rather than doing a live row-by-row comparison a deliberate choice to avoid impacting shard throughput during the verification step.

One specific optimization they mention that improved bulk load performance significantly:

By sorting the data based on the most common index attributes in the collections and inserting it in sorted order, we significantly enhanced the proximity of writes boosting write throughput by 10x.

Idempotency Keys: Exactly-Once Semantics at the API Layer

Second design pattern this post covers comes from a separate Stripe engineering post on idempotency (2017) and their official API documentation.

Stripe’s framing of the problem:

To overcome this sort of inherently unreliable environment, it’s important to design APIs and clients that will be robust in the event of failure, and will predictably bring a complex integration to a consistent state despite them.

Their definition of how idempotency keys work at the API layer, directly from their docs:

Stripe’s idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.

And on key generation:

A client generates an idempotency key, which is a unique key that the server uses to recognize subsequent retries of the same request. How you create unique keys is up to you, but we suggest using V4 UUIDs, or another random string with enough entropy to avoid collisions.

Key expiration, from their docs:

You can remove keys from the system automatically after they’re at least 24 hours old. We generate a new request if a key is reused after the original is pruned.

Their three-principle summary from the idempotency blog post:

Make sure that failures are handled consistently. Have clients retry operations against remote services. Make sure that failures are handled safely. Use idempotency and idempotency keys to allow clients to pass a unique value and retry requests as needed. Make sure that failures are handled responsibly. Use techniques like exponential backoff and random jitter.

What the Idempotency Pattern Looks Like at the Database Layer

Stripe’s blog and docs describe the pattern at the API level. The database implementation for PostgreSQL or MySQL that makes this work is a unique constraint on the idempotency key column. This is standard SQL not Stripe-specific but it’s the mechanism that enforces exactly-once semantics at the storage layer.

-- PostgreSQL: table design that supports idempotency
CREATE TABLE payments (
  id               BIGSERIAL PRIMARY KEY,
  idempotency_key  VARCHAR(255) NOT NULL,
  customer_id      BIGINT NOT NULL,
  amount_cents     BIGINT NOT NULL,
  status           TEXT NOT NULL DEFAULT 'pending',
  created_at       TIMESTAMPTZ DEFAULT now()
);

-- The constraint that enforces exactly-once semantics
CREATE UNIQUE INDEX idx_payments_idempotency_key
  ON payments (idempotency_key);

-- Application insert — fails cleanly on duplicate key
INSERT INTO payments (idempotency_key, customer_id, amount_cents, status)
VALUES ($1, $2, $3, 'pending')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING *;

-- If ON CONFLICT returns nothing:
-- → record already exists, fetch and return it
-- → no duplicate charge created

-- MySQL equivalent
INSERT INTO payments (idempotency_key, customer_id, amount_cents, status)
VALUES (?, ?, ?, 'pending')
ON DUPLICATE KEY UPDATE idempotency_key = idempotency_key;
-- affected rows = 0 means duplicate — application fetches existing record

This pattern converts at-least-once delivery at the network layer into exactly-once semantics at the database layer without distributed transactions, without application-level deduplication that fails under retry pressure.

What Database Engineers Can Take From This

Building a custom database is not the takeaway. That decision was made in 2011 when MongoDB Atlas didn’t exist and Stripe had specific requirements no off-the-shelf tool satisfied. It required years of infrastructure investment. It is not a pattern to copy.

The proxy layer pattern is underused. Stripe routes all application traffic through a proxy that enforces query shape, admission control, and access control. Most teams allow applications to connect directly to the database.

Result: suboptimal queries, missing indexes, and connection pool exhaustion incidents that a proxy layer would have caught earlier.

Design for shard migration before you need it. Stripe’s Data Movement Platform exists because they needed to split shards under load without downtime. If your data has a natural tenant dimension and you haven’t modeled it that way from the start, the eventual migration will be significantly more expensive than building for it early.

Idempotency keys at the database layer are underused. Any table that receives writes over a network is vulnerable to duplicate writes on retry. A unique constraint on an operation identifier is a one-time addition. It prevents an entire class of data integrity incidents. The Stripe docs describe this as a fundamental reliability primitive, not an advanced feature.

Key Takeaway

Stripe’s core storage layer is DocDB built on MongoDB Community in 2011 because MongoDB Atlas didn’t exist yet and no off-the-shelf DBaaS met their requirements.

It now serves 5M queries/second across 2,000+ shards. The two patterns with direct applicability to every database team: a proxy layer that enforces query constraints before they reach the database, and idempotency keys at the database layer on any table that receives writes over a network. Both are design decisions, not scale decisions.

📧 Want the complete PostgreSQL Health Report?

Includes MultiXact checks, vacuum monitoring, checkpoint diagnostics, WAL slot health, and 60+ diagnostic queries — all in one SQL file.

Get it here → $29

📌 Upgrade to Premium ($8/month) — Premium subscribers get the full SQL diagnostic packs, step-by-step playbooks, and the queries I actually run during a Sev-1.

Upgrade to Paid

Sources — everything in this post is drawn directly from:

→ stripe.com/blog — How Stripe’s document databases supported 99.999% uptime with zero-downtime data migrations (June 2024)

→ stripe.com/blog — Designing robust and predictable APIs with idempotency (2017)

→ docs.stripe.com — Idempotent requests (official API docs)

Share

Leave a comment

You’ve seen the free insights. Now imagine having full, unrestricted access to the Sev1 Database Premium Newsletter the same insights that serious database professionals rely on to stay ahead.

💡 Premium Membership :

• Monthly: $7.99/month

• Annual: $80/year not 96$ (early supporter rate)

This isn’t a trial. It’s the real deal, giving you:

• Exclusive database insights

• Premium-only content delivered

• RCA with actionable scrips + queries + explanation

• Ask me anything over the Email

Premium Content:

PREMIUM Content

Thousands of free readers rely on these insights. Now it’s your chance to step up to full access and stay ahead of the curve.

👉 Join today and unlock your premium membership.

Upgrade to PREMIUM

Full access is ready all you need to do is start.

See you inside,
Haider