A junior engineer tasked with building voice routing reaches for Pion (Go) or mediasoup (Node.js) and wires it to a Java service via gRPC. On paper, this works. In production at 50k concurrent voice participants, you hit the wall:

• mediasoup is single-threaded per worker. You spawn 32 workers. Ops screams about process management complexity.

• gRPC serialization overhead on every RTP packet (arriving at 50 packets/sec per publisher) becomes your dominant CPU cost.

• You have zero visibility into the JVM’s interaction with the native media layer.

The real problem: they treated RTP forwarding as someone else’s problem. An SFU is not complicated. It is a UDP packet router with subscriber state. You can build the core in pure Java NIO in an afternoon.

The Failure Mode: Thread-Per-Subscriber at Scale

The naive implementation looks like this:

// DO NOT DO THIS
ExecutorService pool = Executors.newCachedThreadPool();
while (true) {
    DatagramPacket pkt = receive();
    for (Subscriber s : room.subscribers()) {
        pool.submit(() -> s.send(pkt)); // new Runnable allocation per packet per subscriber
    }
}

At 1,000 subscribers receiving 50 RTP packets/sec from one publisher:

• 50,000 task submissions/sec into the thread pool

• Each submit() allocates a FutureTask wrapping a lambda capturing pkt and s

• Eden space fills in seconds; minor GC fires every 200ms

• During GC pause, the receive loop stalls — you drop RTP packets

• Dropped RTP = audio glitches. Users hear choppy voice.

The second failure is copying the payload. If you call pkt.getData() and copy the byte array for each subscriber, you are doing O(n) heap allocations per packet. At 1,000 subscribers, that is 1,000 × 1,400 bytes = 1.4MB of garbage per packet, 50 times per second = 70MB/sec of Eden pressure.

The Flux SFU Architecture

There are four key design decisions here.

Read more

A junior engineer building group video calls sees WebRTC as a peer-to-peer technology and concludes the server’s job is minimal — just exchange SDP offers and ICE candidates over the existing WebSocket connection, then get out of the way. In a two-person call, this works perfectly. The moment a third person joins, the architecture starts showing cracks.

The naive mental model: N participants form a mesh. Each peer connects directly to every other peer. The server stays stateless.

In a 6-person call, that’s N×(N-1) = 30 RTCPeerConnection instances across the group. Each participant encodes their camera stream once and uploads it 5 times — one copy per remote peer. On a mobile device with a 2 Mbps uplink at 720p, you’ve already saturated the connection before encoding overhead. At 10 participants, each person is maintaining 9 simultaneous DTLS-SRTP sessions, negotiating 9 separate ICE candidate exchanges, and your signaling server is managing O(N²) state transitions per room join or leave event.

This is the P2P Mesh failure mode. It doesn’t collapse under server load — it collapses under client load. Users on mobile drop out. Low-bandwidth users drag the entire call quality down. The “framework” hides this because WebRTC libraries abstract the peer connection count from you.

The Failure Modes: Mesh, then MCU

P2P Mesh at scale: Upload bandwidth grows linearly with room size. The client, not your server, is the bottleneck. A Discord-scale voice channel with 25 participants is physically impossible over mesh — no consumer device can sustain 24 simultaneous encoded upload streams.

The MCU “fix”: A Multipoint Control Unit receives all streams, decodes each one, mixes the audio and composites the video into a single output stream, then re-encodes and sends one stream to each participant. Upload problem solved. But now your server is running a real-time video encoder for every active room. At 10,000 concurrent 6-person rooms, you need GPU farms just for mixing. Latency spikes because decode → mix → encode adds 100–200ms minimum. This is how legacy enterprise conferencing (WebEx, early Zoom) was built, and it’s why it needed dedicated server infrastructure.

The Flux Architecture: The SFU

A Selective Forwarding Unit sits between the two extremes. Each participant uploads exactly one copy of their media stream to the SFU. The SFU receives RTP packets from each publisher and forwards them — without decoding or re-encoding — to each subscriber in the room.

Read more

This week we stop treating a scheduler like a stopwatch and start treating it like a living production system. A real scheduler does not just know when to run work. It must also know what to do when work gets stuck, when an upstream service starts behaving badly, and when the application itself is shutting down. That is the mindset behind this integrated project.

Agenda

We will build one control-room style application with five connected abilities:

1. capture exhausted work in a dead-letter queue for operator review,

2. protect remote calls with a circuit breaker,

3. move execution off the request thread into worker pools,

4. stop runaways with timeout cancellation,

5. finish cleanly during shutdown instead of abandoning state.

Think of it like an airport. The scheduler is the tower, the worker pool is the runway team, the circuit breaker is weather control, the timeout is the safety clock, and the dead-letter queue is the inspection bay for flights that could not depart safely.

Where it fits in a distributed scheduler

In a larger task scheduling platform, this module sits between task creation and task completion. Upstream systems create work through an API. The scheduler persists the task, dispatches it to async workers, watches elapsed time, and records the final outcome. If an external dependency becomes flaky, the circuit breaker stops the system from wasting threads on doomed calls. If retries still cannot recover the task, the dead-letter queue preserves the exact failure for manual analysis.

This is common in systems people already use every day: payment retries, report generation, notification delivery, and partner-data sync pipelines. The clever part is not just getting a success path working. The clever part is giving operators a clean answer when something unusual happens.

Component architecture

Read more

By the end of this lesson you will understand exactly what happens between the moment a user clicks “Join Voice” and the moment audio starts flowing. You will build a real STUN server in pure Java — not a wrapper around coturn, not a config file — an actual RFC 5389 binary protocol parser that handles XOR address encoding, lock-free counters, and zero-allocation UDP I/O. This is the foundation every voice calling system in the world is built on.

The Junior Engineer’s Trap

A junior engineer hears “STUN server” and reaches for a library. They find coturn, run apt install coturn, paste a config file into /etc/turnserver.conf, and declare the problem solved. The lesson ends before it begins.

The problem is that coturn is a black box. When your voice calls start failing under load because a NAT mapping expired mid-session, or when ICE candidates arrive with the wrong reflexive address because the STUN response was malformed, you have nowhere to look. You cannot debug a binary you did not write.

More to the point: if you are building a Discord-scale system, you will eventually run your own STUN infrastructure. At 100 million concurrent users, even a 0.1% ICE failure rate means 100,000 broken calls in the same instant. You need to own this protocol down to the byte.

The Failure Mode: Thread-Per-Datagram

The naive implementation looks like this. Open a DatagramSocket. Call receive() in a loop. Spawn a new thread for each packet. Build a response. Send it. This works for ten users in a demo.

At 50,000 binding requests per second — a modest voice call storm during a large event — you have 50,000 DatagramPacket allocations per second landing in Eden space. Each DatagramPacket wraps a byte[]. Fifty thousand arrays per second, each between 20 and 576 bytes, fills Eden in milliseconds. Minor GC fires every 200ms. Each minor GC pauses all threads for 5 to 15ms. During that pause, UDP datagrams queue inside the kernel socket buffer. When GC finishes, you process a burst. Latency spikes. ICE checks time out. Calls drop.

The OS makes it worse. UDP socket receive buffers default to 208 KB on Linux. At 50,000 datagrams per second, you overflow that buffer in under 5ms during a single GC pause. The kernel silently drops the overflow. Your ICE negotiation never completes, and you get no error — just silence.

The thread model fails independently of the GC problem. Each thread context-switch costs between 1 and 5 microseconds. With 1,000 threads live, you are burning CPU cycles on the OS scheduler before you parse a single byte of STUN payload.

The Flux Architecture: Single-Threaded Selector Loop

STUN over UDP maps perfectly to the Reactor pattern. One thread. One DatagramChannel in non-blocking mode. One Selector. No allocation per datagram — we reuse a single ByteBuffer for receives and a separate ByteBuffer for responses.

DatagramChannel (non-blocking)
    → Selector.select()
    → recvBuf.clear(); channel.receive(recvBuf)
    → recvBuf.flip(); StunMessage.parse()
    → StunResponseBuilder.build(msg, sendBuf)
    → sendBuf.flip(); channel.send(sendBuf, sender)
    → back to select()

The STUN RFC 5389 binary format is simple enough to parse entirely with ByteBuffer.get() calls and zero heap allocation beyond the StunMessage record. After escape analysis the JIT will stack-allocate that record in the hot path. We use pre-allocated direct ByteBuffer instances for both read and write, so UDP payloads never touch the heap at all.

The dashboard runs on a Virtual Thread — blocking HTTP is perfectly acceptable there. The critical path (the selector loop) stays single-threaded and allocation-free.

Implementation Deep Dive

GitHub Link :

https://github.com/sysdr/discord-flux-p/tree/main/day65/flux-stun

The STUN Message Format (RFC 5389, Section 6)

Every STUN message starts with a 20-byte header structured exactly like this:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0|     STUN Message Type     |         Message Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Magic Cookie (0x2112A442)                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                     Transaction ID (96 bits)                  |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The Magic Cookie (0x2112A442) is your first validation gate. If it is missing, drop the datagram without sending a response. No point processing garbage.

The top two bits of the message type field must always be zero. If they are not, the packet is not STUN.

Message type 0x0001 = Binding Request. You respond with 0x0101 = Binding Success Response.

The 12-byte Transaction ID is how the client matches your response to the request it sent. You copy it verbatim into the response header.

XOR-MAPPED-ADDRESS Construction

The reflexive address — the client’s public IP as seen by your server — must be XOR’d against the magic cookie before you send it back. This is not optional. RFC 5389 requires it specifically to defeat Application Layer Gateways: NAT devices that scan UDP payloads looking for IP addresses and rewrite them. XOR encoding makes the address unrecognizable to those devices.

For an IPv4 client the math is:

XOR'd port = client_port    XOR (0x2112A442 >> 16)   // XOR against high 16 bits of magic cookie
XOR'd IP   = client_ip_int  XOR 0x2112A442            // XOR against full magic cookie

The ICE agent on the other end knows the magic cookie and reverses the XOR to recover the real address.

The Zero-Allocation Pattern

Pre-allocate both buffers once at startup before the selector loop begins:

ByteBuffer recvBuf = ByteBuffer.allocateDirect(576); // max STUN payload
ByteBuffer sendBuf = ByteBuffer.allocateDirect(576);

allocateDirect puts these buffers in native memory — completely off the Java heap, invisible to the garbage collector. Inside the selector loop:

recvBuf.clear();                                      // reset position/limit, no allocation
InetSocketAddress sender = (InetSocketAddress) channel.receive(recvBuf);
recvBuf.flip();                                       // prepare for reading
StunMessage msg = StunMessage.parse(recvBuf, sender); // reads directly from buffer

Nothing in this path creates a new object on the heap. The StunMessage record is the only candidate, and after a few thousand iterations the JIT’s escape analysis confirms it never leaves the method — it gets stack-allocated and costs nothing.

VarHandle for Lock-Free Stats

Instead of AtomicLong, which boxes every access through an object header, we use VarHandle on a long[]:

private final long[] counters = new long[4]; // requests, responses, errors, invalid
private static final VarHandle VH =
    MethodHandles.arrayElementVarHandle(long[].class);

// In the hot path:
VH.getAndAdd(counters, IDX_REQUESTS, 1L);

This is a single CPU instruction (lock xadd on x86) with no object allocation and no lock overhead.

Working Demo Link :

Production Readiness

These are the numbers you instrument and alert on before you ship a STUN server:

Metric How to Measure Alert Threshold Binding requests per second JMX / Prometheus counter Alert above 80K/sec per instance UDP receive buffer drops netstat -su or /proc/net/udp Any non-zero value is a capacity event GC allocation rate on selector thread VisualVM Profiler Under 1 MB/sec — should be near zero STUN error rate Internal StunStats counter Above 0.1% triggers investigation Response latency P99 Histogram in dashboard Under 5ms

On Linux, tune the kernel socket buffers before starting the server:

sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.rmem_default=26214400

25 MB of receive buffer means you can absorb roughly 40,000 full-size STUN datagrams during a 5ms GC pause without dropping a single packet.

Build, Run, and Verify

Prerequisites

Make sure these are installed before you start:

• JDK 21 or later — run java -version to confirm

• Maven 3.9 or later — run mvn -version to confirm

• Python 3 (for the verify script’s packet parsing step)

• VisualVM — download from visualvm.github.io, used to confirm zero allocation on the selector thread

• netcat (nc) — optional, for raw UDP testing

Step 1 — Generate the Workspace

Build and Start

./start.sh

This compiles the project with Maven and starts two things: the STUN server on UDP port 3478, and the dashboard web server on TCP port 9090.

Expected output:

Building flux-stun...
STUN  server: UDP :3478
Dashboard:    http://localhost:9090
[stun-selector] INFO: STUN server listening on UDP :3478

Step 2 — Open the Dashboard

Navigate to http://localhost:9090 in your browser.

You should see the live stats panel with counters for total requests, responses sent, invalid packets, unique source IPs, and requests per second. All counters start at zero.

Step 3 — Run the Unit Tests

Open a second terminal in the flux-stun directory and run:

mvn test

All three test classes run: StunMessageTest, StunResponseBuilderTest, and StunIntegrationTest. The integration test starts its own STUN server instance on port 13478 (a non-privileged test port), fires single and concurrent requests against it, and verifies the response counters match exactly.

Expected output:

[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 -- StunMessageTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 -- StunResponseBuilderTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 -- StunIntegrationTest
[INFO] BUILD SUCCESS

Step 4 — Run the Demo

./demo.sh

This fires 1,000 STUN Binding Requests from virtual threads against your running server, then uses a Python script to send one additional request manually and parse the raw XOR-MAPPED-ADDRESS attribute from the binary response.

You will see output like:

=== Flux STUN Demo: NAT Traversal ===
Firing 1000 STUN Binding Requests via Virtual Threads...
Storm sent. Check dashboard: http://localhost:9090

Sending a single test request with netcat (hex):
  Response type: 0x0101 (Binding Success)
  Response length: 40 bytes
  Hex: 0101001421...

Switch to the dashboard tab and observe the counters update.

Step 5 — Verify Responses and XOR Decoding

./verify.sh

This script does four things in order:

1. Confirms the server process is running via the PID file

2. Checks that UDP port 3478 is listed as bound using ss -ulnp

3. Sends a STUN Binding Request with a known transaction ID and parses the binary response, printing the decoded XOR-MAPPED-ADDRESS

4. Hits the dashboard API endpoint and prints the current JSON stats

Expected output:

[OK] STUN server running (PID 12345)
[OK] UDP :3478 is open
[OK] Binding Response received
     Type:           0x0101
     Magic Cookie:   0x2112A442
     Transaction ID: 000102030405060708090a0b
[OK] XOR-MAPPED-ADDRESS: 127.0.0.1:XXXXX
     (This is your reflexive/public address as seen by the STUN server)
[OK] Dashboard API responding (HTTP 200)
     Stats: {"requests":1002,"responses":1002,"errors":0,"invalid":0,"uniqueIps":1,"reqPerSec":"..."}
[OK] All verification checks passed.

The reflexive address showing 127.0.0.1 is correct here because the verify script runs on localhost. In a real deployment, a client behind a home router would see their public IP.

Stopping and Cleaning Up

# Stop the running server process only:
./stop.sh

# Remove build artifacts (does not kill the server):
./cleanup.sh

Homework Challenges

Beginner — Add a SOFTWARE attribute (attribute type 0x8022) to every Binding Response. The value is a UTF-8 string containing your server name and version, for example "Flux-STUN/1.0". Pad the attribute value to a 4-byte boundary as required by RFC 5389. Verify the attribute appears in the response using verify.sh and the hex dump it prints. (The existing StunResponseBuilder already includes this — read through the implementation and make sure you understand each byte it writes.)

Intermediate — Implement MESSAGE-INTEGRITY validation (RFC 5389, Section 15.4). This is HMAC-SHA1 over the message content using a shared password. Add a StunCredential record to carry the username and password. Add a validation step in StunServer.handleDatagrams() that checks the HMAC before building a response. Update StunLoadClient to include the USERNAME and MESSAGE-INTEGRITY attributes in outgoing requests. Write a test that confirms a request with a wrong password gets an error response (0x0111) with an ERROR-CODE attribute.

Expert — Add the FINGERPRINT attribute (RFC 5389, Section 15.5). This is a CRC-32 checksum of the entire message XOR’d with 0x5354554e. Run ./demo.sh firing 50,000 requests and use /proc/net/udp or netstat -su to count UDP receive buffer drops. Tune SO_RCVBUF from the default up to 25 MB in StunServer.java and measure the minimum buffer size that produces zero drops on your hardware at that load.

Next: Day 66 — TURN Relay: When STUN Is Not Enough

Gmail still delivers your mail when a SMTP relay blips. Stripe still settles payments after a network timeout. Your bank’s nocturnal batch does not “forget” mid-run because one pod restarted. They share a boring superpower: every attempt is recorded, every status change is legal, every failure is classified, and retries are bounded with backoff.

A scheduler without a lifecycle is a stopwatch with amnesia. At scale, amnesia is an outage you cannot explain.

Agenda

1. Execution ledger (who ran what, when, with what outcome)

2. Status state machine (illegal transitions rejected)

3. Error capture (message + stack persisted)

4. Durable exponential backoff (RETRYING + nextRetryAt)

5. Spring Retry for in-process transient slices

6. Dead letter when hope is gone

7. Build, test, demo

Core Concepts

1. Execution Ledger

Before you optimize throughput, you need a flight recorder. Each attempt gets a UUID executionId, start/end timestamps, duration, node id, and either an outcome or an error+stack. Netflix-style ops dashboards are useless without this row-level truth.

Insight: Metrics tell you rates; the ledger tells you stories. When a VP asks “what happened to job X at 02:14?”, you open one row—not a haystack of logs.

Read more

A junior engineer discovers spring-webrtc or reaches for a JSON library to relay SDP blobs. The code looks like this:

@MessageMapping("/sdp-offer")
public void handleOffer(String sdpJson, SimpMessageHeaderAccessor headers) {
    String sessionId = headers.getSessionId();
    sdpStore.put(sessionId, sdpJson);
    messagingTemplate.convertAndSendToUser(targetUser, "/queue/sdp", sdpJson);
}

At 10 users in a demo this is fine. At 10,000 concurrent voice calls, you have a problem you cannot see until production explodes.

Spring’s SimpMessageHeaderAccessor wraps every message in a Map<String, Object> of header copies. Each SDP blob is 2–4 KB of ASCII. Each JSON serialization produces intermediate String and byte[] objects. GC pressure begins to spike before you hit 500 concurrent sessions. More critically: the session correlation model (sessionId → targetId) lives in a HashMap behind a @Service singleton. The first time two threads race on a state transition — offer arriving while an answer is already in flight — you get a corrupted session with no error, just a silent WebRTC failure on the client.

The Failure Mode

SDP state is a two-party handshake with strict ordering:

Caller → OFFER → Server → forward to Callee
Callee → ANSWER → Server → forward to Caller
Both   → ICE CANDIDATES (trickle) → Server → cross-forward

Three things break at scale with naive implementations:

1. String Interning and Heap Retention

SDP contains repeated attribute prefixes (a=, m=, c=). If you parse into Map<String, String> and retain these maps per session, the String pool bloats. With 1 million sessions at 3 KB each, retained heap from SDP strings alone exceeds 3 GB. The JVM’s G1 collector begins spending 40%+ of wall time reclaiming this.

2. Unsynchronized State Transitions

The session lifecycle is: IDLE → OFFERING → ANSWERING → ESTABLISHED. If a ConcurrentHashMap.put() isn’t paired with a CAS on the state field, two simultaneous answers for the same session both “succeed” and one is silently dropped.

3. Trickle ICE Fan-Out

After SDP exchange, each peer sends 3–10 ICE candidates. These are small messages but they arrive in a burst. A thread-per-connection model spawns hundreds of threads just for ICE candidate forwarding. Under Virtual Threads this is invisible, but if you’re on platform threads (the Spring default until explicitly configured otherwise), you hit OS thread limits fast.

The Flux Architecture

The Flux SDP Exchange system has three components.

SdpSession — Sealed State Machine

A sealed interface with record variants for each lifecycle state. State transitions use VarHandle CAS to prevent races without synchronized blocks.

No inheritance, no nullable fields, no instanceof chains. Pattern matching in switch expressions handles dispatch cleanly.

SdpParser — Zero-Copy Field Extraction

Rather than parsing SDP into a full object graph, we scan the raw bytes and extract only the fields we need: o= (origin), a=fingerprint, a=ice-ufrag, a=ice-pwd, and m= lines. We read from a ByteBuffer line by line using index arithmetic, never constructing intermediate substrings for lines we discard.

SessionStore — Lock-Free CAS Transitions

ConcurrentHashMap<String, SdpSession> with VarHandle-based compare-and-swap for state transitions. The transition OFFERING → ANSWERING is only applied if the current state is Offering, preventing double-answer races.

Implementation Deep Dive

GitHub LinK :

https://github.com/sysdr/discord-flux-p/tree/main/day64/flux-day64

Selective Field Extraction from SDP

SDP is line-delimited. Each line starts with a single letter and =. We only care about a handful of them. The parser reads raw bytes from the WebSocket payload buffer:

record ParsedSdp(String iceUfrag, String icePwd, String fingerprint,
                 String origin, List<String> mediaLines) {

    static ParsedSdp parse(String raw) {
        String iceUfrag = null, icePwd = null, fingerprint = null, origin = null;
        var mediaLines = new ArrayList<String>();
        for (String line : raw.split("\r?\n")) {
            if (line.startsWith("a=ice-ufrag:"))      iceUfrag    = line.substring(12);
            else if (line.startsWith("a=ice-pwd:"))   icePwd      = line.substring(10);
            else if (line.startsWith("a=fingerprint:")) fingerprint = line.substring(14);
            else if (line.startsWith("o="))            origin      = line.substring(2);
            else if (line.startsWith("m="))            mediaLines.add(line.substring(2));
        }
        return new ParsedSdp(iceUfrag, icePwd, fingerprint, origin,
                             Collections.unmodifiableList(mediaLines));
    }
}

This allocates one ParsedSdp record per session instead of a full Map<String, String> with dozens of entries. Memory footprint drops by roughly 70% versus a generic SDP object model.

VarHandle CAS for State Transitions

private static final VarHandle SESSION;
static {
    try {
        SESSION = MethodHandles.lookup()
            .findVarHandle(SdpExchangeHandler.class, "session", SdpSession.class);
    } catch (Exception e) { throw new ExceptionInInitializerError(e); }
}

// Atomic transition: IDLE → OFFERING
boolean registerOffer(ParsedSdp offer, String callerId) {
    var expected = new SdpSession.Idle(callId);
    var next = new SdpSession.Offering(callId, callerId, offer, System.currentTimeMillis());
    return SESSION.compareAndSet(this, expected, next);
}

If two callers race to set an offer for the same callId, exactly one wins the CAS. The loser gets false and can signal an error immediately instead of silently overwriting.

Virtual Thread Per Connection

The WebSocket server dispatches each connection to a Virtual Thread. ICE candidates arrive as additional messages on the same connection. Because Virtual Threads are cheap (~1 KB stack vs ~512 KB for platform threads), a burst of 10,000 trickle ICE messages spawns 10,000 Virtual Threads without exhausting the OS.

var executor = Executors.newVirtualThreadPerTaskExecutor();
serverSocket.accept(null, new CompletionHandler<>() {
    public void completed(AsynchronousSocketChannel ch, Void a) {
        executor.submit(() -> handleConnection(ch));
        serverSocket.accept(null, this); // re-arm
    }
});

Working Demo Link :

Production Readiness

Monitor sdp.session.failed_total broken down by failure reason. A spike in ANSWER_TIMEOUT indicates network issues. A spike in CAS_REJECTED means clients are retrying offers — usually a client-side bug or duplicate connection.

Prerequisites

Before running anything, confirm these tools are installed:

wscat (optional) lets you send raw WebSocket messages manually during exploration. Install it with npm install -g wscat if you want to poke the server by hand.

Build and Run

Step 1 — Generate the workspace

chmod +x project_setup.sh
./project_setup.sh
cd flux-day64

This creates the full Maven project tree with all Java source files and lifecycle scripts.

Step 2 — Start the server

./start.sh

start.sh compiles the project with Maven, packages it into a JAR, and starts the server in the background. It writes the server process ID to .server.pid.

==> Building flux-day64...
==> Starting SDP Gateway (WS :8080, Dashboard :8081)...
==> Server PID 84312
==> Dashboard: http://localhost:8081/dashboard

Two ports will be open:

• ws://localhost:8080 — WebSocket signaling endpoint

• http://localhost:8081/dashboard — live session dashboard

Running the Tests

The unit test suite covers SDP parsing, all state transitions, the fail() spin-CAS loop, and a concurrent race test that verifies exactly one offer wins when 10 Virtual Threads compete simultaneously.

mvn test

Expected output:

[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

Running the Demo

./demo.sh

demo.sh runs LoadTestClient with 20 concurrent call pairs. Each simulated pair goes through the full handshake: OFFER → ANSWER → 3 ICE candidates → ESTABLISH. All 20 pairs run concurrently via Virtual Threads.

Watch the dashboard at http://localhost:8081/dashboard while the demo runs. You will see sessions appear in OFFERING state, transition through ANSWERING, and land on ESTABLISHED within a few hundred milliseconds.

Console output:

[Load] Starting 20 call pairs on localhost:8080
[Load] Done. Success=20 Failed=0
==> Done. Check dashboard: http://localhost:8081/dashboard

Verification

./verify.sh

verify.sh calls /api/sessions via curl and checks two things:

==> Verifying session state via API...
    Sessions established : 20
    CAS collisions       : 0
[PASS] At least one session reached ESTABLISHED
[PASS] No CAS collisions (clean state transitions)

Stopping and Cleaning Up

To stop the running server without removing build artifacts:

./stop.sh

==> Stopped server PID 84312

To remove build artifacts and log files (does not kill the server):

./cleanup.sh

==> Cleaning build artifacts...
==> Done.

When you want a completely clean slate, run stop.sh first, then cleanup.sh.

Homework

Beginner: Add a RINGING state between IDLE and OFFERING. Implement a 30-second ring timeout that transitions the session to FAILED with reason NO_ANSWER.

Intermediate: Replace the String.split("\r?\n") SDP parser with a ByteBuffer-based line scanner that avoids creating a String[] array entirely. Benchmark both with JMH and compare allocation rates.

Expert: Implement SDP munging: before forwarding an OFFER to the callee, rewrite the c= (connection) line to point to your media relay server IP. This is what production TURN servers do to NAT-punch on behalf of clients behind symmetric NAT.

Voice state is not a chat message. It is presence metadata that must be:

• Stored atomically

• Fanned out only to guild channel members

• Used downstream to bootstrap WebRTC signaling

You need opcode-level dispatch. You need lock-free state storage. You need to own the entire path.

The Failure Mode

Naive implementation: a HashMap<String, VoiceState> guarded by synchronized.

At 1M users joining voice channels simultaneously (think: a major game launch, a concert goes live), every VOICE_STATE_UPDATE (Opcode 4) acquires that lock. With average hold time of 200ns and 1M contenders, you are looking at 200 seconds of cumulative wait time per second of wall clock. Throughput collapses.

Second failure: storing VoiceState as mutable POJOs means you cannot safely read state mid-update without copying. Readers and writers race. You get torn reads — a user appears both in channel A and channel B simultaneously.

Third failure: fan-out. If you fan out to all guild members on the synchronized write path, you have coupled IO latency (slow clients) to your state update critical section. One slow WebSocket write blocks every other voice state update for that guild.

The Flux Architecture

Read more

When Uber dispatches surge-pricing jobs, when Netflix refreshes CDN caches, or when your bank runs end-of-day reconciliation, exactly one scheduler instance must decide what runs and when. Run the same cron on every pod and you double-charge customers. Run none and reports never ship.

The fix is not “turn off scheduling on followers.” You need an explicit leader election contract, lease-based heartbeats, automatic failover, coordination primitives, and idempotent execution so retries never corrupt state.

What We Built

Core Concepts

1. Single-Active Scheduler Pattern

In a horizontally scaled cluster, one node holds a lease in shared storage. The lease has an expiry timestamp. The leader renews it on a heartbeat interval (typically one-third of lease duration). Followers observe an unexpired lease owned by someone else and stay idle.

This is how Google’s Chubby-inspired systems, Kubernetes lease objects, and JDBC-backed Spring schedulers keep one brain in charge.

2. Database-Backed Leases

PostgreSQL stores a cluster_leadership row per service. Acquisition uses optimistic versioning plus conditional updates:

• Renew: update only if you still own a non-expired lease

• Takeover: update only if the lease is expired

• Failover: force-expire a stale leader after health checks fail

Read more

A junior engineer tasked with “adding voice calling” reaches for a WebRTC library. They drop in webrtc4j or a JavaScript SDK, annotate a controller with @MessageMapping, and call it done. The signaling layer — the server-side component that brokers SDP offers, answers, and ICE candidates between two peers — becomes a thin relay: receive JSON blob, forward JSON blob. It works in a demo with two browser tabs.

The trap is what’s hidden: that JSON blob is a full SDP offer. It’s 2–5 KB of plaintext describing codec preferences, DTLS fingerprints, RTP payload types, and extension headers. At 100K concurrent call setups, a naive implementation is doing 500 MB/s of String allocation just for SDP parsing. Jackson’s ObjectMapper deserializes each offer into a full object graph — dozens of nested nodes, each a heap allocation, each a GC liability.

Worse: the framework hides the session lifecycle entirely. When a @MessageMapping handler loses its WebSocket connection mid-negotiation, what happens to the in-flight SDP exchange? With Spring’s abstraction, the answer is: you don’t know. The session state evaporates, and neither peer gets a clean error. At scale, that means orphaned ICE candidate buffers accumulating in memory with no cleanup path.

The Failure Mode

Eden Space Exhaustion from SDP Churn

SDP is text-heavy by design. A typical offer contains 40–80 lines, each a String.split()-able field. In Java, String.split() compiles a Pattern, allocates a String[], then allocates each substring. For a 60-line SDP parsed with split("\r\n") followed by per-line split("=", 2), you’re looking at 180+ allocations per offer — all short-lived, all Eden-bound.

At 10K calls/second (a modest mid-size platform), that’s 1.8M allocations per second purely from SDP parsing. Minor GC pauses hit every few hundred milliseconds. At 100K calls/second, you’re triggering GC so frequently that throughput collapses before you’ve forwarded a single RTP packet.

ICE Candidate Trickle Storm

ICE trickle means candidates arrive asynchronously — a caller may send 5–15 candidate: lines over several seconds. A naive session store using HashMap<String, List<IceCandidate>> with boxed keys allocates constantly. More critically, if the session state machine is unguarded, a late-arriving candidate for an already-closed session has no clean rejection path — it inserts into a map for a session that’s been garbage collected, holding the map entry live indefinitely.

The Reconnect Signaling Storm

When a server restarts or a region fails over, thousands of clients reconnect simultaneously and immediately attempt to re-establish calls. Every one of them sends a new SDP offer. If your signaling router processes these synchronously on a thread pool, you exhaust the pool. Queued sessions time out before being processed, generating a cascade of retries that amplifies the load 3–5×.

The Flux Architecture

Read more

A junior engineer wiring up voice chat reaches for Spring Boot’s @EnableWebSocket and wraps everything in TCP. It works on their laptop with two friends. Ship it.

Then reality hits: voice audio is 20ms Opus frames arriving at 50 packets/second per user. At 10,000 concurrent speakers that’s 500,000 packets/sec. TCP’s retransmit-and-wait semantic — built for reliability — becomes your enemy. A single dropped packet stalls the receive buffer until the OS retransmits. You hear a half-second freeze. Every. Single. Time.

WebRTC doesn’t use TCP for media. It uses UDP. Not because UDP is “faster” in some hand-wavy sense, but because for real-time media a stale packet is worse than no packet. If an Opus frame is 20ms late, you don’t want it. You want to interpolate silence and move on.

Spring Boot hides this. It gives you a nice WebSocketHandler, a TextMessage, and zero control over the transport layer beneath it. You cannot drop into raw UDP. You cannot tune socket receive buffers. You cannot implement DTLS yourself. You are building audio on top of a framework that was never designed for it.

The Failure Mode

Here’s what a naive UDP server looks like in Java before this lesson:

DatagramSocket socket = new DatagramSocket(5005);
byte[] buf = new byte[1500];
DatagramPacket packet = new DatagramPacket(buf, buf.length);
while (true) {
    socket.receive(packet); // BLOCKS the calling thread
    socket.send(packet);
}

One thread. Blocking receive. This hits the OS kernel’s receive buffer at ~50K packets/sec on a modern NIC. Anything beyond that gets silently dropped — no exception, no log. It just disappears.

Scale this: spawn a thread per client. 10,000 clients = 10,000 OS threads. At 1MB stack each, that’s 10GB of virtual address space before you’ve processed a single byte of audio. The OS scheduler starts spending more time context-switching than running your code.

The second failure: new byte[1500] inside the receive loop. Each UDP packet allocation goes to the JVM Eden space. At 500K packets/sec that’s 750MB/sec of allocation pressure. The GC fires a minor collection every few hundred milliseconds. During that pause — which lasts 5–50ms on a busy heap — your receive loop stops. The OS kernel’s socket receive buffer fills up. UDP packets are dropped. Users hear glitches.

You cannot afford GC pauses in a media path.

The Flux Architecture

Read more

You already have a catalog of schedules and a binder that turns rows into ScheduledFutures. That is a control plane. Horizontal scale breaks it the moment two JVMs share the same catalog: each node still wakes on the same cron, and both happily “reconcile invoices” at 02:00. Payment platforms call that double-settlement. Warehouse systems call it double-ship. Schedulers call it the reason locks exist.

Airflow’s executors, Kubernetes leader election for controllers, and Redis-backed queue claim patterns all solve variants of the same question: among N healthy processes, which one may enter this critical section right now? integrated lab — Aegis — makes that question tangible. You will watch unprotected ticks collide, then enforce exclusion with database leases, optimistic claims, Redis SET NX, and a Redlock quorum — one dashboard, no toy mocks.

The production insight that separates junior from senior designs is not “use Redis.” It is knowing which failure mode your lock actually covers. A Postgres SELECT FOR UPDATE and a Redis key with TTL fail differently under network partitions. Product managers feel those differences as duplicate emails; SREs feel them as pager storms; architects must name them before choosing a store.

Agenda

1. The single-instance illusion after scale-out

2. Where locks sit in the Task Scheduler architecture

3. Control flow: tick → acquire → run → release

4. Data flow across RDBMS and Redis

5. Lock and task state machines

6. Four real coordination strategies (plus the “none” baseline)

7. Build, test, and demo Aegis

8. Production trade-offs and homework

The Problem Being Solved

Static @Scheduled methods are process-local. Persist the schedule in a database and every replica will still fire. Without a distributed lock, your audit table fills with two EXECUTED rows for the same logical tick — different instanceId, same business effect.

That is not a logging bug. It is a missing mutual-exclusion boundary around work that is not idempotent. Even idempotent handlers benefit from locks: you save CPU, cache stampedes, and load on downstream banks or SaaS APIs that rate-limit by account.

Why It Matters in Distributed Systems

Read more

A junior engineer wires up Spring Boot Actuator, adds spring-boot-starter-actuator to the POM, and calls it done. They get /actuator/metrics/jvm.memory.used out of the box. Beautiful, until they look at what that costs.

Spring Actuator’s default MeterRegistry uses AtomicLong for every counter. Under load — 50,000 connection events per second — every increment is a compare-and-swap on a shared cache line. The CPU burns cycles retrying failed CAS operations as threads collide. At 10 threads that’s noise. At 500 Virtual Threads funneling connection events, that’s measurable throughput degradation. You’ve added observability that slows down the thing you’re observing.

The second trap: metric collection tied to the request path. If scraping your /metrics endpoint blocks on the same lock that your connection handler uses to update state, you’ve just made Prometheus scrapes into a latency spike for every client connected at that moment.

The third trap: string formatting. Spring formats metric names and labels on every read. String.format("gateway_events{server='%s'}", hostname) in a tight loop allocates heap on every invocation. At 1M events/minute, you’re feeding the GC constantly.

The Failure Mode

The naive implementation collapses under two forces simultaneously.

Cache line contention: An AtomicInteger is 16 bytes. Under JVM object layout, a hot counter sits on a 64-byte CPU cache line. When 500 Virtual Threads all increment it, the L3 cache must arbitrate ownership of that cache line across CPU cores. Intel MESIF protocol means each increment — fetch, modify, write-back — stalls other cores. You get cache line bouncing, visible as elevated LLC-load-misses in perf stat output. Throughput flatlines at ~2M increments/second where the unconstrained math says 100M+.

Eden space pressure from metric labels: Prometheus exposition format requires label strings. Generating gateway_events_total{shard="3",type="MESSAGE"}\n on every scrape via naive concatenation creates one String object per metric per scrape interval. At 200 metrics, scraped every 15 seconds, that’s manageable. But if you push metrics instead of pull — formatting on every event — you’ve turned your metrics layer into an allocation firehose.

The rate calculation trap: Storing raw event timestamps to calculate events-per-second means ArrayList<Long> growing unbounded. At 10,000 events/second, you’re storing 600,000 longs per minute before any GC has a chance to clean up. The heap inflates. Stop-the-world pauses start showing up at the worst moments — during traffic spikes, precisely when you need accurate metrics most.

The Flux Architecture

Three principles drive the observability layer:

Wait-free counters via LongAdder: Java’s LongAdder uses Striped64 internals — it maintains an array of Cell objects, each pinned to a thread group via a hash of the thread ID. Increments write to thread-local cells. Reads sum all cells. No CAS contention on increments because threads rarely collide. The tradeoff is that reads are slightly stale — the sum is eventually consistent, not instantaneous. For Prometheus metrics scraped every 15 seconds, this is irrelevant.

Ring buffer for rate calculation: Events Per Second is computed over a sliding window. Instead of storing every timestamp, a fixed-size ring buffer of slot counters tracks events per 100ms bucket. EPS = sum of last 10 buckets / 1 second. The ring buffer is pre-allocated at startup — no heap allocation on the hot path, ever.

Pre-formatted label strings: Metric names and labels are computed once at registration and cached as byte[]. The Prometheus exposition format writer copies pre-built byte arrays into the response buffer. Zero String allocation during scrape.

Implementation Deep Dive

GitHub Link :

https://github.com/sysdr/discord-flux-p/tree/main/day60/flux-gateway-observability

LongAdder as gauge delta: Connected client count is a gauge, not a counter. On connect, call connectedClients.increment(). On disconnect, call connectedClients.decrement(). The longValue() at scrape time gives the current snapshot. The JVM’s Striped64 implementation handles the summation across cells under the hood.

EventRateCalculator ring buffer: The calculator maintains a LongAdder[] buckets array of size 100 (100 buckets × 100ms = 10 seconds of history). A currentBucket index advances every 100ms via a ScheduledExecutorService using a Virtual Thread factory. On advance, the old bucket is zeroed. EPS is sum(buckets) / 10.0. This is O(1) on the write path — just buckets[currentBucket].increment() — with no synchronization needed on the hot path.

VarHandle for atomic bucket index: The currentBucket pointer uses VarHandle.getAndAdd() with modular arithmetic. This is cheaper than AtomicInteger for this access pattern because we only need release/acquire semantics, not full sequential consistency.

Prometheus text format: The exposition format is dead simple:

# HELP gateway_connected_clients Current number of connected WebSocket clients
# TYPE gateway_connected_clients gauge
gateway_connected_clients 4821
# HELP gateway_events_per_second Event processing rate
# TYPE gateway_events_per_second gauge
gateway_events_per_second 12847.30

Pre-build the # HELP and # TYPE lines as byte[] at startup. At scrape time, concatenate with the current value formatted via Long.toString() — the only allocation.

Virtual Thread HTTP server: The metrics HTTP server uses com.sun.net.httpserver.HttpServer with a Virtual Thread executor. Each Prometheus scrape gets its own Virtual Thread. Since scrapes are I/O bound (write response, flush socket), Virtual Threads park during the write and yield the carrier. No blocking, no wasted platform threads.

Class Reference

Working Demo Link :

Production Readiness

The carrier thread count is the canary. If Virtual Threads are parking correctly on I/O, you should see exactly N platform threads (where N = available CPUs) doing real work. If the count grows, something is blocking a carrier — a synchronized block, a native call, or a pinned Virtual Thread.

Key jcmd commands:

# Thread dump to catch pinned virtual threads
jcmd <pid> Thread.dump_to_file -format=json /tmp/threads.json

# GC stats
jcmd <pid> GC.heap_info

# Live memory
jcmd <pid> VM.native_memory

Heap vs off-heap: The ring buffer and pre-formatted label bytes live on-heap but are long-lived — they’ll be promoted to Old Gen immediately and rarely GC’d. The allocation rate should be near zero for the metrics layer itself. If you see consistent Eden allocation from the metrics package, you’ve got a formatting leak.

Build and Run

Prerequisites

• JDK 21+

• Maven 3.9+

• VisualVM 2.x (for thread/heap observation)

• curl (for endpoint verification)

• python3 (optional, for JSON pretty-print in verify.sh)

Setup and Start

cd flux-gateway-observability
chmod +x start.sh demo.sh verify.sh stop.sh cleanup.sh
./start.sh

Expected output:

==> Building flux-gateway-observability...
==> Starting server...
=== Flux Gateway Observability — Day 60 ===
[ObservabilityHttpServer] http://localhost:8080/dashboard
[ObservabilityHttpServer] Prometheus: http://localhost:8080/metrics
[main] Gateway on :7777
[main] Dashboard at http://localhost:8080/dashboard
[main] Prometheus at http://localhost:8080/metrics
[main] Press Ctrl+C to stop.
[GatewayServer] listening on :7777

Run the Load Generator

In a second terminal:

cd flux-gateway-observability
./demo.sh

Expected output:

[LoadGen] starting 50 clients @ 10 events/s each for 30s
[LoadGen] sent=1024 errors=0 remaining=28s
[LoadGen] sent=3148 errors=0 remaining=26s
...
[LoadGen] done. total_sent=15000 total_errors=0

Verify All Endpoints

./verify.sh

Expected output:

==> [1] Checking Prometheus /metrics endpoint...
  [PASS] gateway_connected_clients present
  [PASS] gateway_events_per_second present
  [PASS] gateway_connections_total present

==> [2] Checking JSON snapshot...
  [PASS] connectedClients in snapshot
  [PASS] eventsPerSecond in snapshot

==> [3] Checking dashboard...
  [PASS] dashboard returns 200

==> [4] Live metric values:
{
    "connectedClients": 50,
    "eventsPerSecond": 487.20,
    "totalConnections": 50,
    "totalDisconnections": 0,
    "totalMessageBytes": 49152,
    "connectionErrors": 0
}

==> All checks complete.

View Raw Prometheus Output

curl http://localhost:8080/metrics

Expected:

# HELP gateway_connected_clients Current number of connected WebSocket clients
# TYPE gateway_connected_clients gauge
gateway_connected_clients 50
# HELP gateway_events_per_second Sliding-window event processing rate
# TYPE gateway_events_per_second gauge
gateway_events_per_second 487.20
# HELP gateway_connections_total Total WebSocket connections accepted
# TYPE gateway_connections_total counter
gateway_connections_total 50
...
# HELP gateway_events_by_type_total Events processed broken out by type
# TYPE gateway_events_by_type_total counter
gateway_events_by_type_total{type="MESSAGE"} 9823
gateway_events_by_type_total{type="HEARTBEAT"} 3014
gateway_events_by_type_total{type="VOICE"} 1201
gateway_events_by_type_total{type="REACTION"} 962

Run Unit Tests

cd flux-gateway-observability
mvn test

All three test classes should pass: MetricsRegistryTest, PrometheusExporterTest, EventRateCalculatorTest.

VisualVM Observation

1. Open VisualVM, connect to the running gateway-observability process.

2. Under the Threads tab: you should see N carrier threads (N = CPU count) and the Virtual Thread count spiking to ~50 during the load test, then returning to baseline after the demo completes.

3. Under the Heap tab: the allocation rate should be near zero from the metrics layer — a flat sawtooth from normal JVM overhead, not a climbing ramp.

4. If you see carrier thread count growing beyond N, run jcmd <pid> Thread.dump_to_file -format=json /tmp/td.json and grep for "PINNED" to find the culprit.

Stopping and Cleanup

./stop.sh      # kills the running server process
./cleanup.sh   # removes target/ and build artifacts (does not kill the process)

Homework Challenges

Beginner: Add a gateway_disconnections_total counter broken out by reason code (timeout, error, clean close). Expose it in the Prometheus endpoint with a reason label.

Intermediate: Implement an HDR histogram for event processing latency. Track min/max/p50/p99 latency in nanoseconds using a simple power-of-2 bucket array (no external libraries). Expose as gateway_latency_ns_bucket{le="..."} in Prometheus histogram format.

Expert: Replace the pull-model Prometheus endpoint with a push-model PushGateway client. Batch-write metrics every 5 seconds using a Virtual Thread. Handle PushGateway unavailability with an exponential backoff retry loop using VarHandle-based state to avoid double-sending during retries. The state machine should have states: IDLE → PUSHING → RETRYING → IDLE.

A junior engineer deploying a new Gateway build does this:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30s

They run kubectl rollout restart deployment/gateway, watch the pods cycle, and mark the ticket done. At 3 AM the next deploy, PagerDuty fires. Error rate spikes to 40%. The load balancer logs show 80,000 simultaneous reconnect attempts hitting two healthy nodes that were already at 70% capacity. The cascade takes four minutes to recover.

The root cause: Spring’s graceful shutdown closes the HTTP listener and waits for in-flight requests to complete. WebSocket connections are not requests. They are long-lived, stateful sessions. Spring closes them all simultaneously the moment the drain window expires. Every client reconnects at once. This is the reconnect storm.

The Failure Mode: Thundering Herd on Deploy

At 100,000 concurrent WebSocket sessions, a simultaneous disconnect generates 100,000 TCP SYN packets within 200–400 ms. The surviving gateway nodes, already serving their own session load, must:

• Complete the TLS handshake for each new connection

• Re-authenticate the session token

• Re-subscribe the session to all guild pub/sub channels

• Re-hydrate presence state

Each reconnect costs roughly 8–12 ms of CPU across those steps. At 100K concurrent reconnects, that is 800,000–1,200,000 ms of CPU work arriving in under one second. Two nodes with 16 cores each have a combined capacity of roughly 32,000 ms of CPU per second. You are 25x over capacity. The nodes fall over. The load balancer marks them unhealthy. Now all reconnects hit zero nodes. Full outage.

The fix is not faster hardware. It is not more replicas. It is draining sessions gracefully so clients reconnect in a controlled, staggered fashion across the remaining healthy nodes.

The Flux Architecture: Drain Coordinator

Read more

Last week, schedules lived inside Java method annotations. That is fine for a classroom demo. It collapses the moment a product manager asks, “Can we move the reconciliation job from 2 AM to 3 AM without a deploy?” Shipping a JAR for every cadence change is how early systems age into change-request queues — not how Airflow, Quartz, AWS EventBridge, or Kubernetes CronJob operators work.

Those systems treat schedules as data: durable records that operators create, pause, retarget, and audit. GitHub Actions stores workflow crons in YAML that CI can edit without recompiling runners. Stripe’s billing cycles are configuration rows, not hardcoded timers in payment workers. Chronos — this week’s project — follows the same contract: TaskDefinition rows in an RDBMS, a REST catalog for mutations, a dynamic binder that turns ACTIVE rows into Spring ScheduledFutures, and Runnable / Callable wrappers that normalize execution into a result table.

What changes in your mental model is subtle but permanent. Scheduling stops being “a method that fires” and becomes “a write path with lifecycle.” If definitions are not durable, searchable, and activatable, nobody can answer what will fire at 02:00 across the fleet — which is the question on-call engineers ask when something goes wrong before sunrise.

At ultra-scale the insight sharpens: the schedule catalog is a control plane. Workers may multiply; queues may fan out; locks will come later. The catalog remains the source of truth for what exists. Get the model wrong now, and every later distributed feature becomes a retrofit.

Agenda

1. Why static @Scheduled is a product bottleneck

2. The task definition model as a system contract

3. RDBMS persistence for schedules (transactions, indexes, audits)

4. REST catalog: CRUD plus activate / pause as lifecycle ops

5. Component architecture of Chronos

6. Dynamic binding control flow: DB → cron → execute → record

7. Definition and execution state machines

8. Runnable vs Callable as execution interfaces

9. Fit in the 60-day distributed scheduler roadmap

10. Build, verify, assign, and mastery checklist

Core Concepts

From Code Timers to Catalog Entries

Week 1 wired triggers with @Scheduled(fixedRate|fixedDelay|cron). The compiler baked intent into bytecode. Chronos inverts that: intent lands in a table, and a runtime service binds eligible rows to Spring’s TaskScheduler.

Think of the difference between a playlist burned onto a CD versus a streaming library. The CD cannot change mid-flight; the library can. Production ops teams need the library. Redeploys become risk events — configuration changes should not require them.

The non-obvious implication for system design: create/update is now part of the scheduling surface. Validation, uniqueness, and state transitions belong in services — not only “does the cron fire.” Invalid cron on activate must fail loudly before a ghost ScheduledFuture exists with no matching row.

Definitions Are Contracts

A TaskDefinition answers four operational questions:

Read more

A junior engineer writes this:

List<Event> missed = eventRepo.findByGuildIdAndTimestampAfter(guildId, session.lastSeen());
socket.send(objectMapper.writeValueAsBytes(missed));

It works in staging with five users. It catastrophically fails when 400,000 mobile clients reconnect after a 20-minute outage — the scenario Discord engineers call a “reconnect storm.”

The hidden costs: findByGuildIdAndTimestampAfter does a full index scan with no upper bound. At 100 missed events per client, you are deserializing 40 million Event objects simultaneously. The Eden space fills in seconds. The JVM fires stop-the-world GC collections lasting 800ms each, exactly while the thread pool is already saturated processing reconnects. The database connection pool exhausts. The entire Gateway pod crashes and takes its neighbors with it via a cascading failure.

The framework hid every one of those failure modes behind a clean repository interface.

The Failure Mode: Unbounded Delta Fetching

The naive sync strategy has three structural flaws:

No upper bound on delta size. A client offline for six hours in a busy guild has missed tens of thousands of events. Materializing all of them into heap memory is an OOM waiting to happen.

No event prioritization. Presence changes, role updates, and membership events affect what the client can render. Dumping chat messages on top of them means the client shows a broken UI while waiting for structural data to arrive.

No cursor continuity. Using wall-clock timestamps for sync is incorrect. Clocks drift. Events written within the same millisecond have undefined ordering under timestamp >= ?. You need a monotonic sequence number, not a timestamp.

The Flux Architecture: Three-Phase Cursor Sync

Read more

• A write-through SQLite cache that persists every Gateway event to disk before the UI renders it.

• A cursor-based delta sync protocol: on reconnect, the client sends its last-seen message ID per channel and receives only the diff.

• Dirty-write tracking with optimistic UI updates and server-ACK reconciliation.

• A real-time dashboard showing cache hit rate, dirty row backlog, sync lag, and per-channel cursors.

1. The Junior Engineer’s Trap

A junior Discord clone engineer writes this on reconnect:

void onReconnect() {
    List<Message> messages = restClient.fetchMessages(channelId, limit=50);
    List<Guild>   guilds   = restClient.fetchGuilds();
    ui.render(messages, guilds);
}

It works in development. Then your service has a four-minute outage. It comes back up. Every one of your two million connected clients executes those REST calls simultaneously. Your API servers collapse under a thundering herd. You have traded one outage for two.

The second problem: the UI is blank during reconnect. Users see a spinner. Discord never shows a spinner on reconnect — they see their last known state instantly because it was cached locally. That is not magic. That is engineering.

2. The Failure Mode

Without a local cache, three distinct failure modes hit you at scale.

Thundering Herd on Reconnect

When the Gateway cluster recovers after a partition, every client simultaneously requests full state from the REST tier. A two-million-user guild server can generate 40 million API calls in under 30 seconds. No horizontal scaling saves you from that spike shape — you cannot provision for a vertical cliff.

<!-- IMAGE: thundering-herd-diagram.svg — Reconnect storm spike (naive) vs. cache-first flat request curve (production) -->

Perceived Latency on Cold Start

Cold-start clients must wait for network round-trips before rendering anything. On mobile, that is 300–800ms of blank screen — enough for users to assume the app is broken and force-quit. Each force-quit generates another reconnect, compounding the herd.

Wasted Bandwidth

Fetching 50 messages per channel on every reconnect, when 48 of them have not changed, is pure waste. At Discord’s scale that becomes petabytes of redundant transfer per day, directly inflating infrastructure cost.

3. The Flux Architecture: Write-Through Cache with Delta Sync

Read more

Every system that processes payments overnight, sends digest emails, or refreshes a product catalog depends on a scheduler. Netflix pushes encoding jobs on cron. Uber reconciles ride fares in batch windows. Your phone backs up photos on a fixed interval. The pattern is identical: something must fire at the right time, reliably, without human intervention.

Spring Boot ships with scheduling built in — but built in for one machine. This lesson establishes the foundation layer you will extend across 60 days: heartbeat monitoring, trigger patterns, thread pools, cluster duplication awareness, and the blueprint for a production distributed scheduler.

What makes scheduling a system design problem — not just a coding convenience — is that time itself becomes shared infrastructure. A bug in your REST API might affect one request. A bug in your scheduler can charge ten thousand customers twice, send duplicate emails to every user, or leave stale inventory on a storefront for hours.

Agenda

1. What task scheduling is and where it lives in system design

2. Component architecture of the Scheduler Foundation control plane

3. Spring’s @Scheduled trigger modes: fixedRate, fixedDelay, cron

4. Thread pool customization with ThreadPoolTaskScheduler

5. Task execution control flow — from trigger to metrics

6. Scheduled task lifecycle and state transitions

7. The multi-instance duplication problem

8. Target architecture for a distributed scheduler

Core Concepts

Scheduling as a Control Plane

Think of a scheduler as an operating system for time. Application code declares what should run and when; the scheduler decides how to dispatch work onto threads. In Spring, @EnableScheduling activates a background registrar that scans beans for @Scheduled methods and registers triggers against an internal TaskScheduler.

Three trigger semantics matter:

ModeFiresReal-world usefixedRateEvery N ms from startHeartbeats, health probesfixedDelayN ms after previous finishCleanup that must not overlapcronCalendar expressionReports at 2 AM, invoice cycles

The distinction is subtle but consequential. fixedRate measures from the start of each interval — if a task runs long, the next invocation may queue up immediately after it finishes, potentially stacking executions. fixedDelay waits until the task completes, then starts the countdown — ideal when overlapping runs would corrupt shared state. cron expresses calendar intent: “every Monday at 03:00 UTC,” which is how billing cycles and compliance reports are defined in production.

Stripe’s reconciliation jobs use cron-like windows. Datadog agents use fixedRate heartbeats. Log compaction in Kafka uses fixedDelay so a slow run does not stack.

Thread Pools Are Throughput Levers

By default, Spring uses a single-thread scheduler. One slow task blocks everything else — unacceptable at scale. ThreadPoolTaskScheduler with a custom pool (10 core, 20 max in our project) lets concurrent tasks run in parallel. Wiring it through SchedulingConfigurer replaces the default registrar — the same pattern used inside large e-commerce flash-sale engines where hundreds of micro-tasks fire per second.

Without a pool, your health probe waits while a report generation task holds the only thread for thirty seconds. With a pool, each scheduled method gets its own worker when the trigger fires. The trade-off: more threads mean more memory and more concurrent access to shared resources — which is exactly why Week 3 introduces distributed locks.

The Duplication Trap

Deploy two instances behind a load balancer and every @Scheduled method runs twice. No shared state, no coordination. Instance A and Instance B both charge customers, both generate reports. Our cluster simulation launches two JVMs with distinct instance IDs, persists executions to H2, and surfaces duplicate counts on the dashboard — the same failure mode that burned teams at early-stage fintech startups before they added distributed locks.

This is not a Spring bug. It is the correct behavior of independent clocks. Each JVM has its own scheduler thread, its own trigger registry, and zero awareness of peer nodes. The fix — leader election, distributed locks, or message-queue handoff — comes in later weeks. Week 1 makes the problem visible so you never forget it exists.

Component Architecture

UI / API layer : sit at the top — the human-facing and programmatic entry points:

• Dashboard — the live web UI where operators watch heartbeats, thread utilization, duplicate counts, and the architecture roadmap. In production systems like Airflow or Temporal, this role is filled by a web console; here it is a lightweight Thymeleaf dashboard.

• REST API — programmatic access to the same data. Other services, CI pipelines, or monitoring tools call these endpoints without loading the UI.

• Task Registry — an in-memory catalog of task definitions (name, trigger type, expression, lifecycle). In Week 2 this moves to a persistent database; the box is already positioned where that upgrade lands.

Runtime layer do the actual work:

• Thread Pool — the worker fleet inside a single JVM. When a trigger fires, the pool assigns a thread. Pool size directly caps how many scheduled tasks can run simultaneously on one node.

• Scheduler Core — the brain. It hosts @Scheduled method registration, trigger evaluation, and dispatch to the thread pool. Everything time-related flows through here.

• Execution Store — the audit trail. Every cluster-mode run is persisted with instance ID, timestamp, status, and details. This is your forensic log when duplicates appear.

Future layer:

• Lock Service — not implemented yet, deliberately shown with a dashed border. In Weeks 3–4 you will add Redis- or database-backed locks so only one instance executes a given task tick. The arrow from Instance B toward Lock Service hints at the coordination path both nodes will eventually use.

Instance A and Instance B at the bottom represent horizontally scaled deployments. Both connect upward to Scheduler Core independently — that is the duplication problem visualized. In a corrected architecture, Lock Service sits between instances and execution, gating who may proceed.

Task Execution Control Flow

This flowchart traces what happens from the moment a trigger fires until metrics appear on the dashboard.

Step-by-step walkthrough

• Trigger — the starting event. A clock tick (fixedRate), a completion delay (fixedDelay), or a cron match initiates the pipeline. Nothing runs yet; the scheduler has simply decided now is the time.

• Timer → Thread Pool → Execute — the horizontal chain across the top is the happy path on a single node. The timer hands off to the pool, the pool assigns a worker thread, and the @Scheduled method body runs. If the pool is saturated, tasks queue inside the pool — visible on the dashboard as rising active thread counts.

• Multi? (decision diamond) — the critical fork. After execution completes, the system asks: are multiple instances running? On a single JVM the answer is no, and the flow proceeds straight to Persist Log. In cluster mode the answer is yes — and without a lock, both instances took the execute path independently.

• Duplicate Run — the failure branch. When two instances fire the same trigger, both reach Execute. Neither knows the other ran. Billing runs twice. Reports generate twice. This box exists because Week 1 intentionally omits the lock that would block this path.

• Persist Log → Metrics — regardless of duplication, each execution is recorded in the Execution Store and surfaced as dashboard metrics and Prometheus counters. This is why the cluster simulation panel can count duplicates: the log captures who ran what, even when coordination is missing.

Scheduled Task Lifecycle

Every scheduled task moves through a finite set of states. Understanding this machine is prerequisite to adding retries, timeouts, and idempotency in later lessons.

States explained

Transitions

• tick (IDLE → SCHEDULED): the trigger fires. For fixedRate, this happens on a wall-clock interval regardless of previous run duration. For fixedDelay, it happens N milliseconds after the last run finished.

• dispatch (SCHEDULED → RUNNING): the thread pool assigns a worker. If the pool is full, SCHEDULED persists until a thread frees up — this is backpressure at the scheduling layer.

• done (RUNNING → SUCCESS): clean completion. The task returns to IDLE via the dashed retry/next-cycle path, ready for the next trigger.

• error (RUNNING → FAILED): uncaught exception or resource exhaustion. FAILED can transition to FAILED again on repeated errors (the vertical arrow between SUCCESS and FAILED represents monitoring overlap — a task that succeeded once can fail on the next cycle).

Pool Queue

The green Pool Queue box connected to RUNNING represents tasks waiting for threads when the pool is saturated. In our project with six concurrent pool workloads plus three core scheduler tasks, you will see this queue activity on the Thread Pool dashboard panel during high-frequency bursts.

Why this matters at scale

Production schedulers at companies like Shopify and LinkedIn treat state as a first-class database column, not an in-memory afterthought. When you can query “show me all tasks stuck in RUNNING for more than 10 minutes,” you have operability. This state machine is the conceptual foundation for that capability.

Key Code Snippets

Enable scheduling and custom pool:

@SpringBootApplication
@EnableScheduling
public class SchedulerFoundationApplication { ... }

@Configuration
public class FoundationSchedulingConfigurer implements SchedulingConfigurer {
    public void configureTasks(ScheduledTaskRegistrar registrar) {
        registrar.setScheduler(primaryTaskScheduler);
    }
}

Three trigger types in one service:

@Scheduled(fixedRate = 5000)
public void healthProbe() { ... }

@Scheduled(fixedDelay = 15000)
public void resourceCleanup() { ... }

@Scheduled(cron = "0 * * * * ?")
public void periodicReport() { ... }

Instance identity for cluster observability:

app:
  instance:
    id: ${INSTANCE_ID:node-local}

Each JVM sets INSTANCE_ID differently in cluster mode so the Execution Store can attribute duplicates to specific nodes.

Implementation Guide

GitHub Link :

https://github.com/sysdr/taskscheduler-p/tree/main/week1/scheduler-foundation

Prerequisites

ToolVersionJava21+Maven3.9+curlany recentDocker + Composeoptional (cluster demo via containers)

Verify:

java -version    # should show 21+
mvn -version
curl --version

Phase 1 — Project Setup

1.1 Generate the project

cd scheduler-foundation

Phase 2 — Build & Test

2.1 Compile and run tests

./build.sh all

Success criteria:

• JAR created: target/scheduler-foundation-1.0.0.jar

• 7 tests pass (1 context + 6 integration)

Phase 3 — Run the Application

3.1 One-command demo (recommended first run)

./demo.sh

This will: stop old instances → build → test → start on port 8080 → wait 15s → verify APIs.

Open: http://localhost:8080/

3.2 Manual start modes

./start.sh single     # one JVM on :8080
./start.sh cluster    # two JVMs on :8080 and :8081
./start.sh docker     # two containers (requires Docker Desktop + WSL integration)

3.3 Stop and clean up

./stop.sh             # stop JVM + docker compose
./cleanup.sh          # stop + remove target/, logs/, data/, Docker image
./cleanup.sh --deep   # also prune unused Docker cache

Phase 4 — Verify Each Feature

Work through the dashboard panels left-to-right. Refresh is automatic every 3 seconds.

4.1 Heartbeat (Overview + Heartbeat panel)

What it proves: The scheduler thread is alive.

curl -s http://localhost:8080/api/heartbeat | python3 -m json.tool

Expected fields: executions (increments), elapsedSeconds, intervalMs (10000).

Source: HeartbeatSchedulerService.java — @Scheduled(fixedRateString = "${scheduler.heartbeat.interval-ms}")

4.2 Scheduling Patterns (fixedRate / fixedDelay / cron)

What it proves: All three Spring trigger types work independently.

TaskTriggerIntervalHealth ProbefixedRate5sResource CleanupfixedDelay15s after completionPeriodic Reportcronevery minute

curl -s http://localhost:8080/api/dashboard | python3 -c 
  "import sys,json; d=json.load(sys.stdin); print(d['coreScheduler'])"

Source: CoreSchedulerService.java

4.3 Thread Pool (Thread Pool panel)

What it proves: Custom pool handles concurrent scheduled tasks.

curl -s http://localhost:8080/api/thread-pool | python3 -m json.tool

Watch activeThreads, completedTasks, and the doughnut chart on the dashboard.

Sources:

• SchedulerConfig.java — defines primaryTaskScheduler bean

• FoundationSchedulingConfigurer.java — wires pool to all @Scheduled tasks

• PoolWorkloadService.java — 6 concurrent workloads at different rates

Key config in application.yml:

scheduler:
  pool:
    size: 10
    max-size: 20
    thread-name-prefix: scheduler-pool-

4.4 Cluster Simulation (duplicate executions)

What it proves: @Scheduled on multiple instances = duplicate runs.

./stop.sh
./start.sh cluster

Open both: http://localhost:8080/ and http://localhost:8081/

Wait 2–3 minutes, then:

curl -s http://localhost:8080/api/cluster/status | python3 -m json.tool

Expected: duplicateCounts shows counts > 1 per task when both nodes are running.

Sources:

• ClusterScheduledService.java — report (cron), billing (fixedRate), cleanup (fixedDelay)

• ClusterExecution.java + ClusterExecutionRepository.java — persisted audit log

• Instance ID via INSTANCE_ID env var / app.instance.id in config

4.5 Target Architecture (blueprint panel)

What it proves: Roadmap from single-node scheduler to distributed system.

curl -s http://localhost:8080/api/architecture | python3 -m json.tool

Expected: 7 components (3 CURRENT, 4 PLANNED) + roadmap.currentCapabilities / nextCapabilities.

Source: ArchitectureBlueprintService.java

4.6 Task Registry API

curl -s http://localhost:8080/api/tasks | python3 -m json.tool
curl -s http://localhost:8080/api/tasks/statistics | python3 -m json.tool

Source: TaskRegistryService.java, TaskRegistryController.java

4.7 Prometheus metrics

curl -s http://localhost:8080/actuator/prometheus | head -20
curl -s http://localhost:8080/actuator/health
curl -s http://localhost:8080/actuator/scheduledtasks

Phase 5 — Docker Cluster (Optional)

Requires Docker Desktop with WSL2 integration enabled.

./cleanup.sh
./start.sh docker

Images used (official Docker Hub):

• Build stage: maven:3.9.9-eclipse-temurin-21

• Runtime: eclipse-temurin:21-jre-jammy

Both containers share volume scheduler-data so H2 executions aggregate.

Verify:

docker ps
curl http://localhost:8080/actuator/health
curl http://localhost:8081/actuator/health

Completion Checklist

• ./build.sh all — 7 tests green

• ./demo.sh — dashboard loads at http://localhost:8080/

• Heartbeat counter increments every 10s

• Scheduling Patterns panel shows all three trigger types firing

• Thread Pool chart shows active vs idle threads

• ./start.sh cluster — duplicate counts appear in Cluster Simulation

• Architecture panel shows 7 components + roadmap

• ./cleanup.sh removes runtime artifacts cleanly

Working Demo Link :

What’s Next Preview

Week 1 uses static @Scheduled annotations. Week 2 introduces:

• TaskDefinition entity persisted in a database

• REST API for dynamic task CRUD

• Runtime scheduling from stored definitions (not hard-coded annotations)

The dashboard shell and architecture blueprint you built this week carry forward unchanged.

Quick Command Reference

 cd scheduler-foundation

# Develop cycle
./build.sh all          # compile + test
./demo.sh               # full demo
./start.sh single       # run
./stop.sh               # stop

# Cluster demo
./start.sh cluster

# Docker demo
./start.sh docker

# Cleanup
./cleanup.sh
./cleanup.sh --deep

Assignment

1. Add a fourth scheduled task using a cron expression that fires every 30 seconds. Register it in the task registry API.

2. Change the thread pool size to 5 and observe how highFrequencyPulse execution latency shifts on the dashboard.

3. Launch cluster mode and record how many BILLING_CYCLE executions occur in 3 minutes with two instances. Calculate the duplication factor.

4. Sketch (on paper or in the architecture panel) where a Lock Service would sit between the Scheduler Core and Worker Tasks.

Solution Hints

1. Add @Scheduled(cron = "0/30 * * * * *") in PoolWorkloadService or a new bean; POST a matching entry via /api/tasks.

2. Set scheduler.pool.size: 5 in application.yml, restart, compare activeThreads and bar chart heights before/after.

3. Query GET /api/cluster/status on either port; duplicateCounts.BILLING_CYCLE divided by expected single-instance count (≈2 in 3 min at 90s rate) gives your factor — expect ~2× with two nodes.

4. Lock Service sits between Scheduler Core and execution dispatch — it gates which instance may enter RUNNING state for a given task tick. On Diagram 1, draw an arrow from Lock Service to both Instance A and Instance B, with only one path open per tick.

A junior engineer reaches for Spring’s @RateLimiter annotation from Resilience4j, or wires up Bucket4j with an in-memory ConcurrentHashMap. The code is clean. Tests pass. In staging, with a single node and synthetic load, it holds.

Then production happens.

Your Gateway cluster has twelve nodes behind an L4 load balancer. A credential-stuffing bot opens 120 connections — ten per node. Each node’s local AtomicInteger sees ten IDENTIFY packets. Your limit is 50 per minute. Every node reports: “Under limit.” The bot authenticates 120 sessions in thirty seconds. Your in-memory rate limiter protected nothing.

That is the locality problem. Any rate limiter that does not share state across nodes is theatre.

The Failure Mode

Why the naive fix also fails: The obvious next step is a shared Redis key with INCR + EXPIRE. Junior engineers write this as two round-trips:

long count = redis.incr(key);           // round-trip 1
if (count == 1) redis.expire(key, 60);  // round-trip 2

Under concurrent load, two nodes hit INCR simultaneously before either fires EXPIRE. The key never expires. You’ve built a permanent ban instead of a sliding window. This is a classic TOCTOU race on a distributed counter.

The thread cost: If each IDENTIFY triggers a synchronous Redis call on a platform thread, at 50,000 logins/second across the cluster you need 50,000 threads blocked on network I/O. At 1 MB stack each, that is 50 GB of thread stack memory — before your application does a single unit of work.

GC pressure from key strings: A naive Map<String, RateLimitBucket> where the key is "ratelimit:identify:" + ip allocates a new String object per request. At scale the Eden space saturates in milliseconds. Stop-the-world minor GC pauses hit every 200–400 ms, causing cascading IDENTIFY timeouts on legitimate clients.

The Flux Architecture

Read more

A junior engineer inherits a WebSocket service running fine on a single node. Traffic grows. The obvious fix: throw Nginx in front, configure upstream with three backend servers, call it done.

upstream gateway_cluster {
    server node1:9001;
    server node2:9002;
    server node3:9003;
}

This works for HTTP. It silently destroys WebSocket.

The problem is not Nginx’s configuration. It is the engineer’s mental model. HTTP is stateless — every request is self-contained, and round-robin distribution is correct. WebSocket is stateful. The moment a client completes the HTTP Upgrade handshake, it establishes a persistent TCP connection to a specific backend process. All future frames traverse that exact socket. Routing the upgrade request to node1 and the next frame to node2 is not load balancing — it is data corruption.

Spring Boot’s @EnableWebSocketMessageBroker hides this entirely. The abstraction manages sessions internally, and the developer never sees the connection identity. At 10,000 users this breaks: STOMP session IDs are stored in memory on one node. Any reconnection that lands on a different node starts a cold session. Authentication, subscriptions, and presence state are all gone.

The Failure Mode

Nginx with default configuration silently kills WebSocket connections in three ways.

Timeout eviction. proxy_read_timeout defaults to 60 seconds. A WebSocket connection with no traffic — a user reading, not typing — looks idle to Nginx and gets dropped. The client reconnects. The reconnect hits a different node. Session is lost.

Missing upgrade headers. The HTTP Upgrade handshake requires Connection: Upgrade and Upgrade: websocket headers to be forwarded verbatim to the backend. Nginx strips hop-by-hop headers by default. Without explicit proxy_set_header directives, the backend never sees the Upgrade request and returns a 200 with an HTTP body. The client’s WebSocket library closes immediately.

Source IP erasure. After SSL termination, every connection appears to originate from 127.0.0.1. Rate limiting by IP becomes useless. Per-user throttling breaks. Audit logs show the proxy address, not the real client. You need proxy_set_header X-Forwarded-For $remote_addr — and the backend must trust and parse it, not ignore it.

The deeper failure is architectural. A load balancer that does not understand session locality is not a load balancer for stateful services. It is a random connection redirector.

The Flux Architecture

The Edge Proxy layer in Flux is built on three principles.

Consistent Hashing for Session Locality. Every connection carries a user identity — either in the URL path (/gateway?user_id=42), a cookie, or an X-User-Id header. The proxy hashes this identity against a ring of backend nodes. For any given user, hash(userId) mod ringSize always resolves to the same node, regardless of which proxy instance handles the request. This means reconnections land on the same backend. Session state survives transient network interruptions.

Transparent TCP Tunneling. Once the proxy selects a backend, it opens a raw TCP socket to that node and splices the two streams: client-to-backend and backend-to-client. No WebSocket frame parsing happens at the proxy layer. The proxy is deliberately dumb about application protocol — it moves bytes, not messages. This keeps the hot path allocation-free.

Virtual Thread Bidirectional Pumps. Each tunnel requires two blocking I/O loops: one reading from the client socket and writing to the backend, one reading from the backend and writing to the client. With platform threads, this costs two OS threads per connection — catastrophic at 100K users. With Virtual Threads, both loops are lightweight continuations scheduled on the carrier thread pool. Memory cost drops from ~1MB per thread to ~2KB per virtual thread stack.

Implementation Deep Dive

GitHub Link :

https://github.com/sysdr/discord-flux-p/tree/main/day55/flux-edge-proxy

Consistent Hash Ring

The hash ring stores virtualNodes copies of each backend node to ensure even distribution. Each virtual node maps a hash value to a physical backend address. When routing a connection:

long hash = murmur3(userId);
NavigableMap<Long, InetSocketAddress> tailMap = ring.tailMap(hash);
InetSocketAddress backend = tailMap.isEmpty()
    ? ring.firstEntry().getValue()   // wrap around
    : tailMap.firstEntry().getValue();

TreeMap.tailMap() gives O(log n) lookup. No lock is needed on read paths if the ring is rebuilt via Collections.unmodifiableNavigableMap() on update.

HTTP Upgrade Forwarding

The proxy must forward the client’s original Upgrade request byte-for-byte to the backend, then splice the streams. Critical headers:

Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: <client key>
X-Forwarded-For: <real client IP>
X-User-Id: <extracted user identity>

The backend responds with 101 Switching Protocols. After the proxy forwards the 101 back to the client, both sockets enter tunnel mode — raw byte copying with no further HTTP parsing.

Zero-Allocation Tunnel Loop

void pump(InputStream src, OutputStream dst, byte[] buffer) {
    try {
        int n;
        while ((n = src.read(buffer)) != -1) {
            dst.write(buffer, 0, n);
            dst.flush();
        }
    } catch (IOException ignored) {
        // Either side closed — tunnel is done
    }
}

The buffer is allocated once per tunnel (8KB). No object creation in the loop. No WebSocket frame decoding. The proxy is a pipe, not a parser.

Backend Health Tracking

A HealthMonitor pings each backend over a lightweight HTTP /health endpoint every 5 seconds using Virtual Threads. If three consecutive pings fail, the node is marked UNHEALTHY and removed from the hash ring. Connections in flight are not terminated — the proxy drains them. New connections bypass the failed node until it recovers.

Working Demo Link :

Production Readiness

Metrics to instrument:

MetricAlert ThresholdActive tunnels per node>80% of node capacityUpgrade failure rate>0.1% over 1-minute windowBackend health check latency>200msProxy buffer queue depth>10K pending bytesVirtual thread countMonitor via jcmd <pid> Thread.print

What VisualVM shows: Under load, carrier thread count stays fixed (typically Runtime.getRuntime().availableProcessors() * 2). Virtual thread count grows proportionally to active tunnels. Heap allocation is flat — the reused byte arrays never escape to the old generation.

The reconnection storm problem. When a backend node fails, all its clients reconnect simultaneously. A naive implementation re-hashes all of them to two remaining nodes, doubling load. Consistent hashing minimizes this: only the users whose hash range covered the failed node migrate. Everyone else stays put.

Step-by-Step Guide

Prerequisites

• Java 21+ (java -version should show 21.x or higher)

• Maven 3.9+

• VisualVM (bundled with JDK or download separately)

• Terminal with at least 3 panes (tmux or split terminal)

Execution

cd flux-edge-proxy
./start.sh

start.sh launches three GatewayNode instances on ports 9001–9003, then starts the EdgeProxy on port 8080 with the dashboard on port 8090.

Verification

./demo.sh

The demo script connects 50 WebSocket clients with random user IDs and prints which backend each was routed to. Open http://localhost:8090/dashboard to see the live routing table, per-node connection counts, and hash ring visualization.

Run ./verify.sh to confirm consistent routing: the same user ID always lands on the same node.

In VisualVM, attach to the EdgeProxy process. Observe:

• Carrier thread count: stable at ~16 (matches CPU core count)

• Virtual thread count: grows with active connections, ~1 per active tunnel pair

• Heap allocation rate: near-zero in steady state

Homework

Beginner: Add X-Gateway-Node response header so clients can log which backend they connected to.

Intermediate: Implement weighted consistent hashing. Node3 has 2x capacity — give it 2x virtual nodes on the ring. Verify with verify.sh that approximately 50% of connections land on node3.

Expert: Implement connection draining. When a node is removed from the ring, existing tunnels continue until the client disconnects. New connections never route to that node. Add a /admin/drain?node=9002 endpoint to the dashboard and measure how long the drain takes under the demo load.

What you will build today: A 3-node Gateway cluster where each node maintains persistent TCP connections to every other node, routes events through a binary wire protocol with zero String allocations in the hot path, and enforces explicit backpressure using bounded queues — the same architectural pattern Discord uses to prevent a single slow node from taking down the entire cluster.

What you will understand after today:

• Why frameworks hide the slow consumer problem and how it kills production systems

• How binary framing eliminates garbage pressure in the inter-node hot path

• Why Virtual Threads are the exact right tool for persistent peer connections

• How a SubscriptionTable turns an O(N²) broadcast into a targeted forward

Part 1 — The Theory

The “Spring Boot” Trap

A junior engineer hits the requirement: “When User A on Gateway-1 sends a message to a channel where User B is on Gateway-2, Gateway-1 needs to forward that event.” Their instinct: pull in Spring Cloud Bus, drop a RabbitMQ dependency, annotate with @StreamListener, and call it solved.

Here’s what they just hid from themselves:

// What Spring Cloud Bus does under the hood — every forwarded event
String json = objectMapper.writeValueAsString(event);  // Allocates String + char[] + byte[]
rabbitTemplate.convertAndSend("gateway.events", json); // Copies those bytes again

At 50,000 events/second across a 20-node cluster, that’s 50,000 ObjectMapper.writeValueAsString() invocations per second. Each creates: a String, a backing char[], and a byte[] for the UTF-8 encoding. Three heap allocations per event in the hot path. Your Eden space fills in milliseconds. Minor GC fires every 40–80ms. Users see it as jitter: a 2ms message delivery that occasionally spikes to 60ms. At Discord scale, that 60ms spike is a visible stutter in 100 million chat windows.

The deeper failure: Spring Cloud Bus gives you zero visibility into back-pressure. When RabbitMQ falls behind, your producers block silently on a Channel.basicPublish() inside a framework-managed thread pool. By the time you notice, your Gateway threads are exhausted and new WebSocket connections are being rejected.

The Failure Mode: The Slow Consumer Death Spiral

The architectural failure is the slow consumer problem. Consider a 3-node Gateway cluster:

Gateway-1 → [outbound queue] → TCP → Gateway-2 (overloaded, processing slowly)
                                   → TCP → Gateway-3 (healthy)

If the queue feeding Gateway-2 is unbounded — the classic LinkedList or LinkedBlockingQueue(Integer.MAX_VALUE) — a traffic spike causes queue depth to grow at the rate of (production rate − consumption rate). At 10,000 events/second with Gateway-2 processing at 8,000 events/second, the queue grows by 2,000 entries/second. After 90 seconds: 180,000 event objects on heap. Each holding a byte[] payload. OOM in 10 minutes.

The JVM makes this worse. Those queued events are tenured — they survive enough minor GCs to be promoted to Old Gen. When Old Gen fills, you get a Stop-The-World Full GC. For 30 seconds, every Gateway-1 thread is frozen. Every client connected to Gateway-1 sees a 30-second message gap.

Discord’s real lesson here: their initial presence broadcast was O(N²). Every user status change went to every gateway node. The fix was lazy subscription — a SubscriptionTable mapping channelId → Set<nodeId>. You only forward to nodes that have actual subscribers. Presence updates for a channel with 3,000 members on 2 gateway nodes forward to exactly 2 nodes, not 50.

The Flux Architecture

Read more