In the early days a program named memory by pointing at a real spot on the chip: "put this number in cell 4000." One program, one machine, nothing to be confused about.

Then people noticed how much time computers spent doing nothing. A program waiting on a slow tape drive left the processor idle, so the obvious fix was to keep several programs in memory at once and switch to another one whenever the current one stalled.

That broke the whole scheme. Program A writes to cell 4000. Program B also writes to cell 4000, because why wouldn't it — whoever wrote B had no idea A existed. Same cell. Last write wins, and the other program is silently corrupt.

Worse, B could go read A's data on purpose. Nothing was in the way. There was no wall between programs, just an unspoken agreement between programmers to stay out of each other's cells, which is not a security model.

The lie every process is told

The fix showed up in the Manchester Atlas around 1962, and it's still one of my favourite ideas in computing: lie to every program.

Tell each one it has the entire machine to itself — a clean stretch of memory starting at address 0, all of it its own. The program believes it completely. It writes to "address 0x5000", and hardware quietly rewrites that into a totally different real location before the memory chip ever hears about it.

Those made-up addresses are virtual addresses. Every process gets its own set, and every set is translated differently.

So A's 0x5000 and B's 0x5000 end up in different physical cells. The two processes aren't discouraged from touching each other's memory — they can't even name it. Isolation stops being politeness and becomes structural.

That's the part worth sitting with. The safety doesn't come from a permission check that could be got around. It comes from one program's addresses simply not existing in the other one's vocabulary.

Everything else in this post happens inside that lie.

How the lie is kept

Translating one address at a time would be absurd — a lookup table with a row per byte. So memory gets chopped into fixed-size blocks called pages, and translation happens one page at a time.

Every process gets a page table: a list saying "my page 0 is really block 91, my page 1 is really block 12, my page 2 is really block 47." The program sees a tidy row of pages numbered from zero. Reality is those blocks sitting anywhere in RAM, in any order, with other processes' blocks in between.

Every memory access in the program goes through this. Hardware on the processor splits the address into "which page" and "how far into that page", looks up the real block, and jumps there. It stays fast because the chip keeps the few most recent translations in a tiny cache right beside it, and real programs keep hitting the same handful of pages over and over.

Textbooks all say a page is 4 KB. That's worth checking instead of believing — getconf PAGESIZE on the Mac I'm writing this on prints 16384. Apple Silicon uses 16 KB pages. Linux on x86 really is 4 KB. It's a design choice, not a law of nature.

Android is the live proof. It supported nothing but 4 KB for its entire life, and from Android 15 it runs on 16 KB devices too — which broke apps, because native libraries had been compiled against the old number, some with 4096 hard-coded outright. Pure Kotlin and Java apps were fine. Everything with native code has to be rebuilt, and Google Play made that a hard requirement for apps targeting Android 15 and up — a deadline that has already moved once and now sits at February 1, 2027. An entire ecosystem recompiling itself over a constant nobody thought of as a variable.

How much memory does a process actually get?

On a 32-bit machine an address is 32 bits wide, so the biggest one a program can pronounce is about 4 GB. Put 16 GB of RAM in that box and a single process still can't reach past its own 4 GB — the rest is sitting right there, unnameable.

64-bit blew that ceiling away: roughly 128 terabytes on Linux. Which raises the obvious question. If every process gets 128 TB, how do forty of them fit on a laptop with 16 GB?

They don't, and they don't have to. That number is what a process may address, not memory it was handed. At startup only a few small regions have real memory behind them. Everything else is unmapped — the program never asked for it, so the OS has no record that those addresses mean anything at all. Touch one and there is nothing to look up, and the process gets killed. That's a segfault: not "the machine is out of memory" but "that address means nothing."

And when a program does ask for memory, the OS mostly just writes the request down. Real blocks get attached later, as the program actually touches those pages. So naming is nearly free — only the pages that really get used cost RAM.

The map of a process's memory

So which parts do get mapped? Not one undifferentiated blob. The OS carves the space into regions, in a layout that has barely changed since 1970s Unix.

One small C++ program touches all five:

#include <vector>

// TEXT  — the compiled instructions of every function below.
//         Read-only, and shared between processes running this binary.

int count = 5;          // DATA — starts with a value, so the 5 sits in the file
int total;              // BSS  — starts at zero, so the file stores nothing
int table[1000000];     // BSS  — 4 MB of zeros the file never has to store

int main() {
    int n = count + 1;        // STACK — a local, gone the moment main returns
    int* p = new int(42);     // STACK holds p; HEAP holds the 42 it points at
    std::vector<int> rows(n); // STACK holds rows' bookkeeping,
                              // HEAP holds its n elements

    delete p;                 // the heap block has to be handed back by hand
    return 0;                 // n, p and rows' bookkeeping cost nothing to free
}

How they grow — and do they ever collide?

Stack coming down, heap going up, one gap between them. The obvious worry is what happens when they meet in the middle.

On a 32-bit machine that was a genuine fear. Out of that 4 GB of nameable space only about 3 GB belonged to the program, and a hungry heap and a deep stack really were competing for the same room.

64-bit dissolved the problem rather than solving it. Against 128 terabytes of nameable space, a stack capped at a few megabytes is never going to run into a heap climbing from the other end — the heap would have to outgrow every machine on earth first. They're two ants at opposite ends of a stadium, walking toward each other.

Real allocators don't even grow in a tidy line any more. For large requests they skip the frontier entirely and ask the OS for a fresh region wherever there's space. Print a real process's memory map and it looks less like two regions creeping together and more like a scattering of mapped islands with an ocean of nothing in between.

The stack

Of the two regions sharing that gap, the stack is the simpler one — simple enough that most of its work is done by moving a single register.

It belongs to a thread, not to a process and definitely not to a function. Two threads, two stacks — which is exactly what lets each of them sit in the middle of its own chain of calls at the same time.

Each function call pushes a frame: that call's local variables plus the address to jump back to when it finishes. The function returns, the frame pops. Plates on a pile.

What makes it fast is how boring it is. Making room for a function's locals is one instruction — move the stack pointer down by 32 bytes. Freeing them on return is the same instruction backwards.

And that really is all "freeing" means here. Nothing gets wiped. The pointer moves back to where the frame started, everything past that line counts as free, and the old values sit there untouched until the next call writes over them.

The stack also has a hard ceiling — about 8 MB for a program's main thread on Linux, and ulimit -s on the Mac I'm writing this on reports 8176 KB, which is 8 MB minus one page. Threads spawned later usually get far less; on macOS the default is 512 KB. Cross the line and the program dies with a stack overflow. Infinite recursion is just the fastest known way to find that line.

The heap

The heap is where things go when the stack can't hold them: anything whose size isn't known ahead of time, and anything that has to outlive the function that created it.

One heap per process, shared by every thread in it.

The heap region is a contiguous run of addresses, but what's inside it usually isn't. Objects get freed whenever the program stops needing them, which is nothing like the order they were created in, and that leaves holes.

Which is why putting something on the heap costs more than it looks. When a program needs room for a new value — a list, an object, anything whose size wasn't known in advance — it asks the language's runtime for the space. The runtime keeps a list of every free hole in the heap. It walks that list until it finds one big enough, cuts the requested amount off the front, and writes down that the leftover is still free. If several threads are asking at the same moment they may have to wait their turn, so two of them don't get handed the same hole.

That's the whole job, and none of it is one instruction.

In C and C++ the runtime also can't tidy up to make its own work easier. A block that's in use never gets moved out from under the program — the program is holding its raw address, and moving the block would turn that address into a lie. So the holes stay where they are, and when the heap needs more room the runtime grows it at the edge or asks the OS for a fresh region somewhere else.

Some garbage-collected runtimes — Java's and C#'s among them — do slide live objects together to close the gaps. They can afford to, because the program never holds a raw address that moving would invalidate.

Why the stack is faster

Put the two side by side and it stops being a competition.

Making room on the stack is one instruction. Making room on the heap means finding a hole, splitting it, writing down the leftover, and sometimes waiting on another thread. Then someone has to decide when it dies — a programmer releasing it, or a collector hunting for what nothing points at any more. The stack never has that conversation: the function returned, it's gone.

The stack also stays hot. It's one small patch of memory the program hammers constantly, so it's nearly always sitting in fast on-chip cache, while heap objects are scattered across pages — and chasing pointers between them means trips out to main memory, around a hundred times slower.

All of it comes back to order. The stack frees in exactly the reverse of the order it filled, so it never has holes, never searches, and never has to decide anything. The heap frees in whatever order it likes.

The freedom that makes the heap useful is the same freedom that makes it slow.

Questions for You

1. Why is the stack faster than the heap?

2. What do 32-bit and 64-bit operating systems actually mean?

3. How does a high-level language like Python handle a dynamically sized array — and what happens when it grows to the point where another object is already using the memory right after it?

I use the internet all day and couldn't have told you how it got here. So over a few evenings I went down the rabbit hole, asking one dumb question about everything: what would break without this?

Ask that enough and the whole history lines up behind one idea: every protocol was invented to fix a specific pain in the thing before it. Packets fixed the phone network, TCP/IP fixed packets, DNS fixed TCP/IP — patches all the way down. Here's how it goes.

The whole story on one page

If any dates are worth holding onto, it's two: 1969, when ARPANET sent its first message, and January 1, 1983, the day the internet was really born. Everything else is the why.

It started with a wasteful phone call

In the 1960s, two computers had no natural way to talk. The only model around was the telephone network — circuit switching, where a call reserves one continuous line end to end for the whole conversation.

Fine for people. Terrible for computers, which fire off data in bursts and leave that line idle most of the time — and if any link on the path dies, the call goes with it.

Packet switching threw the reservation out: chop the message into small packets, write the destination on each, and let them find their own way across shared links.

This went live in 1969 as ARPANET. The first message was meant to be "LOGIN"; it crashed after two characters, so the internet's first word was "LO." (I love that. And no — ARPANET wasn't built to survive a nuclear strike; that myth belongs to Paul Baran's earlier work. It was really about sharing computers too expensive to buy twice.)

Teaching networks to talk to each other

By the early '70s there wasn't one network but several — ARPANET, a packet radio network, a satellite network — each with its own rules, none able to talk to the others. Networking between networks is where the word "internet" comes from.

The old protocol, NCP, assumed there was only one network and that it was reliable: it leaned on ARPANET to deliver every byte, in order. A second, flakier network breaks that completely.

In 1974 Vint Cerf and Bob Kahn designed TCP on the opposite bet — don't trust the network at all. Give every host an address that means the same thing everywhere; wrap data in packets any network can carry as dumb cargo; and put the endpoints, not the network, in charge of reordering and resending what's lost. Where two networks meet, a gateway (a router, today) passes packets between them. Each physical network can be as weird as it likes, because TCP treats it as a dumb pipe and fixes correctness at the ends.

Around 1978 they split TCP in two.

Why split it? Reliability isn't free — acknowledgements and resends cost time, and not every app wants to pay. So 1980 added a bare-bones sibling, UDP: fire the data off and don't look back.

TCP for a file, UDP for a live call — the pick comes down to whether losing a packet or waiting for it hurts more.

Then the birthday: on January 1, 1983, ARPANET dropped NCP and switched everyone to TCP/IP overnight — a "flag day," no rollback. Different networks could finally speak one language.

A quick detour: we almost ran out of addresses

A side plot, but a good one. An IPv4 address is 32 bits — about 4.3 billion of them, which felt infinite until everyone started carrying three devices.

Two fixes, opposite in spirit. NAT (1994) is the hack: a whole home shares one public address, the router juggling the rest — cheap, and it quietly broke the idea that every device is directly reachable. IPv6 (1998) is the real fix: 128-bit addresses, effectively endless.

So why, in 2026, are we still not fully on IPv6? Because a flag day only works once. In 1983 the internet was small enough to switch overnight; now every router, OS, and device has to change, so IPv6 has crept along beside IPv4 for 25+ years. We ran out of free IPv4 blocks in 2011 and still run mostly on it — the first sign of a pattern worth holding onto: the better design doesn't win if too much already depends on the old one.

Nobody wants to memorize numbers

Reaching machines by raw IP is like memorizing phone numbers with no contacts app. The first fix was one file — HOSTS.TXT — mapping names to addresses, edited by hand by a single group in California and downloaded by everyone else.

It cracked fast: one team owned every name on earth, everyone re-downloaded the file to stay current, and name clashes had exactly one winner. The tempting fix — "make the central server bigger" — misses the point. The file was already central; that was the bottleneck.

So in 1983 Paul Mockapetris designed DNS: a tree that hands out ownership on the way down.

Own example.com and everything under it is yours — api.example.com exists the moment you say so, no central approval. That's how DNS scales to hundreds of millions of domains with no master list.

Finally, something for humans

Quick untangling: the internet isn't the Web. The internet is the road network; the Web is one kind of traffic on it, next to email and file transfer.

The pain, late-'80s at CERN: documents lived on a zoo of incompatible machines, each with its own login and tools, nothing linking them. Between 1989 and 1991, Tim Berners-Lee shipped three things that only make sense together.

The magic was the hyperlink — one document pointing straight at another anywhere on earth, fetched with a click. So ordinary now it's hard to remember it was ever an invention.

HTTP then grew up in stages:

• 0.9 (1991): one line — GET /page.html — server returns HTML and hangs up.

• 1.0 (1996): headers, status codes, POST, content types (so it could carry more than HTML).

• 1.1 (1997): still the baseline, mostly for keep-alive — reuse one TCP connection instead of a fresh handshake per request. A page with 30 images went from 30 connection setups to one.

And all of it was plaintext — anyone on the wire could read or change it.

Locking the doors

By the mid-'90s people wanted to buy things online, and plaintext HTTP — card numbers in the clear, no proof the site was even real — made that a non-starter. No encryption, no commerce.

Netscape's SSL had a rough childhood (1.0 never shipped, 2.0 was broken, 3.0 finally held). The neat part was where it went: not inside HTTP, but as a new layer between TCP and HTTP, so HTTP ran unchanged over an encrypted pipe. That layer is the whole difference between http:// and https://.

The rename to TLS was mostly politics — in 1996 the IETF took SSL to standardize it and wouldn't ship under Netscape's brand, so TLS 1.0 is basically SSL 3.1 with a new name.

Every version since is an arms race, each one patching a break in the last. (And "SSL certificate" in 2026 almost always means TLS — the old name just won't die.)

The next problem wasn't a missing feature. It was speed.

Chasing speed

HTTP/1.1 reused a connection but answered one request at a time. A slow first request blocks the whole queue behind it — head-of-line blocking. Browsers hacked around it by opening ~6 connections per site, which is as wasteful as it sounds.

HTTP/2 (2015) fixed that with multiplexing: chop requests into labelled chunks and interleave them over one connection.

Except the problem didn't die — it moved. HTTP/2 still rides on TCP, which delivers one strictly-ordered stream. Lose one packet and TCP freezes every interleaved stream until it's resent. Head-of-line blocking just dropped a floor, into TCP, out of HTTP's reach.

The only real cure is to ditch TCP. QUIC (2012) does exactly that: a new transport on UDP that makes reliability per-stream, so a lost packet stalls only its own stream, with TLS built in. HTTP/3 (2022) is just HTTP over QUIC.

Why UDP instead of fixing TCP? You can't remove ordering from TCP and still have TCP — and the internet is ossified: every router and firewall already understands UDP but drops anything unfamiliar. UDP is the escape hatch everyone leaves open, so QUIC hides inside it.

The thing that stuck with me

The dates fell out of my head fast. Two patterns didn't.

Layering is the whole game. TCP added reliability and IP never knew; DNS added names and IP never knew; TLS added encryption and HTTP never knew; QUIC swapped the entire transport and HTTP barely noticed. Each capability slots in as a layer, and everything above it comes along free.

And the installed base beats the better design. IPv6 is better and still isn't finished 25 years on; QUIC had to disguise itself as UDP just to get through the door. The blocker was never the tech — it was everything already built on the old way. Which isn't really about networking: anything big enough eventually becomes the thing you can't easily change.

Questions for You

1. Why was TCP split into two layers (TCP and IP) instead of staying one protocol — and what did that split make possible?

2. HOSTS.TXT was already a central file. Why wasn't "make the central server bigger" the fix, and what did DNS do instead?

3. HTTP/2 killed head-of-line blocking with multiplexing, yet it came back anyway. Where did it reappear, and why couldn't HTTP/2 reach it?

4. IPv6 has existed since 1998 and is clearly better, yet IPv4 is still everywhere in 2026. What's actually blocking the switch — and where else in this story does the same force show up?

I asked myself one day: my laptop has 8 cores. I have 30 browser tabs open, Spotify running, two terminal processes compiling, and a Slack call going. That’s nowhere near 8 things — so how is the CPU keeping up?

I assumed it was some OS magic I’d never looked at closely. When I finally did, the answer wasn’t one thing — it was three distinct layers, each one built to solve a problem the layer below couldn’t handle alone:

• CPU cores — the hardware. The only place instructions actually execute.

• OS/platform threads — the operating system’s unit of execution. Lets multiple programs share the same cores.

• Software/runtime threads — the language runtime’s unit of execution. Lets a single program handle massive concurrency without paying the OS’s price.

Each level exists because the one below it hit a wall.

Level 1 — The CPU Core

A CPU core executes one instruction stream at a time. That’s it. No magic.

If you have 4 physical cores, you have 4 things that can truly run in parallel at any given moment.

The core is the ground floor. Every unit of execution above it — threads, goroutines, coroutines — eventually runs on a CPU core. The levels above are about how we share those cores across more work than cores can hold simultaneously.

Note: Modern CPUs expose more logical cores than physical ones via hyper-threading (Intel) / SMT (AMD) — a 4-core chip often appears as 8 to the OS. But they share the same physical resources; it’s an optimization, not a true doubling.

Physical cores: [ Core 0 ] [ Core 1 ] [ Core 2 ] [ Core 3 ]
Logical cores:  [  0 ][ 1 ] [  2 ][ 3 ] [  4 ][ 5 ] [  6 ][ 7 ]

Level 2 — The OS Thread

Here’s the problem: you have 2 CPU cores and 10 applications that need to run. Without any additional mechanism, only 2 of those apps could ever execute. The other 8 sit frozen forever.

The operating system was built to solve this. It introduces its own unit of execution — the OS thread (also called kernel thread or platform thread) — and runs a scheduler that time-slices the cores across all running threads.

When a thread’s time slice expires, the OS scheduler pauses it, saves its state, loads another thread’s state, and resumes it on the core. From the outside, all 10 apps appear to be running simultaneously. They’re not — they’re taking turns, fast enough that you don’t notice.

2 CPU cores, 10 applications:

Core 0: [App A] → [App C] → [App E] → [App A] → ...
Core 1: [App B] → [App D] → [App F] → [App B] → ...
         ↑ OS scheduler time-slices all 10 apps across 2 cores

OS threads also solve a second problem: a single application doing multiple things at once. A web server needs to handle incoming requests, write to a database, and serve files — simultaneously. Each task gets its own OS thread, all sharing the same process memory.

This mechanism — pausing one thread and resuming another — is called a context switch. It’s the engine that makes Level 2 work.

What a context switch saves

When the OS switches from Thread A to Thread B on a core, it must:

• Save Thread A’s CPU registers (instruction pointer, stack pointer, general-purpose registers)

• Flush the TLB (the CPU’s cache that maps virtual memory addresses to physical RAM) if switching to a different process — this is one of the more expensive parts

• Load Thread B’s saved state into the CPU

The thread doesn’t notice any of this. When it resumes, it looks like nothing happened.

The Cost of Context Switching

OS threads work well until you need a lot of them. Then the cracks show.

The blocking I/O problem hits hardest. A thread waiting on a database response is put to sleep by the OS — it won’t be scheduled. But it still holds its 1–8 MB stack and still counts against the thread cap. You’re not paying context switch cost for it, but you’re paying the memory cost just to keep it parked.

Level 3 — The Software Thread

This idea isn’t new. In 1963, Melvin Conway described coroutines — routines that hand control back and forth without blocking each other. His key insight: you don’t need the OS to switch tasks. You can do it yourself, in user space, far cheaper.

Language runtimes took that and built their own scheduler on top of the OS. Instead of asking the OS to create a thread per unit of work, they create a thin user-space unit of execution and manage scheduling themselves.

The concept took decades to go mainstream:

These go by different names depending on the language:

• Go → goroutine (go func())

• Java 21+ → virtual thread (Thread.ofVirtual())

• Kotlin → coroutine (launch {})

• Erlang/Elixir → process

But the idea is the same: a lightweight, runtime-managed unit of execution that multiplexes onto a small pool of OS threads.

What makes them cheap:

A Go server holding 100,000 open connections doesn’t have 100,000 OS threads — it has 100,000 goroutines, each consuming ~2–8 KB, multiplexed onto a handful of OS threads equal to the number of CPU cores.

The M:N Mapping

The relationship between the three levels is always many-to-few-to-fewer:

• M:N means many software threads map onto a small, fixed pool of OS threads — typically one per CPU core

• The runtime controls the inner mapping (software → OS); the OS controls the outer mapping (OS thread → core)

• Neither layer knows the full picture — they each manage their own slice

How Runtimes Handle Blocking

With OS threads, a blocked thread holds its OS thread hostage — doing nothing, but still consuming stack memory and still counted against the cap.

Software thread runtimes detect the block and immediately hand that OS thread to another software thread. The blocked one is shelved until the I/O completes, then put back in the queue.

→ 100,000 idle TCP connections = 100,000 parked goroutines (~2–8 KB each), not 100,000 blocked OS threads (~1–8 MB each).

Questions for You

1. You have 4 CPU cores, 4 OS threads, and 1,000,000 software threads. How many software threads can truly run in parallel at any given moment, and what happens to the rest?

2. A software thread blocks on a database call. What should the runtime do — and why is that different from what the OS does when a kernel thread blocks?

3. User-space context switching is described as cheaper than kernel context switching. What specifically makes it cheaper — what does the kernel do that the runtime doesn’t have to?

But what is a “connection” really? What happens below the framework, below the HTTP library, below whatever abstraction you’re using today?

This post gives an overview of the layers that make up a web connection — from raw IP and ports, up through TCP, HTTP, and into the real-time protocols built on top of it.

How two computers send data to each other

Start with the simplest possible question: how does a process on your laptop send bytes to a process running on a server somewhere else?

Two things need to line up:

• IP address — gets the packet to the right machine. Think of it as a building address.

• Port — gets the packet to the right process on that machine. Think of it as the unit number inside the building.
  

  

UDP: when you don’t need the guarantee

Not every app needs reliable delivery. A video call can afford a dropped frame — a brief blip is better than stalling to wait for a retransmit. Same for online games, DNS lookups, live metrics.

UDP is the answer. Write bytes to a socket, they go. No handshake, no acknowledgment, no guarantee of arrival or order. No connection state in the kernel.

• UDP → unreliable, unordered, low overhead — right when speed matters more than correctness

• TCP → reliable, ordered, higher latency to start — right for everything else

Most of what we build needs TCP. But knowing when to reach for UDP is worth understanding.

What a TCP connection actually is

Problem: Raw IP packets can get lost, arrive out of order, or be duplicated. There’s no concept of “a conversation” — just individual packets flying around.

→ TCP solves this. Before any data flows, both sides do a three-way handshake:

After the handshake, a TCP connection exists. But it’s not a wire — it’s a data structure maintained by the kernel on both sides, tracking:

• The 4-tuple: (src IP, src port, dst IP, dst port)

• Sequence numbers — to order bytes correctly

• Receive window — how much data the other side can handle

• Buffers — bytes waiting to be sent or read

That 4-tuple is everything. It’s how the OS knows which connection a packet belongs to. It’s also why one server can handle thousands of simultaneous connections on the same port — the server stays on :443, but each client connects from a different ephemeral port (e.g. 54231, 61042). Every connection has a unique 4-tuple.

Client A: (192.168.1.10:54231, 203.0.113.1:443) ─► Server
Client B: (192.168.1.11:61042, 203.0.113.1:443) ─► Same server
Client C: (10.0.0.5   :49810, 203.0.113.1:443) ─► Same server

All three share the same destination port. None of them conflict. The OS routes each packet to the right connection by matching the full 4-tuple.

HTTP: structure on top of bytes

Problem: TCP gives you a reliable byte stream. But bytes alone have no meaning. If a client connects and starts sending data, how does the server know what’s being asked?

→ That’s the problem HTTP solves. It’s an agreed-upon text format — a set of rules both sides implement independently. The client formats its message following the spec. The server parses incoming bytes expecting the same spec.

A request looks like this:

GET /posts HTTP/1.1\r\n
Host: api.example.com\r\n
Accept: application/json\r\n
\r\n

A response looks like this:

HTTP/1.1 200 OK\r\n
Content-Type: application/json\r\n
Content-Length: 42\r\n
\r\n
{"data": ...}

That blank line (\r\n\r\n) is the delimiter — headers above, body below. No magic, no binary encoding. Just text with a rigid format. The server reads raw bytes off the socket, splits on that delimiter, and parses the verb, path, headers, and body from what came before and after.

HTTPS: encrypting the pipe

Problem: One problem with plain HTTP - everything is sent in the clear. Any hop between you and the server can read it — passwords, tokens, session cookies, everything.

→ HTTPS wraps the TCP connection in TLS before HTTP runs on top of it. The two sides negotiate a shared secret during a handshake, and from that point on, the bytes on the wire are encrypted. HTTP itself doesn’t change — it still runs exactly the same way, just over an encrypted transport.

I wrote a more detailed breakdown of how the TLS handshake works here: Simplified overview of the SSL/TLS handshake

The problem HTTP doesn’t solve

HTTP is request/response. The client sends a request, the server sends a response, done. The client always initiates.

That works fine for fetching data on demand. But what if the server needs to push something to the client without being asked? A new chat message, a stock price moving, a build finishing, an LLM streaming its answer token by token.

The naive approach is short polling — the client asks every few seconds. But you can’t win: poll too often and you hammer the server; poll too rarely and updates feel stale.

Three patterns emerged to solve this, all starting from plain HTTP.

Long polling: bending the rules

The idea: what if the client sends a request, but the server just doesn’t respond until it has something to say?

The client sends GET /events. The server holds the connection open — subscribing to some internal event source (e.g. a Redis channel). When an event fires, it responds. The client immediately reconnects and waits again.

It’s pure HTTP. No protocol changes, no special infrastructure, works through any proxy or firewall that speaks HTTP.

The downside: every message costs a full HTTP round trip and a reconnect. Fine when messages are infrequent. Gets expensive when they’re not.

SSE: streaming the response

SSE makes one small but powerful change to the model: the server never closes the response.

The client sends a normal GET with Accept: text/event-stream. The server responds with headers and keeps the body open indefinitely, writing events as they arrive using chunked transfer encoding — no Content-Length, chunks land as they’re ready.

It’s still one single HTTP response. No reconnect overhead per message — the connection stays open.

The constraint: server → client only. At the TCP level the connection is full-duplex, bytes could flow both ways. But HTTP/1.1 is strictly request → response — once the server starts streaming that response, the client’s HTTP stack won’t send another request on it. The constraint lives in the HTTP layer, not TCP.

The standard workaround: SSE for server → client, a separate POST for client → server. That’s exactly how most LLM chat APIs work — you POST a message, you read the streamed answer back over SSE.

WebSocket: bidirectional communication

SSE and long polling both stay inside HTTP’s request/response model. WebSocket doesn’t.

It starts as HTTP — the client sends a regular HTTP/1.1 request with a special header:

GET /chat HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==

The server responds with 101 Switching Protocols. After that handshake, the bytes on the TCP connection are no longer HTTP. They’re WebSocket frames — a lightweight binary framing protocol. Both sides can send at any time, with no request/response pairing.

Summary

Note: This post is intentionally an overview. A few things were left out to keep the focus:

• Socket binding — how a web server actually calls bind() and listen() to claim a port and accept connections. That’s a post on its own.

• DNS — how a domain like api.example.com gets resolved to an IP address before any connection can even start.

• NAT — how your router maps your private IP (192.168.x.x) to a public IP so packets can travel the internet and find their way back.

Questions for You

1. TCP connections are identified by a 4-tuple. What happens if a client’s NAT mapping expires mid-session and the router reassigns its ephemeral port — from the server’s perspective, what does that look like?

2. Two clients are both behind NAT and have no public IP of their own. How can they establish a direct connection and talk to each other?

3. Your app needs the server to push updates to the client. Messages are infrequent and the environment is a restricted corporate network. Would you pick long polling or SSE — and why?

Series Roadmap

1. Simple agent CLI loop

2. Add tools support, allow agent to perform file operations, command execution, search,…

3. MCP support with configurable settings (http://modelcontextprotocol.io)

4. ACP support for IDE integration (http://agentclientprotocol.com)

5. Prompt compression for long-running contexts ← This part

What is prompt compaction?

In the previous parts of this series, our coding agent has grown significantly more capable. It can execute tools, connect to MCP servers, and even operate directly inside the IDE UI thanks to ACP (agentclientprotocol.com).

Problem

When the agent tackles large, long-running tasks, a new limitation emerges: the LLM’s context window. As the agent continues to loop, earlier messages are eventually pushed out of the context. When that happens, the agent may lose sight of the original goal, forget important constraints, and start drifting in the wrong direction.

Solution - Prompt compaction

Prompt compaction is a technique for keeping long-running conversations within an LLM’s context window by condensing past messages instead of dropping them entirely.

Rather than sending the full conversation history on every loop, the agent periodically summarizes or restructures older interactions into a short, high-signal representation that preserves goals, decisions, constraints, and important facts. This compacted prompt replaces verbose history while maintaining continuity.

In short: prompt compaction lets an agent remember what matters, even when the raw conversation becomes too large to fit in context.

High-Level Flow

Implementation

First, define a compaction method. It checks whether the conversation is close to the token limit and, if needed, compresses the history and returns an updated version.

async def compact_messages_if_needed(
    messages,
    current_tokens,
    token_limit,
    recent_count,
):
    # If context is still small, do nothing
    if current_tokens < token_limit:
        return messages

    # Assume first message is system prompt
    system_message = messages[0]
    remaining = messages[1:]

    # Split old vs recent messages
    old_messages = remaining[:-recent_count]
    recent_messages = remaining[-recent_count:]

    # Summarize old messages
    summary = await summarize(old_messages)

    # Rebuild compacted context
    return [
        system_message,
        SystemMessage(content=f"Conversation summary:\n{summary}"),
        *recent_messages,
    ]

The prompt used to guide the AI to generate the summary is:

Summarize the following conversation history concisely.

## Capture:
1. **User's Objective** - What the user wants to achieve (be specific and detailed)
2. **Key Decisions** - What was decided or agreed upon
3. **Files/Resources** - Files, directories, URLs mentioned
4. **Actions Taken** - File edits, commands run, etc.

Output a summary message directly. No preamble or explanation.

## Conversation:
{conversation_text}

Then, after receiving a response from the LLM, read the input token usage and call the compaction method to update the conversation history if needed.

response: AIMessage = await model.ainvoke(messages)
messages.append(response)

input_tokens = response.usage_metadata.get("input_tokens", 0)
messages = await compact_messages_if_needed(
                     messages=messages,
                     current_input_tokens=input_tokens,
                     token_limit,
                     recent_count = 10, # we keep latest 10 messages
                 )
# The reduced message list is then used as the new conversation history.

Source code

Detail code changes at https://github.com/quango2304/code_buddy/commit/94f28877a910bc551464b9b9d5a6b6cd8d36cb7a

Reference

• https://timwappat.info/understanding-context-windows-in-llms/

• https://dev.to/amitksingh1490/how-we-extended-llm-conversations-by-10x-with-intelligent-context-compaction-4h0a

In Part 3 – MCP Support, we add MCP support to our coding agent client. This allows the agent to function as a real coding assistant: it can execute tools and connect to MCP servers.

However, using a CLI doesn’t always provide a great user experience. To improve usability, we want our agent to work directly inside an IDE with a proper UI.

Series Roadmap

1. Simple agent CLI loop

2. Add tools support, allow agent to perform file operations, command execution, search,…

3. MCP support with configurable settings (http://modelcontextprotocol.io)

4. ACP support for IDE integration (http://agentclientprotocol.com) ← This part

5. Prompt compression for long-running contexts

What we’re building in this part

We will convert our agent into an ACP agent, making it compatible with ACP-enabled IDEs (see the supported clients here: https://agentclientprotocol.com/overview/clients).

With this, the agent can be used directly inside IDEs such as JetBrains IDEs, providing a much better development experience.

What is ACP?

https://agentclientprotocol.com/overview/introduction

AI coding agents and editors are tightly coupled but interoperability isn’t the default. Each editor must build custom integrations for every agent they want to support, and agents must implement editor-specific APIs to reach users. This creates several problems:

• Integration overhead: Every new agent-editor combination requires custom work

• Limited compatibility: Agents work with only a subset of available editors

• Developer lock-in: Choosing an agent often means accepting their available interfaces

ACP solves this by providing a standardized protocol for agent-editor communication. Agents that implement ACP work with any compatible editor. Editors that support ACP gain access to the entire ecosystem of ACP-compatible agents. This decoupling allows both sides to innovate independently while giving developers the freedom to choose the best tools for their workflow.

High-Level Flow

Steps to Add ACP Support to the Agent

In this section, we’ll walk through the key steps required to make our agent compatible with ACP (Agent Client Protocol), so it can run inside ACP-supported IDEs.

Step 1: Create a Session Manager

A session represents a single chat thread in the IDE.
It stores the entire conversation history—including user messages, agent responses, and tool calls—so the agent can maintain context across turns.

from dataclasses import dataclass, field

@dataclass
class Session:
    """Represents an ACP session with conversation history."""
    session_id: str
    cwd: str
    # this is the conversation history
    messages: list[BaseMessage] = field(default_factory=list)
    cancelled: bool = False  # Cancellation flag

    def add_system_message(self, content: str):
        self.messages.append(SystemMessage(content=content))

    def add_user_message(self, content: str):
        self.messages.append(HumanMessage(content=content))

    def add_ai_message(self, message: AIMessage):
        self.messages.append(message)

    def add_tool_message(self, content: str, tool_call_id: str):
        self.messages.append(
            ToolMessage(content=content, tool_call_id=tool_call_id)
        )

This session object becomes the backbone of our ACP agent, ensuring every request is handled with full conversational context.

Step 2: Implement the ACP Agent

Create a New Chat Session

When the IDE starts a new chat, ACP calls this method.
Here, we initialize a new session and return the agent’s configuration (modes and models) to the IDE.

async def new_session(
        self,
        cwd: str,
        mcp_servers: list = None,
        **kwargs
) -> NewSessionResponse:
    """
    Handle session/new request.

    Creates a new conversation session with the agent.
    """
    session_id = kwargs.get("session_id")
    session = self.session_manager.create_session(
        cwd=cwd,
        session_id=session_id
    )

    # Initialize with system prompt
    session.add_system_message(load_system_prompt())

    return NewSessionResponse(
        session_id=session.session_id,
        modes=acp.schema.SessionModeState(
            current_mode_id="auto",
            available_modes=[
                acp.schema.SessionMode(
                    id="auto",
                    name="Auto",
                    description="Autonomous coding agent mode"
                )
            ]
        ),
        models=acp.schema.SessionModelState(
            current_model_id=model_id,
            available_models=[
                acp.schema.ModelInfo(
                    model_id=model_id,
                    name=model_id,
                )
            ]
        )
    )

Handle User Messages

Whenever the user sends a message in the IDE chat panel, this method is invoked.

The flow is:

1. Combine the new user message with previous messages

2. Send the full context to the LLM

3. Execute any requested tool calls

4. Stream responses back to the IDE

5. End the turn when no more tool calls are needed

async def prompt(
        self,
        prompt: list[
            TextContentBlock | ImageContentBlock | AudioContentBlock |
            ResourceContentBlock | EmbeddedResourceContentBlock
        ],
        session_id: str,
        **kwargs
) -> PromptResponse:
    """
    Handle session/prompt request.

    Receives user message and processes it through the agent.
    Uses streaming updates to report progress to the IDE.
    """
    session = self.session_manager.get_session(session_id)

    # Extract user text from prompt content
    user_text = self._extract_prompt_content(prompt)
    session.add_user_message(user_text)

    # Get tools and model
    tools = await self._get_tools()
    model = create_model().bind_tools(tools)

    # Agent loop (may involve multiple tool calls)
    while True:
        response: AIMessage = await model.ainvoke(session.messages)
        session.add_ai_message(response)

        if response.tool_calls:
            print(
                f"Tool calls detected: {response.tool_calls}",
                file=sys.stderr
            )

            # Stream partial content to the IDE
            await self._stream_content_blocks(session_id, response)

            # Execute tool calls
            for tool_call in response.tool_calls:
                await self._process_tool_call(
                    session_id,
                    session,
                    tools,
                    tool_call
                )
        else:
            # Final response — stream and end turn
            await self._stream_content_blocks(session_id, response)
            break

    return PromptResponse(stop_reason="end_turn")

This loop is what turns our agent into a true autonomous coding agent inside the IDE.

Step 3: Add the ACP Command Entry Point

Finally, we expose an entry point to run the agent using the ACP SDK.

async def run_acp_agent():
    """Entry point to run the coding agent using ACP SDK."""

    agent = ACPAgent()
    print("ACP Server starting...", file=sys.stderr)

    try:
        await acp.run_agent(agent, use_unstable_protocol=True)
    finally:
        await cleanup_mcp_connections()
        print("ACP Server stopped.", file=sys.stderr)

Once this is running, your IDE can connect to the agent via ACP and interact with it as a first-class coding assistant.

Step 4: Config IDE and test our agent

In Jetbrain IDE AI chat, config the ACP as below

Ouput

Possible Enhancements

So far, we’ve implemented a basic ACP agent that can already run inside ACP-compatible IDEs.
However, ACP offers much more, and there are several areas where we can extend our implementation.

• Ask Mode
  Introduce an approval flow where the agent asks the user to Allow / Deny before executing sensitive tool actions.

• Persistent Sessions
  Store session messages in persistent storage (e.g. disk or database) instead of keeping everything in memory, allowing conversations to survive restarts.

• Rollback Support
  Enable rollback and undo functionality by implementing file read/write operations using ACP client–provided tools.

These features would move the agent from a basic proof of concept to a fully featured IDE-native coding agent.

What’s next?

In Part 5, we will handle long context problems, allow out agent to work in a long context thread.

Source code

Check the details code change for this part in below commit

https://github.com/quango2304/code_buddy/commit/2699e2b5db39de37e3d6749bc5f4875d6b65f58f

In Part 2 - Support tools execution, we added tool execution support for our agent.
Our agent could:

• Read and write files

• Execute shell commands

• Search code

• Edit existing files precisely

All of these tools were implemented by us.

And that’s where an important question appears:

If every agent developer has to re-implement tools themselves, how can we reuse tools built by other developers?

Even more importantly:

How can the same tool be used by different LLM providers without rewriting everything?

This is exactly the problem MCP was created to solve.

Series Roadmap

1. Simple agent CLI loop

2. Add tools support, allow agent to perform file operations, command execution, search,…

3. MCP support with configurable settings (http://modelcontextprotocol.io) ← This blog

4. ACP support for IDE integration (http://agentclientprotocol.com)

5. Prompt compression for long-running contexts

Why Tool Reuse Is Hard

Before MCP, most agents followed this pattern:

1. Developer defines tool schemas

2. Developer teaches the LLM how to call them

3. Developer executes the tool manually

4. Result is passed back to the LLM

This works — but it doesn’t scale.

Problems:

• Every agent re-implements the same tools (filesystem, git, search, etc.)

• Tools are tightly coupled to one LLM provider

• No standard way to discover, describe, or share tools

• Tool authors must target specific agent frameworks

We needed a universal interface between LLMs and tools.

What We’re Building in This Part

In this part, we’ll extend our coding agent to support MCP servers.

Specifically, we will:

• Allow users to configure MCP servers:

  • Local MCP servers via stdio

  • Remote MCP servers via HTTP

• Load MCP tools dynamically at runtime

• Expose MCP tools to the LLM just like native tools

After this step, our agent will no longer be limited to tools we personally wrote.

High-Level Flow

Steps to Add MCP Support to the Agent

Now that we understand what MCP is and how it fits into our agent, let’s walk through the concrete steps needed to support MCP in code.

At a high level, adding MCP support requires three things:

1. Connect to MCP servers and fetch their tool definitions

2. Provide MCP tools to the LLM alongside existing internal tools

3. Execute MCP tools when the LLM requests them

Let’s go step by step.

Step 1: Connect to MCP Servers and Load Tools

The first responsibility of the agent is tool discovery.

Instead of hardcoding tools, we:

• Load MCP server configurations

• Establish connections (stdio or HTTP)

• Ask each MCP server for the tools it exposes

async def get_mcp_tools() -> list[StructuredTool]:
    """Get all tools from connected MCP servers as LangChain tools."""
    print("Loading mcp tools...")
    tools = []

    # Load configs from mcp_servers.json file
    configs = load_mcp_configs()

    # Connect to all MCP servers
    await client_manager.connect_all(configs)

    # Get tools from each connection
    for connection in client_manager.get_connections():
        connection_tools = await _get_tools_from_connection(connection)
        tools.extend(connection_tools)

    return tools

Step 2: Provide All Tools to the LLM

Next, we merge internal tools and MCP tools into a single list.

From the LLM’s perspective, there is no distinction.

async def get_all_tools() -> list[BaseTool]:
    """Get all available tools from all sources."""

    # List of our locally implemented tools
    tools: list[BaseTool] = [
        run_command,
        read_command_output,
        send_command_input,
        replace_file_content,
        grep_search,
        *get_langchain_tools(),
    ]

    # Add tools from MCP servers (stdio or http)
    mcp_tools = await get_mcp_tools()
    tools.extend(mcp_tools)

    return tools

Step 3: Execute Tools When the LLM Requests Them

Once tools are registered, execution follows the standard agent loop.

The LLM can respond in two ways:

• A normal message → stop and wait for user input

• A tool call → execute and continue the loop

while True:
    response: AIMessage = await model.ainvoke(messages)
    messages.append(response)

    # Check if there are tool calls to handle
    if response.tool_calls:
        # Print any reasoning or thinking blocks
        _print_agent_text_output(response)

        # Execute each tool call
        for tool_call in response.tool_calls:
            result = await execute_tool(tools, tool_call)

            tool_message = ToolMessage(
                content=result,
                tool_call_id=tool_call["id"],
            )
            messages.append(tool_message)
    else:
        # No more tool calls → final answer
        _print_agent_text_output(response)
        break

Testing MCP Support with Real Servers

Now that MCP support is wired into the agent, let’s test it with real MCP servers.

To make MCP configuration flexible, we support two configuration locations:

1. Project-level MCP configuration

2. Default MCP configuration shipped with the agent

Project-Level MCP Configuration

At the project level, MCP servers are configured in:

.code-buddy/mcp_servers.json

This file lives where the agent is run, making it easy to customize tools per project.

Example: Adding Context7

We’ll add Context7, an MCP server that allows the agent to retrieve up-to-date documentation for libraries and packages.

{
  "mcpServers": {
    "context7": {
      "url": "https://mcp.context7.com/mcp",
      "headers": {
        "CONTEXT7_API_KEY": "get free at context7.com",
        "Accept": "application/json, text/event-stream"
      }
    }
  }
}

Default MCP Configuration (Shipped with the Agent)

In addition to project-level configs, the agent also ships with default MCP servers.

These live inside the agent source code:

src/mcp/default_mcp_servers.json

Default MCP servers are:

• Enabled automatically

• Useful across many projects

• Easy to override or disable

Example: Adding Auggie

We add Auggie, which allows the agent to retrieve information from an indexed codebase using Augment’s context engine.

Documentation:
https://docs.augmentcode.com/context-services/mcp/overview

{
  "mcpServers": {
    "auggie": {
      "command": "auggie",
      "args": [
        "--mcp"
      ]
    }
  }
}

Notes

• Need to install and login auggie (free)

  npm install -g @augmentcode/auggie@prerelease & auggie login

• We also need to update our system prompt to guide the LLM to always use the Auggie context tools to retrieve the codebase data.

Ouput

(code-buddy) ➜  code_buddy git:(main) ✗ code-buddy                        
Loading mcp tools...
🔧 Starting Auggie MCP Tool Server...
📝 Stdio mode
🔍 Starting workspace indexing in background...
📋 Available tools: codebase-retrieval
✅ MCP tool server started
🔗 Ready for MCP client connections
✅ Workspace indexing complete
Connected to MCP server: auggie
Connected to remote MCP server: context7
Agent ready. Type 'quit' to exit.
------------------------------------------------------------
[You]: Hi, list out your tools
------------------------------------------------------------
[Agent]: Here are the tools I have available:

## File Operations
| Tool | Description |
|------|-------------|
| **read_file** | Read the content of a file |
| **write_file** | Write content to a file (create or overwrite), with optional append mode |
| **copy_file** | Copy a file to a new location |
| **move_file** | Move or rename a file |
| **file_delete** | Delete a file |
| **list_directory** | List files and subdirectories in a directory |
| **file_search** | Recursively search for files matching a regex pattern |

## File Editing
| Tool | Description |
|------|-------------|
| **replace_file_content** | Replace a single contiguous block of content in a file |

## Command Execution
| Tool | Description |
|------|-------------|
| **run_command** | Execute shell commands (foreground or background mode) |
| **read_command_output** | Read output from a background command using its process ID |
| **send_command_input** | Send input to a running background command or terminate it |

## Search
| Tool | Description |
|------|-------------|
| **grep_search** | Search for regex patterns in files using grep |
| **auggie_codebase-retrieval** | Semantic search of the codebase using natural language queries (primary tool for understanding code structure) |

## Documentation (Context7)
| Tool | Description |
|------|-------------|
| **context7_resolve-library-id** | Resolve a library/package name to a Context7-compatible library ID |
| **context7_query-docs** | Query up-to-date documentation and code examples for any programming library or framework |

---

**Key Usage Notes:**
- **codebase-retrieval** is my primary tool for searching and understanding your codebase semantically
- **grep_search** is best for finding exact string/pattern matches across files
- **Context7 tools** help me get current documentation for external libraries
- For editing files, I use **replace_file_content** rather than rewriting entire files
- I can run commands in the background and interact with them via process IDs

Is there something specific you'd like me to help you with?

------------------------------------------------------------
[You]: What this project does?
------------------------------------------------------------
Executing tool: auggie_codebase-retrieval, tool_args: {'information_request': 'What is this project about? Give me a high-level overview of the project structure, main purpose, ke...'}

📦 Tool result:
[{'type': 'text', 'text': 'The following code sections were retrieved:\nPath: README.md\n     1\t# C...
------------------------------------------------------------
[Agent]: ## Project Overview: **Code Buddy**

**Code Buddy** is a **minimal, production-ready coding agent** built with **LangChain** for learning purposes. It's essentially an AI-powered CLI assistant that can interact with your codebase.

... removed to keep the example concise...

### Roadmap Status

- ✅ Simple agent CLI loop
- ✅ Tools (file operations, command execution, search)
- ✅ MCP support with configurable settings
- 🔜 ACP support for IDE integration
- 🔜 Prompt compression for long contexts
- 🔜 Memory, rules, and task management

------------------------------------------------------------
[You]: 

References

• https://substack.com/home/post/p-175538124 - How the MCP server and MCP client communicate with each other?

• https://modelcontextprotocol.io/docs/getting-started/intro

What’s next?

In Part 4, our agent will be usable in the IDE UI (both VSCode and Jetbrain IDEs)

Source code

Check the details code change for this part in below commit

https://github.com/quango2304/code_buddy/commit/0c1f9d3cbce05db0353d115a8579fa3e6ce9e227

In Part 1, we built a simple agent CLI loop that could chat with an LLM while maintaining conversation history. Now it’s time to give our agent superpowers – the ability to interact with the real world through tools.

Series Roadmap

1. Simple agent CLI loop

2. Add tools support, allow agent to perform file operations, command execution, search,… ← This blog

3. MCP support with configurable settings (http://modelcontextprotocol.io)

4. ACP support for IDE integration (http://agentclientprotocol.com)

5. Prompt compression for long-running contexts

What We’re Building in This Part

By the end of this article, our agent will be able to:

• Read and write files

• Execute shell commands

• Search code

• Edit existing files precisely

This is the foundation that separates a “chatbot” from a true coding agent.

How tool calling works?

Before diving into code, let’s understand how tool calling works:

The key insight: You provide the tools and their descriptions, and the LLM response with tool execution info, you will execute it then give back the LLM tool result.

Step 1: List all needed tools

For a coding agent, we need three categories of tools.

File Operations

• read_file

• write_file

• copy_file

• move_file

• file_delete

• list_directory

• file_search: Find files by pattern (example: find all files that have *.py name pattern)

• replace_file_content: Edit existing files precisely

Command Execution

• run_command: Execute shell commands

• read_command_output: Get output from background processes

• send_command_input: Interact with running processes

Search Tools

• grep_search: Search for patterns in files (example: find all files that contain the @tool text)

For now, we only support simple grep search, later we will add the code base indexing capality and semantics search.

Step 2: Implement those tools

Step 2a: Reuse LangChain FileManagementToolkit

Luckily, LangChain’s FileManagementToolkit gives us many tools out of the box, so we do not need to build ourself a lot of tools:

uv add langchain-community

Step 2b: Build the remaining tools

The @tool decorator

The decorator:

• reads function signatures

• extracts parameter descriptions

• uses docstrings for the tool description

from langchain_core.tools import tool
from typing import Annotated

@tool
def my_tool(
    param1: Annotated[str, "Description of param1"],
    param2: Annotated[int, "Description of param2"] = 10,
) -> str:
    """Description visible to the LLM."""
    return f"Result: {param1}, {param2}"

Command Execution Tool

Read implementation details here

File Editing Tool

Read implementation details here

Grep Search Tool

Read implementation details here

Step 4: Aggregating Tools

Let’s add helper functions

The execute_tool function is the bridge between what the LLM requests and what actually happens.

Step 5: Updating the Agent Loop

Now for the crucial change – updating our loop to handle tool calls

Get all tools and supply them to the LLM. Using bind_tools, we give the model the full list of tools along with their input parameters, descriptions, and output definitions.

There’s no magic behind tool execution. The LLM only receives tool definitions and then generates text indicating whether it wants to call a tool, including the tool name and parameters. We execute the tool ourselves and return the results to the LLM to continue the loop.

The Tool Execution Flow

1. User asks: “What files are in the src directory?”

2. LLM returns: tool_calls: [{"name": "list_directory", "args": {"path": "src"}}]

3. We execute list_directory("src")

4. We add the result as a ToolMessage

5. LLM sees the result and formulates the final answer

Step 6: The System Prompt

The system prompt guides how the agent uses tools. Here's the critical section:

# Role
You are an agentic coding AI assistant with access to the developer’s codebase through tools.

# Preliminary tasks
Before executing, gather necessary information. Use information-gathering tools first.

# Making edits
Use replace_file_content for edits. Always read files first to get accurate ranges.

# Tool invocation rules
Arguments must be passed as flat key-value pairs.

❌ WRONG:
{ "query": { "command": "ls -la" }}

✅ CORRECT:
{ "command": "ls -la", "working_dir": "." }

# Available Tools
(List and describe each tool) 

Key points:

• Tell the agent to gather information FIRST

• Be explicit about tool argument format (we found LLMs sometimes wrap args incorrectly)

• List all available tools with usage examples

Read the full prompt at here.

Try It Out

Run the agent

uv sync
uv run python -m src.main

Try to ask “What the readme file contains?”

What’s Next?

In Part 3, we’ll add MCP (Model Context Protocol) support — allowing your agent to connect to external tools and services like databases, APIs, and third-party agents.

Source Code

Check the details code change for this 2nd step in below commit

https://github.com/quango2304/code_buddy

Introduction

Have you ever wondered how tools like Cursor, Copilot, or Claude actually work? Underneath, they all rely on one simple idea: an agent loop.

In this series, I will show step by step with real code explained by the code comment line by line.

In this first tutorial, we’ll build that core piece — a small CLI that keeps a running conversation with an LLM. It’s simple, but everything else is built on top of it.

Series Roadmap

1. Simple agent CLI loop ← This blog

2. Add tools support, allow agent to perform file operations, command execution, search,…

3. MCP support with configurable settings (http://modelcontextprotocol.io)

4. ACP support for IDE integration (http://agentclientprotocol.com)

5. Prompt compression for long-running contexts

Prerequisites

• Python 3.13+ (using uv for package management)

• An API key from Anthropic or Google AI

  • most easy and free to get is Gemini API key get from

    http://aistudio.google.com

  • for Anthropic or GPT, can use some tricks to get a running LLM API server from your existing subscription, example https://github.com/router-for-me/CLIProxyAPI.

• Basic familiarity with Python

Understanding the Agent Loop

Before we write any code, let’s understand what an “agent loop” actually is:

┌─────────────────────────────────────────────────────────┐
│                     AGENT LOOP                          │
├─────────────────────────────────────────────────────────┤
│                                                         │
│    ┌──────────┐                                         │
│    │  User    │                                         │
│    │  Input   │                                         │
│    └────┬─────┘                                         │
│         │                                               │
│         ▼                                               │
│    ┌──────────┐      ┌──────────┐       ┌──────────┐    │
│    │ Message  │ ───► │   LLM    │ ───►  │ Response │    │
│    │ History  │      │  Model   │       │ Handler  │    │
│    └──────────┘      └──────────┘       └────┬─────┘    │
│         ▲                                    │          │
│         │                                    │          │
│         └────────────────────────────────────┘          │
│                   (append to history)                   │
│                                                         │
└─────────────────────────────────────────────────────────┘

The loop is simple:

1. You type something.

2. We add it to the conversation history.

3. We send the entire history to the AI.

4. The AI replies, and we show the answer.

5. We save that reply in the history.

6. Then we back to step 1

Project Setup

We’ll use uv to initialize and manage the project.

1. Create the Project

mkdir code-buddy && cd code-buddy
uv init
echo "3.13" > .python-version

2. Project Structure

code-buddy/
├── src/
│   ├── cli_agent/
│   │   └── cli_agent.py # our main agent running loop here
│   ├── prompt/
│   │   └── system_prompt.md # personality + abilities guide
│   ├── model.py # config LLM model
│   └── main.py # CLI entry point
├── pyproject.toml
└── .env # evironment file

3. Dependencies (pyproject.toml)

4. Environment Configuration (.env)

Just one of Anthropic Configuration or Google Gemini Configuration is required.

# Choose your provider: gemini or anthropic
AI_PROVIDER=gemini

# Anthropic Configuration
ANTHROPIC_API_KEY=sk-your-key-here
ANTHROPIC_BASE_URL=https://api.anthropic.com
CLAUDE_MODEL=claude-sonnet-4-5-20250929

# Google Gemini Configuration
GOOGLE_API_KEY=your-key-here
GEMINI_MODEL=gemini-3-flash-preview

Implementation

Step 1: Model Factory (src/model.py)

Step 2: System Prompt (src/prompt/system_prompt.md)

We’re keeping things simple for now—once we add tools and MCP support, this will grow into a fully capable agent.

You are Code Buddy, a helpful code assistant who answers clearly and concisely.

Step 3: The Agent Loop (src/cli_agent/cli_agent.py)

Note that every call sends the full history plus the new message. The LLM doesn’t “remember” anything — it just receives tokens and generates tokens. All conversation context exists only because we resend it each time.

Step 4: Entry Point (src/main.py)

Running Your Agent

uv sync
uv run python -m src.main

Expected output:

[You]: Hello! What can you help me with?
------------------------------
[Agent]: Hello! I'm Code Buddy...
------------------------------
[You]: What I have just said?
------------------------------
[Agent]: You said "Hello! What can you help me with?"

Optional: Install as CLI Tool

# Create the CLI, which watch for changes
uv pip install -e .  

# Create a shell alias so you can run the agent from anywhere
alias coding-agent='<project_absolute_path>/.venv/bin/code-buddy'  

# Run the CLI agent from any where
code-buddy  

Source Code

Check the details code change for this 1st step in below commit

https://github.com/quango2304/code_buddy

Coming Next: Tools

In Part 2, we’ll give the agent real powers:

• Read/write files

• Execute shell commands

• Search codebases

• Multi-step tool loops

Your agent will go from “chatbot” → “basic coding assistant.”

Enter Garbage Collection (GC): the automatic process of reclaiming memory occupied by objects that are no longer in use.

While there are many complex variations, most modern languages rely on concepts derived from two main strategies: Reference Counting and Mark & Sweep. Let’s dive into how they work, their pitfalls, and who uses them.

1. Reference Counting

This is the most intuitive form of garbage collection. It operates on a simple “popularity contest” logic: an object is only useful if someone is pointing at it.

How It Works

Every object in memory has a hidden counter attached to it.

• Count +1: Whenever you assign the object to a variable or another object references it, the counter goes up.

• Count -1: When a variable goes out of scope or is set to null, the counter goes down.

• The Cleanup: The moment the counter hits zero, the garbage collector knows nobody can reach this object. It immediately destroys it and frees the memory.

The Fatal Flaw: Circular Dependencies

Reference counting is fast and distributed, but it has one major Achilles’ heel: Retain Cycles.

Imagine Object A points to Object B, and Object B points back to Object A. Even if the rest of your program stops using both of them, they still reference each other. Their counters will remain at 1 (or higher), preventing them from ever being deleted. This causes a memory leak.

The Solution: Weak References

To solve the circle problem, languages introduce the concept of Weak References.

• Strong Reference: The default. It increases the reference count. “I own this object.”

• Weak Reference: It points to the object without increasing the reference count. “I know about this object, but I don’t need to keep it alive.”

If you have a Parent-Child relationship, the Parent usually holds a strong reference to the Child, but the Child should hold a weak reference back to the Parent. If the Parent is deleted, the Child’s weak link doesn’t stop the Parent from dying.

2. Mark & Sweep (Tracing GC)

Mark & Sweep takes a more holistic approach. Instead of counting individual links, it views memory as a giant web and asks, “Can I travel from the start of the application to this object?”

How It Works

This method runs in distinct phases, usually triggered when memory runs low.

1. The Roots: The GC identifies “GC Roots.” These are objects that are definitely alive (e.g., local variables in the stack, global variables, active threads).

2. Mark Phase: The collector starts at the Roots and traverses all references (like walking a graph). Every object it visits is “Marked” as alive.

3. Sweep Phase: The collector scans the entire heap memory. Any object that was not marked is considered unreachable garbage. It sweeps them away to reclaim space.

When is it Triggered?

Unlike Reference Counting, which happens continuously, Mark & Sweep is “bursty.” It is typically triggered when:

• Allocation Failure: The program tries to create a new object, but there isn’t enough free heap space.

• Threshold Reached: Memory usage hits a specific pre-defined limit (e.g., the heap is 70% full).

• Manual Invocation: The developer explicitly asks for a collection (e.g., System.gc() in Java), though this is generally discouraged.

The Trade-off

The classic downside of Mark & Sweep is the “Stop-the-World” pause. To safely traverse the web of memory, the GC often has to pause the execution of your program completely. While modern algorithms (like Generational GC) have reduced this pause to milliseconds, it can still cause stuttering in high-performance applications.

Summary: Who Uses What?

Different languages choose different trade-offs based on their goals (latency vs. throughput).

Problem: what happens if a database leaks?

→ If passwords aren’t securely stored, a breach can immediately lead to account takeovers — not just on your platform, but potentially across the user’s entire digital life (passwords are often reused across multiple sites).

Level 0: Plain Text Storage

Storing passwords in plain text is the most dangerous and unfortunately still a surprisingly common mistake.

If the database leaks:

• Attackers immediately obtain every password exactly as users typed them.

• They can log in as any user with no extra effort.

→ Conclusion: This approach should never be used.

Level 1: Hashing

To avoid storing passwords directly, we hash them.
Hashing transforms a password into a fixed-length output using a mathematical function. The output — the hash — cannot be feasibly reversed back into the original password.

A strong hash function:

• Produces completely different output even for tiny changes in input.

• Is one-way (irreversible in practice).

• Is deterministic (same input gives same output).

Examples of secure general-purpose hashing algorithms:

• SHA-256

• SHA-3

• BLAKE2

However…
If attackers get the hashed passwords, they can use rainbow tables — huge precomputed lists mapping common passwords to their hashes.
This allows them to quickly identify weak or frequently used passwords.

→ Conclusion: This is better than plain text, but still not enough.

Level 2: Hashing + Salting

To defend against rainbow tables, we add a salt — a unique, random value combined with the password before hashing:

hash = HASH(password + salt)

The salt is stored alongside the hash in the database.

Salt choices:

• User ID — unique but predictable (not ideal).

• Random per user — preferred for strong uniqueness.

Salting ensures:

• Even if two users choose the same password, their hashes differ.

• Attackers cannot use precomputed rainbow tables (yeah because those precomputed values does not contain the salt)

But There’s Still a Limitation

If attackers obtain both the hash and the salt (which is common in a DB leak), they can still brute-force passwords by repeatedly hashing guesses:

for candidate in common_passwords:
    if HASH(candidate + salt) == stored_hash:
        password = candidate

This matters because:

• Large public lists of 10,000 to 1,000,000 most common passwords are widely available, example: https://lucidar.me/en/security/list-of-1000000-most-common-passwords/

• These lists cover a significant percentage of real users.

• Fast hash functions like SHA-256 can be computed millions to billions of times per second on GPUs.

So attackers can often test all common passwords in seconds to minutes per user.

→ Conclusion: Salting is necessary — but not sufficient on its own.

Level 3: Hashing + Salting + Peppering

A pepper is a secret value that is not stored in the database.
It is kept separately — typically in application configuration or a secure key vault.

 hash = HMAC-SHA256(key = pepper, message = Argon2id(password, salt))

Now, if the database leaks:

• Attackers get the hash and salt.

• But they still cannot brute-force effectively because they do not know the pepper.

Even if they try to brute-force using common passwords, every attempt will be wrong without the pepper.

Note

We assume the system already has rate limiting in place to prevent attackers from brute-forcing login attempts through the API.

Summary

Security isn’t about a single trick — it’s about layered defenses.
Each step adds protection and reduces the attacker’s advantage.

In the user view, how to protect your account: you should not use a common password, and you better should enable 2FA so even when someone get your password, they still cannot access your account.

Open question: If attacker can update the database, what stops them from replacing other users’ hashes/salts with their own?

Subscribe now

Leave a comment

Below is the simple flow

1. You write a Python .py source file.

2. When run, Python compiles the source into bytecode (automatically, at runtime, lazyly only when a module is used).

3. The Python VM interprets and executes the bytecode step-by-step.

4. The program produces its output and effects.

What is bytecode?

Bytecode is a low-level, interpreter-friendly version of your Python code.
It is not machine code, but a simplified set of instructions that the CPython VM understands.

Why does Python use bytecode instead of executing the source directly?

• Bytecode is already parsed and processed, so the interpreter does less work while running.

• Python can optimize code during compilation (e.g., constant folding) before execution.

• Without bytecode, the interpreter would have to re-parse and analyze the source every time, making execution much slower.

The PYTHONDONTWRITEBYTECODE environment variable

You may notice in the first diagram that bytecode is generated at runtime by the compiler. If this happened every time the code runs, it would be inefficient. In reality, Python caches the compiled bytecode into a .pyc file so it can be reused on the next run.

What is PYTHONDONTWRITEBYTECODE? → is an environment variable that controls whether Python creates and saves bytecode cache files (.pyc) in the __pycache__ folder.

• PYTHONDONTWRITEBYTECODE=0 (default):
  Python writes the compiled bytecode to the cache folder (__pycache__) so it can be reused.

• PYTHONDONTWRITEBYTECODE=1:
  Python does not write .pyc files. The code is recompiled into bytecode every time it runs. This is mainly used in testing or in environments where writing files is not allowed.

Can try to run your python app using PYTHONDONTWRITEBYTECODE=0 and PYTHONDONTWRITEBYTECODE=1 then check the __pycache__ folder is generated or not.

Pre-generate Bytecode at Build Time (optional)

You can generate .pyc files ahead of time using the compileall module. This is useful for packaging or deployment so the application doesn’t spend time compiling on first run.

Run below command and see that the __pycache__ folder is generated without running the program.

python -m compileall your_project/

Bonus

CPython runs Python in an interpreted mode, executing bytecode step-by-step. However, some alternative Python implementations use JIT (Just-In-Time) compilation, which detects frequently-used code paths (hot paths) and compiles them into machine code for faster performance (e.g., PyPy).

Read more about what is Interpret, JIT, AOT here

I’ll keep the intro short — you’ve probably heard of OAuth before, but let’s make sure we’re on the same page.

OAuth (Open Authorization) is a delegation protocol for apps to ask for limited access to your stuff on another service — without you giving them your password.

Think of when you log into a website using Google or GitHub. That “Sign in with Google” button? That’s OAuth doing its thing. It lets the app act on your behalf (say, read your email or fetch your profile) without ever touching your actual credentials.

When OAuth isn’t secure

Even though OAuth is powerful, there are a few spots in the flow where things can go wrong if you’re not careful — token theft, CSRF, replay attacks, and so on.

So, how to secure your OAuth flow?

PKCE, State, and Nonce are added to protect against token theft, CSRF, and replay attacks.

What is PKCE, State, Nonce?

PKCE (Proof Key for Code Exchange)

PKCE adds a layer of verification between the client app and the authorization server.
It works by generating a secret (code verifier) before the authorization request and sending a hashed version (code challenge) to the authorization server.

At the token exchange step, the OAuth server verifies the code challenge against the code verifier:

• This ensures that even if an attacker steals the authorization code, they cannot exchange it for a token without the code verifier.

• Commonly used in mobile and single-page apps where client secrets can’t be stored securely.

State

The state parameter is used to maintain state between the authorization request and callback.
Your app sends a random state value in the request, and the OAuth provider sends it back during the callback.

Your server then checks that the received state matches the one you generated:

• This protects against CSRF attacks, where an attacker could trick a user into visiting a malicious callback URL like
  /callback?code=attacker_code&state=attacker_state.

• Without this check, a user could unknowingly log into an attacker’s account.

Nonce

Nonce (Number used once) is mainly used in OpenID Connect (OIDC) — an identity layer built on top of OAuth 2.0.

• The app generates a random nonce value and includes it in the authentication request.

• When the ID token (JWT) is received, the app verifies that the token’s nonce claim matches the original one.

• This prevents replay attacks — ensuring that an attacker cannot reuse a valid ID token to impersonate a user.

  

  Reference

• https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow-with-pkce

• https://auth0.com/docs/secure/attack-protection/state-parameters

• https://stackoverflow.com/questions/46844285/difference-between-oauth-2-0-state-and-openid-nonce-parameter-why-state-cou

Interpreter: Line-by-Line Execution

An interpreter reads and executes code line by line at runtime, without converting the whole program into machine code beforehand. This makes it great for flexibility and rapid development, but can sacrifice performance.

takes a source program 𝑃, and runs 𝑃, that means affecting the world through running 𝑃

Advantages:

• Fast to start (no need to compile first).

• Great for scripting.

• Highly portable (runs on any platform with the interpreter).

• Better support for reflection, dynamic code loading, and runtime code generation than AOT compilers

Disadvantages:

• Slower execution (each line must be interpreted on the fly).

• Higher CPU overhead at runtime.

Examples:

• Python (if you use Pypy, it can support JIT compile)

• Ruby in the early day (support JIT since version 2.6)

• PHP in the early day (support JIT since version 8.0)

Just In Time (JIT)

A Just-In-Time compiler blends the benefits of interpretation and compilation. The code is initially interpreted, but parts of it are compiled during runtime — often those that are executed frequently (called hot code) — for better performance.

Advantages:

• Speeds up performance after the first run.

• Adapts dynamically to runtime conditions.

• Enables advanced optimizations (e.g., type inference, inline caching).

• Better support for reflection, dynamic code loading, and runtime code generation than AOT compilers

Disadvantages:

• Slower startup time (compilation happens during execution).

• Higher memory usage.

Examples:

• Java (HotSpot JVM), C# (.NET CLR), JavaScript (V8 engine), Dart (debug mode)

AOT (Ahead of time)

Ahead-Of-Time (AOT) compilation means translating source code into machine code before execution. The result is a native executable optimized for the target platform.

takes a source program 𝑃, produces a new program 𝑃𝑎𝑠𝑚. When we run 𝑃𝑎𝑠𝑚, it has the same (hopefully) effect on the world as 𝑃

Advantages:

• Very fast startup and runtime performance.

• Lower runtime memory overhead.

Disadvantages:

• Longer build times.

• Less runtime flexibility (no dynamic optimizations).

• Require platform-specific builds.

• reflection, runtime type inspection, or dynamic code loading — are harder to support in AOT (yeah if support reflection, it’s hard to optimize the code size during build time because the compiler wont know the code is unused or not)

Examples: C, C++, Rust, Go, Swift.

References

• https://cs.stackexchange.com/questions/56478/jit-aot-interpretation-what-are-differences-between-them

• https://www.howtogeek.com/what-is-jit-how-apples-rules-are-holding-back-iphone-game-emulators/

  Subscribe now

The two diagrams below show simplified versions of the TLS 1.2 and TLS 1.3 handshake processes.
(They’re much more complex in reality, but these focus on the key ideas.)

The main point to understand is:

• During the handshake, the client verifies the server’s certificate chain using the trusted CA (Certificate Authority) public keys preinstalled in the client’s operating system or browser.

• Because of this, it’s possible to install custom certificates for specific purposes.

Why Custom Certificates Are Used

1. Intercepting and decrypting network traffic (for debugging)
   Tools like Proxyman, Fiddler, or Charles Proxy work by acting as a “man in the middle.”
   They install their own TLS certificate so they can capture and decrypt HTTPS requests.

2. Company-installed certificates
   Some organizations install custom TLS certificates on company laptops to inspect or secure network traffic.

   • Apps running on the laptop work fine because the system trusts that certificate.

   • However, if you run the same app inside a Docker container, the container doesn’t automatically trust that company certificate.
     → To fix this, you need to add the certificate to the container’s trusted store.

As a developer, how to avoid your app requests being captured?

→ Use SSL Pinning

SSL Pinning is a security technique where a client application verifies the server’s SSL/TLS certificate against a known value (certificate or public key), instead of relying solely on the operating system’s or browser’s built-in certificate authorities (CAs).
This helps prevent Man-in-the-Middle (MITM) attacks by ensuring the app only trusts the expected server certificate.

There are two main types of SSL Pinning:

1. Static Pinning:
   The server’s certificate or public key is hardcoded directly into the application.

   • Pros: Very secure if managed correctly.

   • Cons: If the server certificate changes (for example, during renewal), the app must be updated and redeployed to include the new certificate or key.

2. Dynamic Pinning:
   The app fetches and stores (pins) the server’s certificate or public key during the first successful connection(often called “trust on first use,” or TOFU).

   • On future connections, the app verifies that the certificate matches the one previously pinned.

   • Pros: More flexible, doesn’t require app updates for certificate renewals.

   • Cons: Less secure initially — if the first connection is compromised, the app could pin a malicious certificate.

References: https://www.ibm.com/docs/en/cloud-paks/z-modernization-stack/2023.4.1?topic=handshake-tls-12-protoco

Subscribe now